diff --git a/.agents/skills/journal-build/references/native-provenance.md b/.agents/skills/journal-build/references/native-provenance.md index 740de5fe..885203b0 100644 --- a/.agents/skills/journal-build/references/native-provenance.md +++ b/.agents/skills/journal-build/references/native-provenance.md @@ -5,6 +5,12 @@ requires the OCaml runtime and mldoc. XMLm is a locked dependency used by mldoc' OPML units; an unused XMLm unit may be removed by the OCaml linker. Any unresolved XMLm reference is rejected. No application anchor or global `-linkall` is needed. +## Explicit sync-network diagnostic role + +`--sync-network-probe` is a separate producer/import role, not a production feature or a raw-object exception. Its schema-2 envelope, hash-bound source/link records and Dune rule name the whitelisted `tool/sync_acceptance_probe/native_probe.exe.o`; imports reject absent or different roles/targets. Probe source identity includes that directory's native sources as well as the unchanged production source selection. A complete hash-bound current Dune/lock/static-link configuration snapshot permits this explicitly authorized diagnostic target to be produced from an uncommitted worktree; later changes fail closed. Ordinary production imports retain their existing producer-commit configuration guard. Probe receipts must themselves bind the probe role and cannot become production acceptance. Original compiler, archive, runtime header, target C/GMP, mldoc, source and final-input checks remain in force. Manifests separately record production-code source validation, `production_acceptance=false` and `cross_abi_validated=false`. + +The diagnostic requires `--configuration debug`. New producers bind configuration and the actual queried Dune profile in immutable source/link records and retain the Dune query with its rule; import compares these against the requested configuration and selected target. Historical envelopes without configuration/profile proof require a producer rerun, never an inferred default. Final acceptance re-queries the same profile and whitelisted target and rejects a changed normalized rule even when file/tool hashes still match. + ## Ordinary archive inputs (schema 2) New producers use `schema_version: 2`, with hash-bound `source` and `link` records diff --git a/.agents/skills/journal-build/references/targets.md b/.agents/skills/journal-build/references/targets.md index 9a1a9b3f..e76a479d 100644 --- a/.agents/skills/journal-build/references/targets.md +++ b/.agents/skills/journal-build/references/targets.md @@ -11,6 +11,8 @@ All targets are arm64 with deployment target >=26.0. Choose an external `--cache Default iOS runtime builds and raw `JOURNAL_OCAML_OBJECT`/extra-object injection are refused. Existing `lui_probe_host.py` callers keep `--native-object`: the helper selects isolated `--probe-native-object` or explicit `--link-only` when absent, with its own host-local cache. Probe import requires external staged Swift/plist, a distinct `-probe` bundle ID, a matching macOS/Simulator object and runtime/bridge symbols; it hashes the object before/after building. It never restamps, accepts device builds or claims production source/ABI proof. Warm-start Simulator inputs must already match Simulator. +The separate real-App sync diagnostic uses `--sync-network-probe` on both the macOS `--producer-only` invocation and the Simulator `--experimental-host-restamp --native-manifest` import. It whitelists `tool/sync_acceptance_probe/native_probe.exe.o`, keeps locked production dependencies and maintained Swift sources, and packages `com.logseq.journal.sync-probe` with `JournalSyncNetworkProbe=true`. It is not `--probe-native-object`: raw imports, stubs, device builds and ordinary macOS Apps are refused for this role. Role/target/source/configuration evidence and cache identities are distinct; cross-role imports fail closed. See the [probe runbook](../../../../tool/sync_acceptance_probe/README.md) for review gates, isolated runner opt-in, private launch inputs and heartbeat supervision. Only the main acceptance owner installs/launches a fresh device; these diagnostics never establish ordinary production acceptance. + Host ARM64 OCaml/runtime restamping is an experimental compatibility route, **not verified full iOS cross-compilation or ABI acceptance**. Real target C/Swift and signing checks do not establish full OCaml ABI or startup acceptance. Simulator signing is adhoc. macOS development signing needs `--sign-identity` plus `JOURNAL_MACOS_TEAM_ID`; device mode requires the existing signing inputs shown below. diff --git a/.agents/skills/journal-build/scripts/journal_build.py b/.agents/skills/journal-build/scripts/journal_build.py index 245fd47f..747a2881 100644 --- a/.agents/skills/journal-build/scripts/journal_build.py +++ b/.agents/skills/journal-build/scripts/journal_build.py @@ -125,6 +125,24 @@ def __init__(self, args, run_dir): "commands": [], "mode": mode, "cross_abi_validated": False, "runtime_acceptance": False, "installed": False, "ignored_environment": ignored} + def native_entry(self): + if getattr(self.args, "sync_network_probe", False): + return {"role": "sync-network-probe", "dune_target": "tool/sync_acceptance_probe/native_probe.exe.o"} + return {"role": "production", "dune_target": "app/native_embed.exe.o"} + + def native_sources(self, source): + return {name: digest for name, digest in source.items() + if Path(name).suffix in {".ml", ".mli", ".c", ".h"} + and (not {"tool", "test", "apple-tests"}.intersection(Path(name).parts) + or (getattr(self.args, "sync_network_probe", False) + and name.startswith("tool/sync_acceptance_probe/")))} + + def native_configuration(self, source): + return {name: digest for name, digest in source.items() + if Path(name).name in {"dune", "dune-project", "dune-workspace"} + or name.endswith((".opam", ".opam.locked")) + or name == ".agents/skills/journal-build/scripts/native_static_link_flags.sh"} + def save(self): (self.run_dir / "manifest.json").write_text(json.dumps(self.report, indent=2) + "\n") @@ -394,12 +412,42 @@ def ordinary_dependency_files(self, name, dependency): def imported_native(self, source): """Read producer records as data; never execute their recorded commands.""" envelope = json.loads(self.args.native_manifest.read_text()) + entry = self.native_entry() + if envelope.get("native_entry", {"role": "production", "dune_target": "app/native_embed.exe.o"}) != entry: + raise ValueError("native entry role/target differs from explicit build mode") version = envelope.get("schema_version") if type(version) is not int or version not in (1, 2): raise ValueError("unsupported native manifest schema_version") self.native_schema_version = version directory = self.args.native_manifest.resolve().parent pre, link = [read_evidence(envelope[name], directory) for name in ("source", "link")] + expected_build = {"configuration": self.args.configuration, + "dune_profile": "dev" if self.args.configuration == "debug" else "release"} + if any(record.get("native_build") is None for record in (pre, link)): + raise ValueError("native configuration/profile proof missing; rerun native producer with requested configuration") + if any(record["native_build"] != expected_build for record in (pre, link)): + raise ValueError("producer configuration/profile differs from requested build; rerun native producer") + if "dune_rule" in envelope: + rule = read_evidence(envelope["dune_rule"], directory) + query = rule.get("query", []) + if (rule.get("native_build") != expected_build + or query[:4] != ["exec", "--", "dune", "rules"] + or query.count("--profile") != 1 + or query[-4:] != ["--profile", expected_build["dune_profile"], "--format=json", entry["dune_target"]]): + raise ValueError("Dune configuration/profile proof differs or is missing; rerun native producer") + if getattr(self.args, "sync_network_probe", False): + if version != 2 or any(record.get("native_entry") != entry for record in (pre, link)): + raise ValueError("probe native entry role/target is not hash-bound by producer records") + if pre.get("native_configuration_sha256") != self.native_configuration(source): + raise ValueError("probe native build configuration differs from current inputs") + rule = read_evidence(envelope["dune_rule"], directory) + if (rule.get("native_entry") != entry + or not rule["original_object"]["path"].endswith("/" + entry["dune_target"]) + or rule["command"].count("-o") != 1 + or rule["command"][rule["command"].index("-o") + 1] != entry["dune_target"]): + raise ValueError("probe Dune target differs from explicit native entry") + elif any(record.get("native_entry", entry) != entry for record in (pre, link)): + raise ValueError("native entry role/target differs from producer records") xmlm = read_evidence(envelope["xmlm_force_link"], directory) if version == 1 else None if version == 2 and "xmlm_force_link" in envelope: raise ValueError("ordinary-archive schema must not reinterpret a legacy XMLm force-link proof") @@ -408,7 +456,7 @@ def imported_native(self, source): # Build/tool/docs commits may advance HEAD; native bytes/configuration below must still match exactly. self.run("git", "merge-base", "--is-ancestor", pre["commit"], self.report["source_sha"]) removed = self.run("git", "diff", "--name-only", "--diff-filter=D", pre["commit"], "--").splitlines() - for name in sorted(set(source) | set(removed)): + for name in ([] if getattr(self.args, "sync_network_probe", False) else sorted(set(source) | set(removed))): if (Path(name).name in {"dune", "dune-project", "dune-workspace"} or name.endswith((".opam", ".opam.locked")) or name == ".agents/skills/journal-build/scripts/native_static_link_flags.sh"): original = subprocess.check_output(["git", "show", pre["commit"] + ":" + name], cwd=ROOT) @@ -421,10 +469,8 @@ def imported_native(self, source): raise ValueError(f"producer {package} differs from locked commit") native_sources = dict(pre["native_source_sha256"]) for name, digest in link["device_c_source_sha256"].items(): - native_sources["logseq_sync/lib/effect_runner/platform/" + name] = digest - required = {name: digest for name, digest in source.items() - if Path(name).suffix in {".ml", ".mli", ".c", ".h"} - and not {"tool", "test", "apple-tests"}.intersection(Path(name).parts)} + native_sources["logseq_sync/lib/native_crypto/" + name] = digest + required = self.native_sources(source) mismatch = [name for name in sorted(set(required) | set(native_sources)) if native_sources.get(name) != required.get(name)] if mismatch: @@ -467,6 +513,8 @@ def imported_native(self, source): raise ValueError("host object still provides DNS/crypto; target C must provide them") self.host_object, self.native_files = host, inputs self.report["native_source_sha"] = pre["commit"] + if getattr(self.args, "sync_network_probe", False): + self.report["production_code_source_validated"] = True self.report["abi_limit"] = ("Host ARM64 producer only; no Apple App or iOS ABI validation" if getattr(self.args, "producer_only", False) else "EXPERIMENTAL: host ARM64 OCaml/runtime restamped; full iOS runtime ABI is not validated") @@ -482,6 +530,11 @@ def validated_native(self, envelope, directory, pre, link): """Reuse a fixed complete object, with a hash-bound prior validation receipt.""" receipt = read_evidence(envelope["validated_build"], directory) previous = receipt["inputs"]["dependencies"] + if getattr(self.args, "sync_network_probe", False) and ( + receipt.get("native_entry") != self.native_entry() + or receipt.get("production_acceptance") is not False + or receipt["inputs"]["flags"].get("native_entry") != self.native_entry()): + raise ValueError("validated receipt does not prove the explicit probe role") version = envelope["schema_version"] if (receipt["status"] != "built" or receipt["mode"] != "experimental-host-restamp" or previous.get("source_sha", receipt["source_sha"]) != pre["commit"] @@ -568,11 +621,11 @@ def add(values): add(self.forced_xmlm_files(compiler, link, xmlm)) # The producer removed only host DNS/crypto from a local archive copy. command = self.ordinary_command if version == 2 else link["command"] - effect = next(Path(p).parent for p in command if p.endswith("/logseq_sync_effect_runner_impl.cmxa")) - for name in ("logseq_sync_effect_runner_impl.cmxa", "logseq_sync_effect_runner_impl.a"): + effect = next(Path(p).parent for p in command if p.endswith("/logseq_sync_native_crypto.cmxa")) + for name in ("logseq_sync_native_crypto.cmxa", "logseq_sync_native_crypto.a"): original_path = next(p for p in inputs if Path(p).name == name) add({str(effect / name): inputs[original_path]}) - patched = effect / "liblogseq_sync_effect_runner_impl_stubs.a" + patched = effect / "liblogseq_sync_native_crypto_stubs.a" add({str(patched): link["patched_host_stub_archive_sha256"]}) if "platform_crypto_stubs.o" in self.run("ar", "-t", patched): raise ValueError("producer host DNS/crypto C member was not excluded") @@ -638,6 +691,12 @@ def preflight(self): if probe and (not self.info["CFBundleIdentifier"].endswith("-probe") or any(path == ROOT or ROOT in path.parents for path in (self.swift_dir, self.plist))): raise RuntimeError("probe mode requires external staged Swift/plist and a distinct -probe bundle ID") + if getattr(a, "sync_network_probe", False): + if self.swift_dir != ROOT / "swift" or self.plist != ROOT / "apple/Info.plist": + raise RuntimeError("sync-network-probe requires maintained real Swift App sources and Info.plist") + self.info.update(CFBundleIdentifier="com.logseq.journal.sync-probe", + CFBundleName="Journal Sync Probe", CFBundleDisplayName="Journal Sync Probe", + JournalSyncNetworkProbe=True) if a.version: self.info["CFBundleShortVersionString"] = a.version if a.build_number: @@ -686,15 +745,20 @@ def preflight(self): "mode": "native" if getattr(a, "producer_only", False) else self.report["mode"], "info": self.info, "sign_identity": a.sign_identity}} self.inputs["flags"]["teams"] = {name: self.env.get(name) for name in ("JOURNAL_IOS_TEAM_ID", "JOURNAL_MACOS_TEAM_ID")} + if getattr(a, "sync_network_probe", False): + self.inputs["flags"]["native_entry"] = self.native_entry() + self.report.update(native_entry=self.native_entry(), production_acceptance=False, + production_code_source_validated=bool(a.native_manifest)) self.report.update(inputs=self.inputs, target=triple, sdk_name=sdk_name, architecture="arm64", cache_key=fingerprint(self.inputs)) self.save() def native_link_rule(self, workspace): """Read the current rule even when Dune's incremental build ran no linker.""" - rules = json.loads(self.opam("exec", "--", "dune", "rules", "--root", ROOT, "--no-config", + query = list(map(str, ("exec", "--", "dune", "rules", "--root", ROOT, "--no-config", "--cache=disabled", "--build-dir", workspace / "dune", "--profile", - "dev" if self.args.configuration == "debug" else "release", "--format=json", "app/native_embed.exe.o")) + "dev" if self.args.configuration == "debug" else "release", "--format=json", self.native_entry()["dune_target"]))) + rules = json.loads(self.opam(*query)) if len(rules) != 1: raise ValueError("producer needs exactly one native Dune link rule") action, cwd = rules[0]["action"], ROOT @@ -710,6 +774,7 @@ def native_link_rule(self, workspace): compiler = self.prefix / "bin/ocamlopt.opt" if command[0] != str(compiler) or sha(compiler) != self.inputs["toolchain"]["compiler_sha256"]: raise ValueError("Dune producer compiler identity differs from preflight") + self.native_rule_query = query return cwd, command def producer_pin(self, name): @@ -751,10 +816,16 @@ def produce_native(self, native, workspace): raise ValueError("original native object hash changed before production") self.verify_target(native, "MACOS") cwd, original = self.native_link_rule(workspace) + self.selected_rule = (workspace, self.args.configuration, self.native_entry(), (cwd, original)) + native_build = {"configuration": self.args.configuration, + "dune_profile": self.native_rule_query[self.native_rule_query.index("--profile") + 1]} directory = workspace / "native-producer" envelope_path = directory / "native-input.json" - rule = {"working_directory": str(cwd), "command": original, + rule = {"working_directory": str(cwd), "command": original, "native_build": native_build, + "query": self.native_rule_query, "original_object": {"path": str(native), "sha256": sha(native)}} + if getattr(self.args, "sync_network_probe", False): + rule["native_entry"] = self.native_entry() reused = directory.exists() if reused: if not envelope_path.is_file() or json.loads((directory / "dune-rule.json").read_text()) != rule: @@ -768,9 +839,9 @@ def dump(name, value): return {"path": str(path), "sha256": sha(path)} def absolute(arg): return Path(os.path.abspath(cwd / arg)) - effects = [absolute(arg) for arg in original if arg.endswith("/logseq_sync_effect_runner_impl.cmxa")] + effects = [absolute(arg) for arg in original if arg.endswith("/logseq_sync_native_crypto.cmxa")] if len(effects) != 1: - raise ValueError("producer needs one original effect-runner archive") + raise ValueError("producer needs one original native-crypto archive") effect = effects[0].parent copied = directory / "host-stubs" copied.mkdir() @@ -778,17 +849,24 @@ def absolute(arg): def record(path): path = absolute(path) self.bind_native_inputs(inputs, {str(path): sha(path)}) - for name in ("logseq_sync_effect_runner_impl.cmxa", "logseq_sync_effect_runner_impl.a"): + for name in ("logseq_sync_native_crypto.cmxa", "logseq_sync_native_crypto.a"): record(effect / name) shutil.copyfile(effect / name, copied / name) - stub_name = "liblogseq_sync_effect_runner_impl_stubs.a" + stub_name = "liblogseq_sync_native_crypto_stubs.a" patched = copied / stub_name + record(effect / stub_name) shutil.copyfile(effect / stub_name, patched) if self.run("ar", "-t", patched).splitlines().count("platform_crypto_stubs.o") != 1: raise ValueError("producer needs exactly one original host DNS/crypto C member") self.run("ar", "-d", patched, "platform_crypto_stubs.o", env=dict(self.env, ZERO_AR_DATE="1")) - if "platform_crypto_stubs.o" in self.run("ar", "-t", patched).splitlines(): + members = self.run("ar", "-t", patched).splitlines() + if "platform_crypto_stubs.o" in members: raise ValueError("host DNS/crypto member was not removed from the owned archive copy") + indexes = {"__.SYMDEF", "__.SYMDEF SORTED", "__.SYMDEF_64", "__.SYMDEF_64 SORTED"} + if not any(member not in indexes for member in members): + # Apple ar can leave a stale symbol index after deleting the last object. + header = f"{'__.SYMDEF':<16}{0:<12}{0:<6}{0:<6}{'100644':<8}{8:<10}`\n" + patched.write_bytes(b"!\n" + header.encode("ascii") + bytes(8)) command, includes = [original[0]], [] for index, arg in enumerate(original[1:], 1): previous = original[index - 1] @@ -841,9 +919,7 @@ def record(path): pre = {"commit": self.report["source_sha"], "lui": self.lui_sha, "signal": self.producer_pin("ocaml-signal")["resolved_sha"], "datascript": self.producer_pin("datascript-ocaml-native")["resolved_sha"], - "native_source_sha256": {name: digest for name, digest in self.inputs["source"].items() - if Path(name).suffix in {".ml", ".mli", ".c", ".h"} - and not {"tool", "test", "apple-tests"}.intersection(Path(name).parts)}, + "native_source_sha256": self.native_sources(self.inputs["source"]), "installed_lui_source_sha256": {str(p): sha(p) for p in (self.prefix / "lib/lui").rglob("*") if p.is_file() and p.suffix in {".ml", ".mli"}}, "production_native_object": str(directory / "production-host.o")} @@ -852,7 +928,11 @@ def record(path): "compiler_version": self.run(original[0], "-version"), "new_dependency_provenance": dependencies, "patched_host_stub_archive_sha256": sha(patched), "device_c_source_sha256": {p.name: sha(p) for p in - (ROOT / "logseq_sync/lib/effect_runner/platform").glob("platform_crypto_stubs.*")}} + (ROOT / "logseq_sync/lib/native_crypto").iterdir() if p.suffix in {".c", ".h"}}} + pre["native_build"] = link["native_build"] = native_build + pre["native_entry"] = link["native_entry"] = self.native_entry() + if getattr(self.args, "sync_network_probe", False): + pre["native_configuration_sha256"] = self.native_configuration(self.inputs["source"]) # Reject unbound/unknown command forms before executing the current Dune-derived relink. self.ordinary_native_inputs(pre, link) link["command"] = self.ordinary_command @@ -861,7 +941,8 @@ def record(path): if any(sha(path) != digest for path, digest in inputs.items()): raise ValueError("native link input hash changed during production") pre["production_native_object_sha256"] = link["production_host_sha256"] = sha(pre["production_native_object"]) - envelope = {"schema_version": 2, "source": dump("source.json", pre), "link": dump("link.json", link), + envelope = {"schema_version": 2, "native_entry": self.native_entry(), + "source": dump("source.json", pre), "link": dump("link.json", link), "dune_rule": dump("dune-rule.json", rule)} dump("native-input.json", envelope) self.phase = "verify-producer" @@ -897,7 +978,7 @@ def compile(self, workspace): self.target_c = workspace / "platform_crypto.o" self.run(self.tools["clang"], "-target", self.triple, "-isysroot", self.sdk, "-O2", "-fno-strict-aliasing", "-fwrapv", "-pthread", "-D_FILE_OFFSET_BITS=64", "-Wall", "-g", - "-I", self.native_include, "-c", ROOT / "logseq_sync/lib/effect_runner/platform/platform_crypto_stubs.c", + "-I", self.native_include, "-c", ROOT / "logseq_sync/lib/native_crypto/platform_crypto_stubs.c", "-o", self.target_c) self.verify_target(self.target_c) else: @@ -907,12 +988,14 @@ def compile(self, workspace): self.opam("exec", "--", "dune", "build", "--root", ROOT, "--no-config", "--cache=disabled", "--build-dir", dune_dir, "--trace-file", self.run_dir / "native-trace.json", "--profile", "dev" if a.configuration == "debug" else "release", - "-j", str(a.jobs), "app/native_embed.exe.o") - native = dune_dir / "default/app/native_embed.exe.o" + "-j", str(a.jobs), self.native_entry()["dune_target"]) + native = dune_dir / "default" / self.native_entry()["dune_target"] self.native_gmp = dune_dir / "default/app/libgmp.a" self.report["linked_gmp"] = {"path": str(self.native_gmp), "sha256": sha(self.native_gmp)} native_units = self.verify_ocaml_units(native, require_xmlm=False) self.report["native_object"] = {"path": str(native), "sha256": sha(native), "mode": self.report["mode"]} + if getattr(a, "sync_network_probe", False): + self.report["native_object"]["native_entry"] = self.native_entry() if native_units is not None: self.report["native_object"]["ocaml_units"] = native_units if getattr(a, "producer_only", False): @@ -1083,6 +1166,11 @@ def assemble(self, product, app): def verify_inputs(self): self.phase = "verify-inputs" + if hasattr(self, "selected_rule"): + workspace, configuration, entry, captured = self.selected_rule + if (self.args.configuration != configuration or self.native_entry() != entry + or self.native_link_rule(workspace) != captured): + raise RuntimeError("selected native Dune rule changed during build; rerun with stable inputs") if (source_inputs() != self.inputs["source"] or files(self.swift_dir) != self.inputs["swift"] or files(self.lui) != self.inputs["lui"] or sha(self.plist) != self.inputs["plist"] or files(self.entitlements) != self.inputs["entitlements"]): @@ -1117,6 +1205,7 @@ def arguments(): parser.add_argument("--lui-package", type=Path, help="local LUI platform/apple package") parser.add_argument("--preflight", action="store_true", help="check inputs without compiling") parser.add_argument("--producer-only", action="store_true", help="native macOS producer evidence only; no Swift, App packaging or signing") + parser.add_argument("--sync-network-probe", action="store_true", help="separate real-App Simulator network fault diagnostic; explicit producer/import role, never production acceptance") parser.add_argument("--link-only", action="store_true", help="explicit stub build; not runnable acceptance") parser.add_argument("--experimental-host-restamp", action="store_true", help="explicit existing host ARM64 compatibility route; full iOS ABI unverified") parser.add_argument("--native-manifest", type=Path, help="hash-verified source/link/XMLm producer evidence envelope") @@ -1131,6 +1220,12 @@ def arguments(): parser.add_argument("--provisioning-profile", type=Path, help="existing device profile; never downloaded") parser.add_argument("--sign-entitlements", type=Path, help="existing device signing entitlements") a = parser.parse_args() + if a.sync_network_probe and a.configuration != "debug": + parser.error("sync-network-probe requires --configuration debug") + if a.sync_network_probe and (a.link_only or a.probe_native_object + or not ((a.platform == "macos" and a.producer_only) + or (a.platform == "ios-simulator" and a.experimental_host_restamp and a.native_manifest))): + parser.error("sync-network-probe requires macOS --producer-only or explicit Simulator native-manifest import") if a.producer_only and (a.platform != "macos" or a.link_only or a.native_manifest or a.experimental_host_restamp or a.probe_native_object or a.app_dir or a.app_directory or a.sign_identity != "-" or a.provisioning_profile or a.sign_entitlements): diff --git a/.agents/skills/journal-build/scripts/run_isolated_simulator.py b/.agents/skills/journal-build/scripts/run_isolated_simulator.py index 8f316813..b8a3269b 100644 --- a/.agents/skills/journal-build/scripts/run_isolated_simulator.py +++ b/.agents/skills/journal-build/scripts/run_isolated_simulator.py @@ -3,6 +3,7 @@ import argparse import hashlib import json +import os from pathlib import Path import re import shutil @@ -36,6 +37,8 @@ def main(): parser.add_argument('--runtime', required=True) parser.add_argument('--device-type', required=True) parser.add_argument('--jobs', type=jobs, default=4) + parser.add_argument('--sync-network-probe', action='store_true', + help='separate real-App diagnostic; never ordinary production acceptance') args = parser.parse_args() evidence = args.evidence_dir.expanduser().resolve() if evidence == ROOT or ROOT in evidence.parents: @@ -54,11 +57,14 @@ def main(): def save(): (run / 'session.json').write_text(json.dumps(report, indent=2) + '\n') - def command(label, *argv, cwd=ROOT): + if args.sync_network_probe: + report.update(role='sync-network-probe', production_acceptance=False) + + def command(label, *argv, cwd=ROOT, env=None): argv = list(map(str, argv)) report['commands'].append(dict(label=label, argv=argv)) save() - result = subprocess.run(argv, cwd=cwd, text=True, capture_output=True, + result = subprocess.run(argv, cwd=cwd, env=env, text=True, capture_output=True, timeout=1800 if label in {'producer', 'simulator'} else 300) (run / (label + '.stdout')).write_text(result.stdout) (run / (label + '.stderr')).write_text(result.stderr) @@ -69,7 +75,8 @@ def command(label, *argv, cwd=ROOT): return result def build(label, *options): - result = command(label, 'bash', BUILDER, *options, + role = ['--sync-network-probe'] if args.sync_network_probe else [] + result = command(label, 'bash', BUILDER, *options, *role, '--lui-package', args.lui_package.resolve(), '--cache-dir', evidence / 'build cache', '--jobs', args.jobs) matches = re.findall(r'^manifest: (.+)$', result.stderr, re.MULTILINE) @@ -77,6 +84,10 @@ def build(label, *options): raise RuntimeError(f'{label}: missing unique current manifest') path = Path(matches[0]) manifest = json.loads(path.read_text()) + if args.sync_network_probe and ( + manifest.get('native_entry', {}).get('role') != 'sync-network-probe' + or manifest.get('production_acceptance') is not False): + raise RuntimeError(f'{label}: missing explicit probe role') report[label] = dict(manifest=str(path), sha256=hashlib.sha256(path.read_bytes()).hexdigest()) return manifest @@ -100,6 +111,10 @@ def build(label, *options): app = Path(target['artifact']['app']) report['app'] = str(app) report['bundle_id'] = target['artifact']['info']['CFBundleIdentifier'] + if args.sync_network_probe and ( + report['bundle_id'] != 'com.logseq.journal.sync-probe' + or target['artifact']['info'].get('JournalSyncNetworkProbe') is not True): + raise RuntimeError('Simulator artifact is not the separate sync probe') for name, digest in target['artifact']['files'].items(): if hashlib.sha256((app / name).read_bytes()).hexdigest() != digest: raise RuntimeError(f'artifact changed after build: {name}') @@ -119,7 +134,16 @@ def build(label, *options): command('bootstatus', 'xcrun', 'simctl', 'bootstatus', udid, '-b') verify_manifest_sources(target, source_inputs()) command('install', 'xcrun', 'simctl', 'install', udid, app) - command('launch', 'xcrun', 'simctl', 'launch', udid, report['bundle_id']) + launch_env = None + if args.sync_network_probe: + container = Path(command('data-container', 'xcrun', 'simctl', 'get_app_container', + udid, report['bundle_id'], 'data').stdout.strip()).resolve() + control = Path(tempfile.mkdtemp(prefix='sync-network-probe-', dir=container / 'tmp')).resolve() + control.chmod(0o700) + report['network_probe'] = dict(directory=str(control), status=str(control / 'status.json'), + offline_signal=str(control / 'offline')) + launch_env = dict(os.environ, SIMCTL_CHILD_JOURNAL_SYNC_NETWORK_PROBE_DIRECTORY=str(control)) + command('launch', 'xcrun', 'simctl', 'launch', udid, report['bundle_id'], env=launch_env) command('container', 'xcrun', 'simctl', 'get_app_container', udid, report['bundle_id'], 'app') command('screenshot', 'xcrun', 'simctl', 'io', udid, 'screenshot', run / 'launch.png') report['status'] = 'launched' diff --git a/.agents/skills/journal-build/scripts/test_journal_build.py b/.agents/skills/journal-build/scripts/test_journal_build.py index 87ab4244..51a896f5 100644 --- a/.agents/skills/journal-build/scripts/test_journal_build.py +++ b/.agents/skills/journal-build/scripts/test_journal_build.py @@ -60,6 +60,13 @@ def cli(self, *args, env=None): return subprocess.run(["bash", str(SCRIPT), *args], text=True, capture_output=True, env=env) + def test_release_network_probe_is_rejected(self): + for args in [('macos', '--producer-only', '--switch', 'unused'), + ('ios-simulator', '--experimental-host-restamp', '--native-manifest', '/unused')]: + result = self.cli(*args, '--sync-network-probe', '--configuration', 'release') + self.assertNotEqual(result.returncode, 0) + self.assertIn('debug', result.stderr) + def test_help_is_read_only_and_describes_modes(self): result = self.cli("--help") self.assertEqual(result.returncode, 0, result.stderr) @@ -115,6 +122,17 @@ def test_probe_object_cannot_select_device_or_production_modes(self): self.assertNotEqual(result.returncode, 0) self.assertIn('probe mode', result.stderr) + def test_sync_network_probe_only_accepts_producer_or_simulator_import(self): + for args in [('macos', '--switch', 'unused'), + ('macos', '--link-only'), + ('ios-simulator', '--link-only'), + ('ios-device', '--experimental-host-restamp', '--native-manifest', '/tmp/input.json'), + ('ios-simulator', '--probe-native-object', '/tmp/probe.o')]: + with self.subTest(args=args): + result = self.cli(*args, '--sync-network-probe', '--preflight') + self.assertNotEqual(result.returncode, 0) + self.assertIn('sync-network-probe requires', result.stderr) + def test_foreign_dune_root_and_runtime_cannot_change_provenance(self): for name in ("DUNE_ROOT", "DUNE_WORKSPACE", "DUNE_CROSS_TARGET", "DUNE_INSTRUMENT_WITH", "OCAMLLIB", "MACOSX_DEPLOYMENT_TARGET", "CAML_LD_LIBRARY_PATH", "OCAMLFIND_COMMANDS"): @@ -781,7 +799,7 @@ def producer_fixture(self): check=True, capture_output=True) repo = root / "repo" (repo / "app").mkdir(parents=True) - platform = repo / "logseq_sync/lib/effect_runner/platform" + platform = repo / "logseq_sync/lib/native_crypto" platform.mkdir(parents=True) (repo / "app/unit.ml").write_text("let value = 1\n") for name in ("platform_crypto_stubs.c", "platform_crypto_stubs.h"): @@ -803,17 +821,19 @@ def producer_fixture(self): history.write_text("original archive") forced = root / "xmlm-force/xmlm.cmxa" cmx = root / "xmlm-force/xmlm.cmx" - effect = root / "effect/logseq_sync_effect_runner_impl.cmxa" + effect = root / "effect/logseq_sync_native_crypto.cmxa" dummy = "a" * 64 inputs = {str(history): self.build.sha(history), str(compiler): dummy, str(forced): dummy, str(forced.with_suffix('.a')): dummy, str(effect): dummy, str(effect.with_suffix('.a')): dummy, str(native): host_hash, str(header): self.build.sha(header)} - pre = dict(commit=commit, lui=pins['lui'], signal=pins['signal'], datascript=pins['datascript'], + pre = dict(native_build=dict(configuration='release', dune_profile='release'), + commit=commit, lui=pins['lui'], signal=pins['signal'], datascript=pins['datascript'], production_native_object=str(native), production_native_object_sha256=host_hash, native_source_sha256={k: v for k, v in source.items() if k.endswith('.ml')}, installed_lui_source_sha256={}) - link = dict(command=[str(compiler), str(effect), str(forced)], compiler_sha256=dummy, + link = dict(native_build=dict(configuration='release', dune_profile='release'), + command=[str(compiler), str(effect), str(forced)], compiler_sha256=dummy, compiler_version="5.3.0", production_host_sha256=host_hash, direct_native_input_sha256={k: v for k, v in inputs.items() if k not in (str(native), str(header))}, device_c_source_sha256={p.name: self.build.sha(p) for p in platform.iterdir()}, @@ -835,7 +855,7 @@ def producer_fixture(self): archive = Path(link['new_dependency_provenance'][name]['installed_archive']) inputs.update({str(archive): dummy, str(archive.with_suffix('.a')): dummy}) inputs.update({str(cmx): dummy, str(package / 'lib/xmlm/xmlm.cmx'): dummy, - str(effect.parent / 'liblogseq_sync_effect_runner_impl_stubs.a'): dummy}) + str(effect.parent / 'liblogseq_sync_native_crypto_stubs.a'): dummy}) receipt = dict(status='built', mode='experimental-host-restamp', source_sha=commit, production_link=dict(runtime_mldoc_xmlm_retained=True, cross_abi_validated=False), inputs=dict(source=source, dependencies=dict(producer=dict(envelope), files=inputs, @@ -846,12 +866,42 @@ def producer_fixture(self): path = root / 'input.json' path.write_text(json.dumps(envelope)) build = self.build.Build(SimpleNamespace(native_manifest=path, experimental_host_restamp=True, - link_only=False, platform='ios-device'), root) + link_only=False, platform='ios-device', configuration='release'), root) build.run = partial(build.run, cwd=repo) build.report['source_sha'], build.lui_sha = commit, pins['lui'] yield SimpleNamespace(build=build, source=source, history=history, native=native, header=header, envelope=envelope, path=path, receipt=receipt, receipt_path=receipt_path, repo=repo) + def test_native_import_rejects_cross_role_envelopes_before_consuming_inputs(self): + with self.producer_fixture() as f: + f.envelope['native_entry'] = dict(role='sync-network-probe', + dune_target='tool/sync_acceptance_probe/native_probe.exe.o') + f.path.write_text(json.dumps(f.envelope)) + with self.assertRaisesRegex(ValueError, 'native entry role'): + f.build.imported_native(f.source) + f.envelope.pop('native_entry') + f.path.write_text(json.dumps(f.envelope)) + f.build.args.sync_network_probe = True + with self.assertRaisesRegex(ValueError, 'native entry role'): + f.build.imported_native(f.source) + + def test_probe_native_identity_and_source_set_are_distinct(self): + with self.producer_fixture() as f: + probe = f.repo / 'tool/sync_acceptance_probe' + probe.mkdir(parents=True) + (probe / 'native_probe.ml').write_text('let probe = true\n') + (probe / 'test_net.ml').write_text('let network = true\n') + (probe / 'dune').write_text('(executable (name native_probe))\n') + source = self.build.source_inputs() + ordinary = f.build.native_entry() + self.assertEqual(ordinary, dict(role='production', dune_target='app/native_embed.exe.o')) + self.assertNotIn('tool/sync_acceptance_probe/test_net.ml', f.build.native_sources(source)) + f.build.args.sync_network_probe = True + self.assertNotEqual(ordinary, f.build.native_entry()) + self.assertEqual(f.build.native_entry()['dune_target'], 'tool/sync_acceptance_probe/native_probe.exe.o') + self.assertIn('tool/sync_acceptance_probe/test_net.ml', f.build.native_sources(source)) + self.assertIn('tool/sync_acceptance_probe/dune', f.build.native_configuration(source)) + def test_consumed_archives_may_change_or_disappear_with_fixed_object_receipt(self): with self.producer_fixture() as f: f.history.write_text("later unrelated compilation") @@ -868,7 +918,7 @@ def ordinary_producer_fixture(self, *, entry=None, relative=False, warnings=Fals root = Path(directory).resolve() repo, package, effect = root / 'repo', root / 'package', root / 'effect' (repo / 'app').mkdir(parents=True) - platform = repo / 'logseq_sync/lib/effect_runner/platform' + platform = repo / 'logseq_sync/lib/native_crypto' platform.mkdir(parents=True) for name in ('platform_crypto_stubs.c', 'platform_crypto_stubs.h'): (platform / name).write_text('/* target source */\n') @@ -889,11 +939,11 @@ def ordinary_producer_fixture(self, *, entry=None, relative=False, warnings=Fals def run(*argv, cwd=root): subprocess.run(list(map(str, argv)), cwd=cwd, check=True, capture_output=True) run(compiler, '-c', effect / 'effect.ml') - effect_archive = effect / 'logseq_sync_effect_runner_impl.cmxa' + effect_archive = effect / 'logseq_sync_native_crypto.cmxa' run(compiler, '-a', '-o', effect_archive, effect / 'effect.cmx') (effect / 'stub.c').write_text('int harmless_stub;\n') run('xcrun', 'clang', '-c', effect / 'stub.c', '-o', effect / 'stub.o') - stub = effect / 'liblogseq_sync_effect_runner_impl_stubs.a' + stub = effect / 'liblogseq_sync_native_crypto_stubs.a' run('ar', 'rcs', stub, effect / 'stub.o') provenance = {} for name, source in [('mldoc', 'let parse text = String.length text\n'), @@ -944,7 +994,8 @@ def run(*argv, cwd=root): commit = subprocess.check_output(['git', '-C', str(repo), 'rev-parse', 'HEAD'], text=True).strip() with patch.object(self.build, 'ROOT', repo): source = self.build.source_inputs() - pre = dict(commit=commit, **pins, production_native_object=str(native), + pre = dict(native_build=dict(configuration='release', dune_profile='release'), + commit=commit, **pins, production_native_object=str(native), production_native_object_sha256=self.build.sha(native), native_source_sha256={'app/unit.ml': self.build.sha(repo / 'app/unit.ml')}, installed_lui_source_sha256={}) @@ -953,7 +1004,8 @@ def run(*argv, cwd=root): for item in provenance.values(): archive = Path(item['installed_archive']) inputs.extend([archive, archive.with_suffix('.a')]) - link = dict(command=command, working_directory=str(root), compiler_sha256=self.build.sha(compiler), + link = dict(native_build=dict(configuration='release', dune_profile='release'), + command=command, working_directory=str(root), compiler_sha256=self.build.sha(compiler), compiler_version=subprocess.check_output([compiler, '-version'], text=True).strip(), production_host_sha256=self.build.sha(native), direct_native_input_sha256={str(p): self.build.sha(p) for p in inputs}, @@ -969,14 +1021,14 @@ def save(): path.write_text(json.dumps(envelope)) save() build = self.build.Build(SimpleNamespace(native_manifest=path, experimental_host_restamp=True, - link_only=False, platform='ios-simulator'), root) + link_only=False, platform='ios-simulator', configuration='release'), root) build.run = partial(build.run, cwd=repo) build.report['source_sha'], build.lui_sha = commit, pins['lui'] yield SimpleNamespace(build=build, source=source, link=link, pre=pre, native=native, envelope=envelope, path=path, root=root, save=save, runtime=runtime, provenance=provenance) @contextmanager - def native_dune_producer(self): + def native_dune_producer(self, single_member=False, sync_probe=False): """A real Dune complete-object rule, including removable host C providers.""" with self.ordinary_producer_fixture() as f: repo = self.build.ROOT @@ -991,7 +1043,7 @@ def native_dune_producer(self): # Mirror opam exec's selected compiler PATH; omit ambient OCaml fallback even locally. compiler_bin = dune.parent if switch else original_compiler.parent f.build.env['PATH'] = str(compiler_bin) + os.pathsep + os.defpath - effect = repo / 'logseq_sync/lib/effect_runner' + effect = repo / 'logseq_sync/lib/native_crypto' effect.mkdir(exist_ok=True) original = f.root / 'effect' for path in original.glob('*'): @@ -1002,24 +1054,36 @@ def native_dune_producer(self): 'int logseq_journal_crypto_call(void) { return 1; }\n') obj = c.with_suffix('.o') subprocess.run(['xcrun', 'clang', '-c', c, '-o', obj], check=True, capture_output=True) - stub = effect / 'liblogseq_sync_effect_runner_impl_stubs.a' + stub = effect / 'liblogseq_sync_native_crypto_stubs.a' + if single_member: + stub.unlink() subprocess.run(['ar', 'rcs', stub, obj], check=True, capture_output=True) - archive = effect / 'logseq_sync_effect_runner_impl.cmxa' + archive = effect / 'logseq_sync_native_crypto.cmxa' subprocess.run([compiler, '-a', '-o', archive, effect / 'effect.cmx', - '-cclib', '-llogseq_sync_effect_runner_impl_stubs'], check=True, capture_output=True) + '-cclib', '-llogseq_sync_native_crypto_stubs'], check=True, capture_output=True) # The fixture rule is opaque to the builder: it must read actual Dune argv. - command = [str(compiler), '-output-complete-obj', '-w', '@1..3', '-o', 'app/native_embed.exe.o', - '-cclib', '-Llogseq_sync/lib/effect_runner', '-cclib', 'app/libgmp.a', - '-I', 'logseq_sync/lib/effect_runner', 'logseq_sync/lib/effect_runner/' + archive.name, + target = 'tool/sync_acceptance_probe/native_probe.exe.o' if sync_probe else 'app/native_embed.exe.o' + rule_directory = repo / Path(target).parent + rule_directory.mkdir(parents=True, exist_ok=True) + relative_root = os.path.relpath(repo, rule_directory) + if sync_probe: + (rule_directory / 'native_probe.ml').write_text('let probe = true\n') + (rule_directory / 'test_net.ml').write_text('let network = true\n') + command = [str(compiler), '-output-complete-obj', '-w', '@1..3', '-o', target, + '-cclib', '-Llogseq_sync/lib/native_crypto', '-cclib', 'app/libgmp.a', + '-I', 'logseq_sync/lib/native_crypto', 'logseq_sync/lib/native_crypto/' + archive.name, f.provenance['xmlm']['installed_archive'], f.provenance['mldoc']['installed_archive'], 'app/unit.cmx'] def atom(value): return json.dumps(str(value)) (repo / 'dune-project').write_text('(lang dune 3.0)\n') - (repo / 'app/dune').write_text('(rule (target native_embed.exe.o) (deps libgmp.a unit.cmx unit.o ' - + ' '.join(atom('../logseq_sync/lib/effect_runner/' + path.name) - for path in effect.iterdir() if path.suffix in ('.cmxa', '.a')) - + ') (action (chdir .. (run ' + ' '.join(map(atom, command)) + '))))\n') + dependencies = ['app/libgmp.a', 'app/unit.cmx', 'app/unit.o'] + [ + 'logseq_sync/lib/native_crypto/' + path.name + for path in effect.iterdir() if path.suffix in ('.cmxa', '.a')] + (rule_directory / 'dune').write_text('(rule (target ' + Path(target).name + ') (deps ' + + ' '.join(atom(relative_root + '/' + name) for name in dependencies) + + ') (action (chdir ' + atom(relative_root) + ' (run ' + + ' '.join(map(atom, command)) + '))))\n') shutil.copyfile(stub, repo / 'app/libgmp.a') workspace = f.root / "workspace with spaces 'and quotes'" workspace.mkdir() @@ -1030,7 +1094,7 @@ def opam(*argv): f.build.opam = opam f.build.args = SimpleNamespace(platform='macos', configuration='release', jobs=1, producer_only=True, link_only=False, experimental_host_restamp=False, - native_manifest=None, probe_native_object=None) + native_manifest=None, probe_native_object=None, sync_network_probe=sync_probe) f.build.report['mode'] = 'native-producer' f.build.tools = {'swift': '/unexpected-swift-command'} f.build.prefix = f.root / 'package' @@ -1049,11 +1113,116 @@ def opam(*argv): ('datascript-ocaml-native', f.pre['datascript'])]})) f.build.opam('exec', '--', 'dune', 'build', '--root', repo, '--no-config', '--cache=disabled', '--build-dir', dune_dir, '--profile', 'release', - '-j', '1', 'app/native_embed.exe.o') - native = dune_dir / 'default/app/native_embed.exe.o' + '-j', '1', target) + native = dune_dir / 'default' / target f.build.report['native_object'] = dict(path=str(native), sha256=self.build.sha(native), mode='native-producer') yield SimpleNamespace(**vars(f), workspace=workspace, original_native=native, compiler=compiler) + def test_producer_import_requires_matching_configuration_and_profile(self): + with self.native_dune_producer() as f: + envelope = f.build.produce_native(f.original_native, f.workspace) + f.build.args.native_manifest = envelope + f.build.args.configuration = 'debug' + with self.assertRaisesRegex(ValueError, 'configuration/profile'): + f.build.imported_native(f.build.inputs['source']) + f.build.args.configuration = 'release' + records = json.loads(envelope.read_text()) + rule_path = Path(records['dune_rule']['path']) + original_rule = json.loads(rule_path.read_text()) + for profile in ('dev', None): + rule = dict(original_rule) + rule['query'] = list(rule.get('query', ['--profile', 'release'])) + rule['query'][rule['query'].index('--profile') + 1] = profile + rule_path.write_text(json.dumps(rule)) + records['dune_rule']['sha256'] = self.build.sha(rule_path) + envelope.write_text(json.dumps(records)) + with self.assertRaisesRegex(ValueError, 'configuration/profile'): + f.build.imported_native(f.build.inputs['source']) + rule_path.write_text(json.dumps(original_rule)) + records['dune_rule']['sha256'] = self.build.sha(rule_path) + for name in ('source', 'link'): + path = Path(records[name]['path']) + value = json.loads(path.read_text()) + value.pop('native_build', None) + path.write_text(json.dumps(value)) + records[name]['sha256'] = self.build.sha(path) + envelope.write_text(json.dumps(records)) + with self.assertRaisesRegex(ValueError, 'rerun.*producer'): + f.build.imported_native(f.build.inputs['source']) + + def test_final_verification_requeries_selected_rule(self): + with self.native_dune_producer() as f: + f.build.produce_native(f.original_native, f.workspace) + f.build.swift_dir = f.root / 'swift' + f.build.lui = f.root / 'lui' + f.build.entitlements = f.root / 'entitlements' + f.build.plist = f.root / 'Info.plist' + f.build.plist.write_bytes(b'plist') + f.build.inputs.update(swift={}, lui={}, entitlements={}, plist=self.build.sha(f.build.plist), signing={}) + f.build.inputs['toolchain'].update(hashes={}, sdk_identity={'settings': {}}) + f.build.tools = {} + f.build.args.native_manifest = Path('already-imported') + rule_path = self.build.ROOT / 'app/dune' + original = rule_path.read_text() + changed = original.replace('"@1..3"', '"@1..3" "-g"') + self.assertNotEqual(original, changed) + rule_path.write_text(changed) + with self.assertRaisesRegex(RuntimeError, 'Dune.*rule changed'): + f.build.verify_inputs() + + def test_probe_producer_binds_target_sources_configuration_and_warm_cache(self): + with self.native_dune_producer(sync_probe=True) as f: + envelope_path = f.build.produce_native(f.original_native, f.workspace) + envelope = json.loads(envelope_path.read_text()) + entry = dict(role='sync-network-probe', dune_target='tool/sync_acceptance_probe/native_probe.exe.o') + self.assertEqual(envelope['native_entry'], entry) + pre = self.build.read_evidence(envelope['source'], envelope_path.parent) + self.assertEqual(pre['native_entry'], entry) + self.assertIn('tool/sync_acceptance_probe/test_net.ml', pre['native_source_sha256']) + self.assertIn('tool/sync_acceptance_probe/dune', pre['native_configuration_sha256']) + self.assertEqual(f.build.produce_native(f.original_native, f.workspace), envelope_path) + self.assertFalse(f.build.report['producer']['link_executed']) + f.build.args.native_manifest = envelope_path + f.build.args.sync_network_probe = False + with self.assertRaisesRegex(ValueError, 'native entry role'): + f.build.imported_native(f.build.inputs['source']) + f.build.args.sync_network_probe = True + changed = dict(f.build.inputs['source']) + changed['tool/sync_acceptance_probe/dune'] = '0' * 64 + with self.assertRaisesRegex(ValueError, 'build configuration'): + f.build.imported_native(changed) + changed = dict(f.build.inputs['source']) + changed['tool/sync_acceptance_probe/test_net.ml'] = '0' * 64 + with self.assertRaisesRegex(ValueError, 'native sources'): + f.build.imported_native(changed) + + def test_producer_links_after_stripping_single_member_stub_archive(self): + with self.native_dune_producer(single_member=True) as f: + original = self.build.ROOT / 'logseq_sync/lib/native_crypto/liblogseq_sync_native_crypto_stubs.a' + original_digest = self.build.sha(original) + members = subprocess.check_output(['ar', '-t', original], text=True).splitlines() + self.assertEqual([name for name in members if not name.startswith('__.SYMDEF')], + ['platform_crypto_stubs.o']) + envelope_path = f.build.produce_native(f.original_native, f.workspace) + envelope = json.loads(envelope_path.read_text()) + link = self.build.read_evidence(envelope['link'], envelope_path.parent) + staged = envelope_path.parent / 'host-stubs' / original.name + self.assertEqual(subprocess.check_output(['ar', '-t', staged], text=True).splitlines(), + ['__.SYMDEF']) + self.assertEqual(staged.stat().st_size, 76) + self.assertEqual(self.build.sha(original), original_digest) + dune_stub = f.original_native.parent.parent / 'logseq_sync/lib/native_crypto' / original.name + self.assertEqual(link['direct_native_input_sha256'][str(dune_stub)], original_digest) + self.assertEqual(link['direct_native_input_sha256'][str(staged)], self.build.sha(staged)) + self.assertEqual(link['patched_host_stub_archive_sha256'], self.build.sha(staged)) + f.build.args.native_manifest = envelope_path + imported = f.build.imported_native(f.build.inputs['source']) + self.assertEqual(imported['ocaml_units'], dict(runtime=True, mldoc=True, xmlm=False)) + symbols = subprocess.check_output(['nm', '-g', envelope_path.parent / 'production-host.o'], text=True) + self.assertNotRegex(symbols, r'(?m)\b[TDS] _logseq_(?:transport_dns_start|journal_crypto_call)$') + self.assertFalse(f.build.report['cross_abi_validated']) + self.assertFalse(f.build.report['runtime_acceptance']) + def test_producer_uses_current_dune_rule_even_after_a_noop_build(self): self.assertTrue(hasattr(self.build.Build, 'produce_native'), 'producer-only entry is missing') with self.native_dune_producer() as f: @@ -1069,6 +1238,8 @@ def test_producer_uses_current_dune_rule_even_after_a_noop_build(self): source = self.build.read_evidence(envelope['source'], envelope_path.parent) link = self.build.read_evidence(envelope['link'], envelope_path.parent) self.assertEqual(link['compiler_sha256'], self.build.sha(f.compiler)) + staged = envelope_path.parent / 'host-stubs/liblogseq_sync_native_crypto_stubs.a' + self.assertIn('stub.o', subprocess.check_output(['ar', '-t', staged], text=True).splitlines()) self.assertEqual(self.build.sha(f.original_native), f.build.report['native_object']['sha256']) self.assertNotEqual(source['production_native_object'], str(f.original_native)) f.build.args.native_manifest = envelope_path @@ -1407,8 +1578,8 @@ def test_fixed_object_source_and_current_c_headers_remain_strict(self): elif case == 'runtime_header': f.header.write_text('changed ABI header') else: - key = {'native_source': 'app/unit.ml', 'target_c': 'logseq_sync/lib/effect_runner/platform/platform_crypto_stubs.c', - 'target_header': 'logseq_sync/lib/effect_runner/platform/platform_crypto_stubs.h'}[case] + key = {'native_source': 'app/unit.ml', 'target_c': 'logseq_sync/lib/native_crypto/platform_crypto_stubs.c', + 'target_header': 'logseq_sync/lib/native_crypto/platform_crypto_stubs.h'}[case] f.source[key] = 'f' * 64 with self.assertRaises((ValueError, RuntimeError)): f.build.imported_native(f.source) diff --git a/.agents/skills/journal-mli-spec/SKILL.md b/.agents/skills/journal-mli-spec/SKILL.md index b7447917..610393cb 100644 --- a/.agents/skills/journal-mli-spec/SKILL.md +++ b/.agents/skills/journal-mli-spec/SKILL.md @@ -1,151 +1,22 @@ --- name: journal-mli-spec -description: Write, change or review Journal OCaml .mli specifications as readable English contracts, starting with TLDR and an immediate usage example. Applies to every .mli under spec/ and other .mli files serving as specification contracts. +description: Write, change or review Journal OCaml .mli interfaces with minimal English comments. Prefer types and names; document only important semantics not inferable from signatures. Applies to all .mli files. --- # Journal MLI Spec -Let readers understand the module's responsibility, core objects, inputs, outputs, usage order and essential lifecycle from the first two sections. Follow the repository's existing contract-change authorization boundaries; this skill does not itself authorize changes to specifications, implementations, Dune files or other interfaces. +Prefer types and clear names over comments in every `.mli` interface, including specifications under `spec/`. Follow existing authorization boundaries: this skill does not authorize contract, implementation, Dune or unrelated interface changes. -## Language +## Minimal comments -Use English throughout the template and the resulting `.mli` declarations, documentation, comments, section headings and explanatory text in examples. Preserve externally defined names and domain data where required; do not translate literals whose exact value is part of a contract. +- Omit comments when declarations already communicate the meaning. Delete redundant explanations, repeated signatures, migration narratives and unnecessary examples. +- Keep only the shortest necessary English rule for important semantics that types cannot express: runtime identity/freshness, ownership transfer and ACKs, cancellation versus physical join, durable outcomes, ordering, matching and budget accounting. Never remove the sole statement of such a contract merely to reduce length. +- Place each rule beside its relevant declaration; state it once and refer to the owning contract rather than duplicating it. Use odoc comments when useful, without mandatory sections. +- No required TLDR, usage example, section order, line target or template. Do not repeat primary signatures in documentation. Add a small example only when it resolves an otherwise unclear usage constraint more briefly than prose. +- Preserve English declarations and comments, externally defined names and contract literals. Do not change declarations, abstraction, type equalities or protocols for documentation layout. Do not invent APIs or bypass sealed boundaries in examples. -## Fixed structure +## Verification -Organize each specification in this order using odoc sections. The first two sections are required. Include sections 3-6 only when applicable; do not add empty sections. +For comment-only edits, prove interface declarations unchanged with normalized OCaml AST comparison (ignoring documentation attributes and locations) or an equally reliable check. Parse and type-check with real dependencies where available; report unavailable checks honestly. Measure actual comment byte/line reduction when simplification is requested, not just formatting changes. Check that essential non-typeable contracts remain and guidance contains no contradictory formatting mandates. -| Order | Section | Contents | -| --- | --- | --- | -| 1 | TLDR | One-sentence responsibility, core types, inputs, outputs, primary entry points and constraints essential to correct use | -| 2 | Minimal usage example | One typical flow, including necessary result handling, state installation or resource cleanup | -| 3 | Construction and validation | Configuration, limits, parameters and constructors for validated values | -| 4 | Capability details | Group related inputs, results, errors and operations by business capability | -| 5 | Lifecycle and recovery | Cancellation, release, retry, persistence, invalidation and stale-event handling | -| 6 | Observation and helpers | State projections, comparison, diagnostics, codecs and other helpers | - -The opening module synopsis belongs to TLDR; do not introduce a separate section before it. For interfaces exporting only types or capabilities, the second section shows how to consume the value or reach its actual owner's execution entry point. Do not add APIs merely to produce an example. - -## TLDR contents and boundaries - -- Prefer actual exported declarations showing a small set of core definitions and primary entry points. Make essential business records, variants and errors directly readable; internal state and controlled handles may remain abstract. Do not hide every input and output behind unexplained abstract types. -- Aim for roughly 30-60 lines; small interfaces can be shorter. This is a reading goal, not a hard limit. Do not compress formatting, remove essential guarantees or add unnecessary declarations to meet a line count. -- State the module's responsibility, ownership of state or resources, whether it performs I/O or persists data, and the normal call order. Include these facts only where relevant. -- Keep facts that affect understanding or correct use in TLDR: success may mean only queue admission, cancellation may not prove rollback, and shutdown may require an actual completion ACK. Put detailed field limits and exhaustive failure branches in later sections. -- Do not redeclare names already exported in TLDR. OCaml types must be declared before use; an abstract type cannot be expanded under the same name later in the signature. Move declarations when dependencies permit. If dependencies such as an existing large GADT prevent this, summarize the primary signatures in a TLDR documentation code block and link to the actual declarations with `{!name}`. Keep excerpts accurate and distinguish them from additional APIs. -- Do not add wrappers, change type equalities, weaken abstraction or redesign a protocol for layout. If a structural change requires a contract change, identify the issue and rationale and follow the existing authorization boundary. - -Emphasize the definitions appropriate to the interface's role: - -| Interface role | Core definitions | -| --- | --- | -| Domain data | Core records/variants, validation entry points and primary conversions | -| Pure reducer | Observable phases, events, instructions, `initial` / `step` and transition results | -| I/O or resources | Handles, principal errors, create/use/close entry points and the meaning of success and closure | -| UI | `props` / `action` / `view`, with the owner of state and event handling | -| Protocol or codec | Message direction, core messages, encode/decode entry points and failure results | - -## Second section: minimal usage example - -Place one complete typical flow immediately after TLDR. Put examples for special branches beside the relevant capability. - -- Use actual public entry points. Do not bypass `.mli` boundaries, manufacture sealed values or call implementation internals. -- For reducers, show installation of the returned state before ordered interpretation of instructions. State when callbacks must queue a later serialized turn. -- For resource interfaces, show acquisition, use and required release. For I/O, show how success and errors propagate. Use `Fun.protect`, `result` or a particular close protocol only when supported by the actual types and API. -- Examples may reference declarations appearing later in the file. Identify caller-supplied variables and callbacks; do not present unexported convenience functions as module APIs. -- Reflect the module's actual persistence and ownership semantics rather than borrowing guarantees from another layer. - -## Detailed sections - -Group by business capability or lifecycle, keeping related types, errors and functions close together. Use sections for navigation; split modules only for distinct concepts or owners. Do not mechanically place all types in one block and all functions in another. - -Document contracts that types cannot express: preconditions, success guarantees, state/resources retained on failure, subsequent actions and lifecycle. Short operations need not mechanically fill every category. - -Near each session, continuation, ticket, receipt or lease definition, explain its creator, holder, binding scope, invalidation conditions and whether it survives restart. Distinguish compile-time type guarantees from runtime owner/generation validation; an abstract type or GADT does not itself prove current authority. - -Represent state, input events, output instructions and operation results separately. For complex state machines, add essential transition rules, especially stale or duplicate events, cancellation races, unknown outcomes, cleanup failures and actual termination conditions. Enumerating phases alone does not describe the complete protocol. - -Preserve real differences between layers. Admission, execution results, durable outcomes and presentation confirmation are separate facts. Explain `None`, empty pages, missing items and partial results with continuations according to their actual semantics. - -Use `(** ... *)`, `{1 ...}`, `{2 ...}`, `{!name}` and `{[ ... ]}` so both source and odoc output are navigable. Keep unrelated module-wide rules out of individual function comments. - -## Illustrative template - -This asynchronous bounded-read reducer illustrates the organization. It is not a replacement contract for an existing interface. Use the actual module's names, types and behavior rather than copying the illustrative protocol. - -```ocaml -(** Coordinate bounded reads and their completion events. - - {1 TLDR} - - This module owns process-local request state; the runner owns actual I/O. - [step] is pure and does not persist state. *) - -type t -type ticket -type request = private { key : string; limit : int } -type error = Invalid_request of string | Read_failed of string -type reply = (string list, error) result -type phase = Active | Draining | Stopped - -type event = - | Requested of request - | Read_completed of ticket * reply - | Shutdown_requested - -type instruction = private - | Run_read of ticket * request - | Cancel_read of ticket - | Publish of ticket * reply - -type transition = private - { next : t - ; instructions : instruction list - } - -val initial : t - -(** Install [next] before interpreting [instructions] in order. *) -val step : t -> event -> transition -val phase : t -> phase - -(** Shutdown enters [Draining]; [Stopped] requires actual reads and callbacks - to have finished. *) - -(** {1 Minimal usage example} - - The caller supplies the [state] reference, input [event] and runner's [interpret]: - {[ - let transition = step !state event in - state := transition.next; - List.iter interpret transition.instructions - ]} - [interpret] queues completion events for a later serialized turn. *) - -(** {1 Construction and validation} *) - -val maximum_limit : int - -(** Require a nonempty [key] and [limit] within [1..maximum_limit]. *) -val request : key:string -> limit:int -> (request, error) result - -(** {1 Lifecycle and recovery} - - Completions match the original ticket. Unrelated or duplicate completions - leave state unchanged. [Draining] accepts completions but no new requests. *) - -(** {1 Observation and helpers} *) - -val pending_count : t -> int -val error_message : error -> string -``` - -## Acceptance - -1. Hide everything after the second section: a reader can still explain the responsibility, core objects, inputs, outputs and basic usage flow. -2. TLDR is first and the usage example is second. Both agree with the actual declarations. All explanatory text and headings in the `.mli` are in English. -3. Type dependencies are valid, without duplicate declarations, invented APIs, weakened sealed/private boundaries or changed protocol semantics. -4. Detailed sections expose the limits, failure handling and termination conditions needed by callers. Handle persistence and invalidation are explicit. -5. Organize only interfaces within the authorized task; do not automatically migrate other `.mli` files. Choose checks using the repository's [final validation scope](../journal-code-review/references/validation.md#final-validation). - -Project references: [compact pure interface](../../../logseq_db_types/lib/squuid.mli), [UI inputs and actions](../../../app/journal_composer_view.mli), and [database contract organized by lifecycle](../../../logseq_overlay_db/spec/database.mli). These illustrate useful structure; they are not assertions that every reference already conforms to this template. +Choose remaining checks using [final validation scope](../journal-code-review/references/validation.md#final-validation). Keep changes within the authorized interfaces; do not automatically migrate other files. diff --git a/AGENTS.md b/AGENTS.md index 728c1751..e50b59f6 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -3,7 +3,7 @@ - Follow [UX guidelines](docs/ux-guidelines.md). Prefer native bonsai-ui/SwiftUI layout, scrolling and animation. Custom coordination through per-row geometry, shared geometry state or repeated scrolling scans needs an explained native API limitation and explicit approval. - Protect existing work. Keep temporary evidence, reports, screenshots and generated hosts in the task's external evidence directory; reusable formal tests belong in existing test directories. - Do not edit OCaml under `spec/` without an explicit request to change its `.mli` contracts, or any Dune file without an explicit request. If a spec blocks development, report the contract issue, suggested change and rationale. Do not edit OCaml in bonsai_flutter. -- All `.mli` specifications, including every `**/spec/**/*.mli`, must follow [journal-mli-spec](.agents/skills/journal-mli-spec/SKILL.md). Read it when writing, changing or reviewing these interfaces: TLDR is the first section and the minimal usage example is the second, followed by construction/validation, capability details, lifecycle/recovery and observation/helpers as applicable. This formatting requirement does not authorize contract changes or unrelated interface migrations. +- All `.mli` interfaces, including every `**/spec/**/*.mli`, must follow [journal-mli-spec](.agents/skills/journal-mli-spec/SKILL.md). Read it when writing, changing or reviewing these interfaces: prefer types and names, with only the shortest necessary comments for important semantics not inferable from signatures. This guidance does not authorize contract changes or unrelated interface migrations. - Use English for `.mli` declarations, documentation, comments, section headings and explanatory text in usage examples. Preserve externally defined names and domain data where required. - Before adding a bug regression, follow [regression scope](.agents/skills/journal-code-review/references/validation.md#regression-scope): identify the production owner and reproduce through its public pure reducer boundary first. - Use `spec-dev-tool --help` for decision documents and its AGENT WORKFLOW. For simplification requests, run `spec-dev-tool guide find-simplifications` or `spec-dev-tool guide find-test-simplifications` inside this worktree as appropriate. diff --git a/app/application.ml b/app/application.ml index 578547d5..2543fd5c 100644 --- a/app/application.ml +++ b/app/application.ml @@ -51,17 +51,18 @@ type unlock_ui = | Unlock_idle | Unlock_waiting of Unlock.unlock_challenge | Unlock_queued of Unlock.unlock_challenge + | Unlock_native_queued of Unlock.graph_scope | Unlock_active of Unlock.unlock_submission | Unlock_downloading of Unlock.unlock_submission | Unlock_retry of Unlock.unlock_challenge * string | Unlock_preparation_error of string | Unlock_complete of Unlock.unlock_attached -let unlock_pending = function Unlock_queued _ | Unlock_active _ -> true | _ -> false +let unlock_pending = function Unlock_native_queued _ | Unlock_queued _ | Unlock_active _ -> true | _ -> false let unlock_ui_challenge = function | Unlock_waiting challenge | Unlock_queued challenge | Unlock_retry (challenge, _) -> Some challenge | Unlock_active submission -> Some (Unlock.unlock_submission_challenge submission) - | Unlock_idle | Unlock_downloading _ | Unlock_preparation_error _ | Unlock_complete _ -> None + | Unlock_native_queued _ | Unlock_idle | Unlock_downloading _ | Unlock_preparation_error _ | Unlock_complete _ -> None ;; let unlock_ui_error = function | Unlock_retry (_, message) | Unlock_preparation_error message -> Some message @@ -69,6 +70,7 @@ let unlock_ui_error = function ;; let unlock_rejected = function + | Unlock_native_queued _ -> Unlock_preparation_error "Unable to unlock the graph. Try again." | Unlock_queued challenge -> Unlock_retry (challenge, "Unable to unlock the graph. Try again.") | Unlock_active submission -> Unlock_retry (Unlock.unlock_submission_challenge submission, "Unable to unlock the graph. Try again.") @@ -82,7 +84,10 @@ let reconcile_unlock previous (snapshot : Graph_service.snapshot) = && snapshot.selected_graph = Some scope.graph_id in let same_challenge challenge = Option.fold ~none:false ~some:(Unlock.equal_unlock_challenge challenge) (unlock_ui_challenge previous) in - match snapshot.unlock with + match previous,snapshot.unlock with + | Unlock_native_queued scope,Unlock.Unlock_inactive when current_scope scope + && snapshot.startup.awaiting_e2ee_password && snapshot.last_error=None -> previous + | _,_ -> match snapshot.unlock with | Unlock.Unlock_challenge challenge when current_scope (Unlock.unlock_challenge_scope challenge) -> if unlock_pending previous && same_challenge challenge then previous else Unlock_waiting challenge @@ -791,7 +796,9 @@ let apply_manager_state ~discard state (manager_state : Graph_service.state) = | None | Some _ -> state in let state = - if owner_changed then ( + if owner_changed + || Option.bind state.manager (fun manager -> manager.selected_graph) <> snapshot.selected_graph + then ( let forget graph_id copies = Option.fold ~none:copies ~some:(fun id -> Local_copies.remove (Logseq_db_types.Graph_types.Uuid.to_string id) copies) graph_id in let copies = forget (Option.bind state.manager (fun manager -> manager.selected_graph)) state.local_copies in @@ -1798,6 +1805,7 @@ module Root_navigation = struct | Capture_closed | Capture_discarded | Capture_picker_requested of Journal_asset_import.source + | Capture_picker_unavailable of string | Capture_asset_picked of Journal_asset_import.staged * int option | Capture_assets_picked of Journal_asset_import.staged list * int option * string option @@ -1918,6 +1926,12 @@ module Root_navigation = struct ; capture_pick_source = source ; capture_pick_request = state.capture_pick_request + 1 } + | Capture_picker_unavailable message -> + if not state.capture_picker_armed then state + else set_composer_error + {state with capture_picker_armed=false} + (picker_owner state) + (Some (Local_capture_failure message)) | (Capture_asset_picked (_, _) as event) | (Capture_assets_picked (_, _, _) as event) -> let items, request_id, error = @@ -2981,7 +2995,8 @@ let manager_props state dispatch : Journal_manager_view.props = let error = match unlock_ui_error state.unlock with Some _ as error -> error | None -> error in let pending = unlock_pending state.unlock in let challenge_ready = Option.is_some - (Option.bind state.manager (fun manager -> Unlock.unlock_challenge manager.unlock)) in + (Option.bind state.manager (fun manager -> Unlock.unlock_challenge manager.unlock)) + || Option.fold ~none:false ~some:(fun manager->manager.Graph_service.startup.awaiting_e2ee_password && manager.resident_scopes<>[]) state.manager in let graph_name = Option.bind state.manager (fun snapshot -> Option.bind snapshot.selected_graph (fun selected -> @@ -3565,7 +3580,7 @@ let start | Graph_service.Client_command (Reconcile_authenticated_user _), Full -> () | _ -> let operation = match request with - | Graph_service.Client_command (Submit_e2ee_password _) -> "submitE2eePassword" + | Graph_service.Client_command (Submit_e2ee_password _ | Submit_native_password _) -> "submitE2eePassword" | Client_command (Restore_local_account _) -> "restoreLocalAccount" | Client_command (Select_graph _) -> "selectGraph" | Client_command (Refresh_catalog _) -> "refreshCatalog" @@ -4153,7 +4168,7 @@ let start let owner = unlock_owner !state_ref in let result = dispatch_worker (Graph_service.Client_command command) in match command with - | Graph_service.Submit_e2ee_password _ -> + | Graph_service.Submit_e2ee_password _ | Submit_native_password _ -> (match result with | Worker.Accepted request_id -> unlock_request := Some (request_id, owner); @@ -6836,13 +6851,7 @@ let start then { state with capture_picker_armed = false } else state) | Journal_asset_import.Unavailable message -> - update (fun state -> - if not state.capture_picker_armed then state - else set_composer_error - (report_ui_error {state with capture_picker_armed=false} - ~operation:"selectAttachment" ~phase:"native") - (picker_owner state) - (Some (Local_capture_failure message))) + update (fun state -> step_root state (Capture_picker_unavailable message)) | Journal_asset_import.Removed token -> update (fun state -> finish_cleanup (with_cleanup (fun ~discard -> match current_composer_owner state with @@ -6984,9 +6993,14 @@ let start | Submit_e2ee_password -> let password = Journal_capture.source snapshot.e2ee_password in let challenge = Option.bind snapshot.manager (fun manager -> Unlock.unlock_challenge manager.unlock) in + let native_scope=Option.bind snapshot.manager(fun manager-> + if not manager.Graph_service.startup.awaiting_e2ee_password then None else + List.find_opt(fun(scope:Unlock.graph_scope)->Some scope.graph_id=manager.selected_graph + && scope.graph_generation=manager.startup.graph_generation + && scope.account.account_generation=manager.startup.account_generation)manager.resident_scopes) in if (unlock_pending snapshot.unlock) - || Option.is_none challenge + || (Option.is_none challenge && Option.is_none native_scope) || String.equal (String.trim password) "" then Effect.ignore else ( @@ -6996,14 +7010,17 @@ let start Effect.bind (update (fun state -> { state with - unlock = Unlock_queued (Option.get challenge) + unlock = (match challenge with Some challenge->Unlock_queued challenge + | None->Unlock_native_queued(Option.get native_scope)) ; e2ee_password = Journal_capture.create ~session_number:state.next_local_sequence ~source:"" ; next_local_sequence = Int64.succ state.next_local_sequence })) - ~f:(fun () -> send_manager (Graph_service.Submit_e2ee_password (Option.get challenge, password)))) + ~f:(fun () -> send_manager (match challenge with + | Some challenge->Graph_service.Submit_e2ee_password(challenge,password) + | None->Graph_service.Submit_native_password(Option.get native_scope,password)))) | Request_local_cache_reset -> update (fun state -> { state with graph_picker_open = true; modal = No_modal }) | Timeline_retry value -> @@ -7587,7 +7604,7 @@ let start {root with resident_states=Graph_drafts.add key next root.resident_states} in ignore(checkpoint_drafts snapshot); match !draft_writer with - |None->Error "A durable draft lock could not be created for this account." + |None->Error "The submitted draft could not be removed for this account." |Some writer when not !draft_storage_blocked-> (match Draft_store.Writer.enqueue writer (fun()->draft_snapshot snapshot) with |Error cause->Error cause |Ok()->flush_drafts()) diff --git a/app/application.mli b/app/application.mli index 05a134ab..79f21cf3 100644 --- a/app/application.mli +++ b/app/application.mli @@ -109,6 +109,7 @@ module Root_navigation : sig | Capture_closed | Capture_discarded | Capture_picker_requested of Journal_asset_import.source + | Capture_picker_unavailable of string | Capture_asset_picked of Journal_asset_import.staged * int option | Capture_assets_picked of Journal_asset_import.staged list * int option * string option diff --git a/app/impl/sync_lifecycle/README.md b/app/impl/sync_lifecycle/README.md new file mode 100644 index 00000000..f3bd5815 --- /dev/null +++ b/app/impl/sync_lifecycle/README.md @@ -0,0 +1,62 @@ +# App sync lifecycle ownership + +`Journal_sync_lifecycle.Core` (`logseq_journal.sync_lifecycle`) is the App-owned +native assembly for a borrowed open graph or an App-owned mirror. Its virtual interface is +`app/spec/sync_lifecycle/core.mli`; the implementation is this directory's +`core.ml`, provided by `journal_sync_lifecycle_impl`. + +## Implemented and relocated + +The assembly originated in +`logseq_sync/lib/service_coordinator/core.{ml,mli}`. The old library/target +`logseq_sync.service_coordinator` is removed, not forwarded or wrapped. + +- `start_native`: check the borrowed database identity, assemble native transport, + crypto, assets and worker owner, start the generic Sync Service, bind ingress, + and activate the database subscription. +- `projection_changed`: forward projection and postcommit notifications. +- `set_network`: update the per-graph service network gate. +- `close`: consume the Service close right, including checked retirement of + owned resources and subscription release. The borrowed database and upload + store remain caller-owned and must outlive close. +- `on_change` / `on_error`: deliver service observations and subscription errors + to the supplied callbacks. These callbacks are not a UI dispatcher. + +The existing native offline-import integration case now uses the App interface. +Its mixed asset/worker integration suite remains in +`logseq_sync/test/asset_owner_lane`; only tests depend upward on the App library. +No production Sync library depends on App or Worker/LUI. + +## Account and production registration + +`Account` owns the account upload store, bounded graph-service registry, catalog +requests, selection, network gates and checked local deletion. Account replacement +and eviction retain ownership when physical service retirement fails. + +`Core.start_owned_native` opens an existing mirror or activates an absent mirror +from the transport-owned snapshot source. It closes its owned database only after +Service retirement and subscription release. `start_native` continues to borrow +its caller's database. + +The native Worker/LUI service now selects `Journal_sync_registration`, which +assembles this controller and `direct_sync_runner`, publishes external lifecycle +observations to Worker, and attaches borrowed database owners for scoped reads. +The injected legacy Worker/LUI constructor remains for existing callers/tests; +legacy reducer and runner source has not yet been removed. Compilation alone is +not evidence that retirement, host handoff and local-first restoration are fully +migrated. Those paths require production-boundary regression coverage before the +old runtime can be safely deleted. + +No production Sync library depends on App or Worker/LUI. This lifecycle library +also has no Worker/LUI dependency; registration composes the two above it. + +## Sync package remains generic + +- `logseq_sync/lib/{crypto,assets,transport,native_crypto,native_transport,worker_owner}`: + production capabilities and native backends. +- `logseq_sync/{spec,lib}/{execute,sync_service,sync_dsl}` and + `logseq_sync/lib/service_lifecycle`: generic execution, service and retirement. +- `logseq_sync/{spec,lib}/{pure_reducer,protocol}`: pure Sync policy and protocol. + +App composes these capabilities; it does not duplicate Sync policy or move the +generic Service into the application. diff --git a/app/impl/sync_lifecycle/account.ml b/app/impl/sync_lifecycle/account.ml new file mode 100644 index 00000000..c8349786 --- /dev/null +++ b/app/impl/sync_lifecycle/account.ml @@ -0,0 +1,466 @@ +module Dsl = Sync_dsl.Core +module Graph = Logseq_db_types.Managed_graph +module Uuid = Logseq_db_types.Graph_types.Uuid +module Http_client = Logseq_sync_transport.Http_client +module Http = Logseq_sync_native_transport.Http +module Native = Logseq_sync_native_transport.Http_eio + +let ( let* ) = Result.bind + +type account = + { origin : string + ; account : string + ; root : string + } + +type service = + { set_network : Dsl.network_gate -> (unit, Dsl.error) result + ; close : unit -> (unit, Dsl.error) result + } + +let service ~set_network ~close = { set_network; close } + +let protect callback = + try callback () with + | Eio.Cancel.Cancelled _ as error -> raise error + | _ -> Error (Dsl.Unexpected_failure "Account lifecycle callback failed") +;; + +let of_core core = + service ~set_network:(Core.set_network core) ~close:(fun () -> + try + Core.close core; + Ok () + with + | Core.Service.Close_failed { error = Some error; _ } -> Error error + | Core.Service.Close_failed { error = None; _ } -> + Error (Dsl.Unexpected_failure "Graph service retirement failed") + | error -> protect (fun () -> raise error)) +;; + +type catalog = + account:account + -> check_current:(unit -> (unit, Dsl.error) result) + -> gate:(unit -> Dsl.network_gate) + -> (Graph.t list, Dsl.error) result + +let decode_catalog source = + let open Yojson.Safe.Util in + let schema = function + | `Null -> { Graph.major = 0; minor = 0; exact = false } + | `String value -> + let component value = + match int_of_string_opt value with + | Some value when value >= 0 -> value + | _ -> failwith "schema" + in + (match String.split_on_char '.' value with + | [ major ] -> { Graph.major = component major; minor = 0; exact = false } + | [ major; minor ] -> + { Graph.major = component major; minor = component minor; exact = true } + | _ -> failwith "schema") + | _ -> failwith "schema" + in + try + let rows = Yojson.Safe.from_string source |> member "graphs" |> to_list in + if List.length rows > 10000 then failwith "catalog capacity"; + let graphs = + List.filter_map + (fun row -> + match member "graph-ready-for-use?" row with + | `Null | `Bool false -> None + | `Bool true -> + let graph_id = + member "graph-id" row |> to_string |> Uuid.of_string |> Result.get_ok + in + let name = member "graph-name" row |> to_string in + if + String.length name = 0 + || String.length name > 512 + || (not (String.is_valid_utf_8 name)) + || String.contains name '\000' + then failwith "name"; + let encrypted = + match member "graph-e2ee?" row with + | `Null -> true + | `Bool value -> value + | _ -> failwith "encryption" + in + Some + { Graph.graph_id + ; name + ; schema = schema (member "schema-version" row) + ; encrypted + } + | _ -> failwith "readiness") + rows + in + Ok graphs + with + | _ -> Error (Dsl.Protocol_failure "Invalid graph catalog") +;; + +let http_catalog ~backend ~token = + fun ~account ~check_current ~gate -> + let* () = + Http.validate_base_url (Uri.of_string account.origin) + |> Result.map_error (fun message -> Dsl.Invalid_input message) + in + let* epoch = + match gate () with + | Dsl.Closed -> Error Dsl.Network_closed + | Dsl.Open { authorization_epoch } -> Ok authorization_epoch + in + let client = Http_client.create ~backend ~token:(token ~account) in + let admit_network () = + let* () = check_current () in + match gate () with + | Dsl.Open { authorization_epoch } when authorization_epoch = epoch -> Ok () + | Dsl.Open _ | Dsl.Closed -> Error Dsl.Network_closed + in + let scope = { Http_client.check_current; admit_network } in + let sent_request = ref None in + let* response = + Http_client.perform client ~scope ~epoch ~authenticated:true (fun token -> + let request = + Http.catalog + ~base_url:(Uri.of_string account.origin) + ~token:(Option.value token ~default:"") + in + sent_request := Some request; + request) + in + let* () = admit_network () in + if response.Native.status = 404 + then Error (Dsl.Protocol_failure "Graph catalog not found") + else + let* () = + Http.validate_response_content_type (Option.get !sent_request) response.headers + |> Result.map_error (fun message -> Dsl.Protocol_failure message) + in + decode_catalog response.body +;; + +let native_catalog ~sw ~clock ~network ~authenticator ~deadlines ~token = + let backend = + Http_client.native_backend ~sw ~clock ~network ~authenticator ~deadlines + in + http_catalog ~backend ~token +;; + +type entry = + { graph : Graph.t + ; service : service + ; live : bool ref + } + +type owner = + { account : account + ; generation : int64 + ; database : Sqlite3.db + ; upload_store : Logseq_sync_worker_owner.Core.store + ; mutable active : bool + ; mutable entries : entry list + } + +type t = + { maximum_services : int + ; fetch : catalog + ; start : + account:account + -> account_generation:int64 + -> graph_generation:int64 + -> graph:Graph.t + -> upload_store:Logseq_sync_worker_owner.Core.store + -> network:Dsl.network_gate + -> on_change:(Core.Service.status -> unit) + -> (service, Dsl.error) result + ; remove : account:account -> graph:Graph.t -> (unit, Dsl.error) result + ; on_change : Graph.t -> Core.Service.status -> unit + ; mutable owner : owner option + ; mutable generation : int64 + ; mutable next_graph : int64 + ; mutable request : int64 + ; mutable network_revision : int64 + ; mutable network : Dsl.network_gate + ; mutable graphs : Graph.t list + ; mutable selected : Uuid.t option + ; mutable busy : bool + } + +let create ~maximum_services ~catalog ~start ~delete_local ~on_change = + if maximum_services <= 0 + then Error (Dsl.Invalid_input "Service capacity must be positive") + else + Ok + { maximum_services + ; fetch = catalog + ; start + ; remove = delete_local + ; on_change + ; owner = None + ; generation = 0L + ; next_graph = 0L + ; request = 0L + ; network_revision = 0L + ; network = Dsl.Closed + ; graphs = [] + ; selected = None + ; busy = false + } +;; + +let mutate t callback = + if t.busy + then Error (Dsl.Capacity_exceeded "Account lifecycle mutation in progress") + else ( + t.busy <- true; + Fun.protect ~finally:(fun () -> t.busy <- false) (fun () -> protect callback)) +;; + +let current t = + match t.owner with + | Some owner when owner.active -> Ok owner + | Some _ | None -> Error Dsl.Stale_context +;; + +let close_entry entry = + entry.live := false; + protect entry.service.close +;; + +let retire t = + t.generation <- Int64.succ t.generation; + t.request <- Int64.succ t.request; + t.graphs <- []; + t.selected <- None; + match t.owner with + | None -> Ok () + | Some owner -> + owner.active <- false; + List.iter (fun entry -> entry.live := false) owner.entries; + let rec close = function + | [] -> Ok () + | entry :: rest -> + let* () = close_entry entry in + owner.entries <- rest; + close rest + in + let* () = close owner.entries in + if Sqlite3.db_close owner.database + then ( + t.owner <- None; + Ok ()) + else Error (Dsl.Storage_failure "Upload store is still busy") +;; + +let logout t = mutate t (fun () -> retire t) + +let replace t account = + mutate t (fun () -> + let* _ = + Dsl.identity + ~origin:account.origin + ~account:account.account + ~graph:"account" + ~account_generation:0L + ~graph_generation:0L + in + if Filename.is_relative account.root || not (Sys.is_directory account.root) + then Error (Dsl.Invalid_input "Account root must be an existing absolute directory") + else + let* () = retire t in + let database = + Sqlite3.db_open (Filename.concat account.root "asset-upload-intents.sqlite") + in + match Logseq_db_storage.Asset_upload_store.initialize_database database with + | Error message -> + ignore (Sqlite3.db_close database); + Error (Dsl.Storage_failure message) + | Ok () -> + t.network <- Dsl.Closed; + t.owner + <- Some + { account + ; generation = t.generation + ; database + ; upload_store = Logseq_sync_worker_owner.Core.store database + ; active = true + ; entries = [] + }; + Ok ()) +;; + +let catalog t = t.graphs +let selected t = t.selected + +let service_count t = + match t.owner with + | None -> 0 + | Some owner -> List.length owner.entries +;; + +let refresh t = + let* owner = current t in + t.request <- Int64.succ t.request; + let request = t.request in + let network_revision = t.network_revision in + let check_current () = + if + owner.active + && t.generation = owner.generation + && t.request = request + && t.network_revision = network_revision + then Ok () + else Error Dsl.Stale_context + in + let result = + protect (fun () -> + t.fetch ~account:owner.account ~check_current ~gate:(fun () -> t.network)) + in + let* () = check_current () in + let* () = if t.busy then Error Dsl.Stale_context else Ok () in + let* graphs = result in + let ids = List.map (fun graph -> graph.Graph.graph_id) graphs in + if List.length (List.sort_uniq compare ids) <> List.length ids + then Error (Dsl.Protocol_failure "Duplicate catalog graph") + else ( + t.graphs <- graphs; + if not (Option.fold ~none:true ~some:(fun id -> List.mem id ids) t.selected) + then t.selected <- None; + Ok graphs) +;; + +let restore_catalog t graphs = + mutate t (fun () -> + let* _ = current t in + let ids = List.map (fun graph -> graph.Graph.graph_id) graphs in + if List.length ids > 10000 || List.length (List.sort_uniq compare ids) <> List.length ids + then Error (Dsl.Protocol_failure "Invalid local catalog") + else (t.graphs <- graphs; Ok ())) +;; + +let release_graph t id = + mutate t (fun () -> + let* owner = current t in + match List.find_opt (fun entry -> entry.graph.Graph.graph_id = id) owner.entries with + | None -> Ok () + | Some entry -> + let* () = close_entry entry in + owner.entries <- List.filter (fun current -> current != entry) owner.entries; + if t.selected = Some id then t.selected <- None; + Ok ()) +;; + +let find_graph t id = + match List.find_opt (fun graph -> graph.Graph.graph_id = id) t.graphs with + | None -> Error (Dsl.Invalid_input "Graph is not in the current catalog") + | Some graph -> Ok graph +;; + +let select t id = + mutate t (fun () -> + let* owner = current t in + let* graph = find_graph t id in + match List.find_opt (fun entry -> entry.graph.Graph.graph_id = id) owner.entries with + | Some entry when !(entry.live) -> + t.selected <- Some id; + Ok () + | Some _ -> Error Dsl.Stale_context + | None -> + let* () = + if List.length owner.entries < t.maximum_services + then Ok () + else ( + let oldest = List.hd (List.rev owner.entries) in + if t.selected = Some oldest.graph.Graph.graph_id then t.selected <- None; + let* () = close_entry oldest in + owner.entries <- List.filter (fun entry -> entry != oldest) owner.entries; + Ok ()) + in + let live = ref true in + let admitted = ref false in + let pending = ref None in + let on_change status = + if !live && owner.active && t.generation = owner.generation + then + if not !admitted + then pending := Some status + else if t.selected = Some id + then t.on_change graph status + in + t.next_graph <- Int64.succ t.next_graph; + let result = + protect (fun () -> + t.start + ~account:owner.account + ~account_generation:owner.generation + ~graph_generation:t.next_graph + ~graph + ~upload_store:owner.upload_store + ~network:t.network + ~on_change) + in + let* service = + match result with + | Ok service -> Ok service + | Error _ as error -> + live := false; + error + in + owner.entries <- { graph; service; live } :: owner.entries; + t.selected <- Some id; + admitted := true; + Option.iter on_change !pending; + Ok ()) +;; + +let deselect t = + mutate t (fun () -> + t.selected <- None; + Ok ()) +;; + +let set_network t network = + mutate t (fun () -> + let* owner = current t in + t.network <- network; + t.network_revision <- Int64.succ t.network_revision; + let results = + List.map + (fun entry -> + if !(entry.live) + then protect (fun () -> entry.service.set_network network) + else Ok ()) + owner.entries + in + List.fold_left + (fun result next -> + match result with + | Error _ -> result + | Ok () -> next) + (Ok ()) + results) +;; + +let delete_local t id = + mutate t (fun () -> + let* owner = current t in + let* graph = find_graph t id in + if t.selected = Some id then t.selected <- None; + t.request <- Int64.succ t.request; + let* () = + match + List.find_opt (fun entry -> entry.graph.Graph.graph_id = id) owner.entries + with + | None -> Ok () + | Some entry -> + let* () = close_entry entry in + owner.entries + <- List.filter (fun entry -> entry.graph.Graph.graph_id <> id) owner.entries; + Ok () + in + let* () = t.remove ~account:owner.account ~graph in + t.graphs <- List.filter (fun graph -> graph.Graph.graph_id <> id) t.graphs; + Ok ()) +;; diff --git a/app/impl/sync_lifecycle/core.ml b/app/impl/sync_lifecycle/core.ml new file mode 100644 index 00000000..96712304 --- /dev/null +++ b/app/impl/sync_lifecycle/core.ml @@ -0,0 +1,350 @@ +module Dsl = Sync_dsl.Core +module Execute = Execute.Core +module Service = Logseq_sync_sync_service.Core +module Transport = Logseq_sync_transport.Core +module Http = Logseq_sync_transport.Http_client +module Crypto = Logseq_sync_crypto.Core +module Assets = Logseq_sync_assets.Core +module Worker = Logseq_sync_worker_owner.Core +module Database = Logseq_overlay_db.Database +module Types = Logseq_overlay_db.Types + +let ( let* ) = Result.bind + +type t = + { identity : Dsl.identity + ; worker : Worker.t + ; service : Service.t + ; database_owner : Execute.database_owner + ; bound : Service.ingress option ref + ; owns_database : bool + } + +let identity t = t.identity +let worker_owner t = t.worker +let database_owner t = t.database_owner + +let ingress t = + match !(t.bound) with + | Some ingress -> Ok ingress + | None -> Error (Dsl.Invalid_input "Sync ingress is not bound") +;; + +let projection_changed t = + let* ingress = ingress t in + let* () = Worker.projection_changed t.worker ~identity:t.identity in + ingress.outbox_committed ~owner:t.database_owner ~identity:t.identity +;; + +let set_network t = Service.set_network t.service +let close_database_owner owner = + let* database = Execute.database_if_open owner in + match database with + | None -> Ok () + | Some database -> Database.close database |> Result.map_error (function + | Types.Close_failed message -> Dsl.Storage_failure ("Database close failed: " ^ message) + | Fatal_close message -> Dsl.Storage_failure ("Database fatal close: " ^ message)) +;; + +let close t = + (try Service.close t.service with error -> + let backtrace = Printexc.get_raw_backtrace () in + Printf.eprintf "journal native-close stage=service-close\n%!"; + Printexc.raise_with_backtrace error backtrace); + if t.owns_database then + match close_database_owner t.database_owner with + | Ok () -> () + | Error error -> + Printf.eprintf "journal native-close stage=database-close\n%!"; + raise (Service.Close_failed { error = Some error; cause = None }) +;; + +let listen_error = function + | Types.Listen_database_closed -> Dsl.Storage_failure "Sync database is closed" + | Subscription_inactive -> Dsl.Storage_failure "Sync subscription is inactive" + | Subscription_already_active -> Dsl.Storage_failure "Sync subscription is already active" + | Listen_capacity_unavailable -> Dsl.Capacity_exceeded "Sync projection subscription" + | Listen_fatal_state message -> Dsl.Storage_failure message +;; + +let read_error = function + | Types.Database_closed -> Dsl.Storage_failure "Database closed" + | Snapshot_released -> Dsl.Storage_failure "Snapshot released" + | Snapshot_generation_invalidated -> Dsl.Storage_failure "Snapshot generation invalidated" + | Read_limit_exceeded -> Dsl.Capacity_exceeded "Graph identity read" + | Stale_read_cursor -> Dsl.Storage_failure "Stale graph identity cursor" + | Invalid_read_request message -> Dsl.Invalid_input message + | Fatal_read_state message -> Dsl.Storage_failure message +;; + +let start_native_assembled ~sw ~clock ~network ~authenticator ~(config : Service.config) ~deadlines + ~liveness ~cleanup_seconds ~root ~make_owner ~owns_database ~upload_store + ~namespace ~encrypted ~token ~on_change ~on_error = + let identity = config.identity and limits = config.limits in + let* () = + if namespace = "" || String.length namespace > 1024 + then Error (Dsl.Invalid_input "Sync import namespace") + else if Filename.is_relative root + then Error (Dsl.Invalid_input "Sync storage root must be absolute") + else Ok () in + let validate_database database = + let* snapshot = Database.current_snapshot database |> Result.map_error read_error in + let graph = Fun.protect ~finally:(fun () -> Database.release_snapshot snapshot) + (fun () -> Database.graph_info snapshot) in + let* graph = graph |> Result.map_error read_error in + if Logseq_db_types.Graph_types.Uuid.to_string graph.graph_uuid <> identity.graph + then Error (Dsl.Identity_mismatch "Sync graph database") else Ok () in + let staging_directory = Filename.concat root "sync-staging" in + let* () = + try + (try Unix.mkdir staging_directory 0o700 with Unix.Unix_error (Unix.EEXIST, _, _) -> ()); + if (Unix.lstat staging_directory).st_kind = Unix.S_DIR then Ok () + else Error (Dsl.Storage_failure "Sync staging path is not a directory") + with Unix.Unix_error (error, operation, _) -> + Error (Dsl.Storage_failure (operation ^ ": " ^ Unix.error_message error)) in + let* transport = Transport.create_native + ~verify_snapshot:Database.inspect_snapshot_storage ~sw ~clock ~network ~authenticator + ~identity ~limits ~deadlines ~cleanup_seconds ~liveness + ~staging_directory ~encrypted ~token in + let* database_owner = make_owner transport in + let* database = Execute.database_if_open database_owner in + let* () = match database with None -> Ok () | Some database -> validate_database database in + let crypto = Crypto.create ~identity ~limits ~encrypted ~backend:Crypto.apple_backend in + let bound = ref None and subscription = ref None and retired = ref false in + let acquired = ref false in + let gate = ref Dsl.Closed and active_assets = ref 0 in + let effect_binding : Execute.effect_scope Eio.Fiber.key = Eio.Fiber.create_key () in + let worker_epoch : int64 Eio.Fiber.key = Eio.Fiber.create_key () in + let check requested = + if requested <> identity then Error (Dsl.Identity_mismatch "Sync coordinator identity") + else if !retired then Error Dsl.Stale_context else Ok () in + let authority () = + let* () = check identity in + match !gate with + | Dsl.Closed -> Error Dsl.Network_closed + | Open { authorization_epoch } -> Ok authorization_epoch in + let http = Http.create ~token ~backend:(Http.native_backend ~sw ~clock ~network + ~authenticator ~deadlines) in + let http_scope epoch : Http.scope = + let current () = + let* actual = authority () in + if actual <> epoch then Error Dsl.Stale_context + else match Eio.Fiber.get effect_binding with + | Some scope -> Execute.check_current scope + | None -> (match Eio.Fiber.get worker_epoch with + | Some admitted when admitted = epoch -> check identity + | Some _ -> Error Dsl.Stale_context + | None -> Error (Dsl.Invalid_input "Asset HTTP has no admitted operation")) in + { check_current = current + ; admit_network = (fun () -> + let* () = current () in + match Eio.Fiber.get effect_binding with + | Some scope -> Execute.admit_network scope + | None -> current ()) } in + let network : Assets.network = + { get = (fun ~identity:requested ~asset ~version ~maximum_wire_bytes -> + let* () = check requested in + let* epoch = authority () in + Http.get_asset http ~scope:(http_scope epoch) ~epoch ~identity:requested + ~asset:asset.Logseq_db_types.Asset_descriptor.uuid ~version ~maximum_wire_bytes) + ; put = (fun ~identity:requested ~asset ~version ~wire -> + let* () = check requested in + let* epoch = authority () in + Http.put_asset http ~scope:(http_scope epoch) ~epoch ~identity:requested + ~asset ~version ~wire) } in + let* assets = Assets.create ~root:(Filename.concat root "logseq-db-worker/assets") ~identity ~limits ~crypto + ~key:(Assets.native_key ~identity ~crypto) ~network in + let notify ~owner ~identity = + match !bound with + | None -> Error (Dsl.Storage_failure "Sync postcommit ingress is not installed") + | Some ingress -> ingress.Service.outbox_committed ~owner ~identity in + let worker = Worker.create ~clock ~namespace ~assets ~store:upload_store + ~notify_outbox_changed:notify in + let module Raw_worker = (val Worker.worker_owner worker : Execute.WORKER_OWNER) in + let module Bound_worker = struct + include Raw_worker + + let bind_runtime ~identity ~database_owner ~limits ~cleanup_seconds + ~(network : Execute.worker_network) = + Raw_worker.bind_runtime ~identity ~database_owner ~limits ~cleanup_seconds + ~network:{ with_admission = (fun callback -> + network.with_admission (fun () -> + let* epoch = authority () in + Eio.Fiber.with_binding worker_epoch epoch callback)) } + end in + let activate_subscription listener = + let notify_change _ = + match Worker.database_changed worker ~owner:database_owner ~identity with + | Error error -> on_error error + | Ok () -> (match notify ~owner:database_owner ~identity with + | Ok () -> () | Error error -> on_error error) in + Database.activate_subscription listener ~notify:notify_change + |> Result.map_error listen_error in + let subscribe database = + if !retired || Option.is_some !subscription then Ok () + else + let* () = validate_database database in + let* listener, snapshot = Database.listen database |> Result.map_error listen_error in + Database.release_snapshot snapshot; + subscription := Some listener; + match !bound with None -> Ok () | Some _ -> activate_subscription listener in + let inspect requested = + if requested <> identity then Error (Dsl.Identity_mismatch "Sync observation identity") + else + let* database = Execute.database_if_open database_owner in + match database with + | None -> Ok None + | Some database -> + let* () = subscribe database in + Database.inspect_sync database |> Result.map Option.some + |> Result.map_error (fun error -> Dsl.Sync_read_failure error) in + let release_subscription () = + Option.iter Database.unlisten !subscription; + subscription := None in + let acquire requested = + let* () = check requested in + if !acquired + then Error (Dsl.Identity_mismatch "Sync coordinator already acquired") + else + let* database = Execute.database_if_open database_owner in + let* () = match database with None -> Ok () | Some database -> subscribe database in + acquired := true; + Ok () in + let with_asset scope callback = + incr active_assets; + Fun.protect ~finally:(fun () -> decr active_assets) + (fun () -> Eio.Fiber.with_binding effect_binding scope callback) in + let diagnose_cleanup stage = Result.map_error (fun error -> + Printf.eprintf "journal native-close stage=%s\n%!" stage; + error) in + let caps : Execute.capabilities = + { validate_identity = (fun requested -> + if requested = identity then Ok () + else Error (Dsl.Identity_mismatch "Sync coordinator identity")) + ; validate_limits = (fun supplied -> + if supplied = limits then Ok () else Error (Dsl.Invalid_input "Sync coordinator limits")) + ; websocket_liveness = liveness; network_deadlines = deadlines; cleanup_seconds + ; acquire + ; fence_network = (fun requested -> + gate := Dsl.Closed; + Transport.fence transport ~identity:requested) + ; close_network = (fun requested -> Transport.close_network transport ~identity:requested) + ; authorize_network = (fun requested ~epoch -> + let* () = check requested in + let* () = Transport.authorize transport ~identity:requested ~epoch in + gate := Dsl.Open { authorization_epoch = epoch }; + Ok ()) + ; validate_resource = Assets.validate_resource assets + ; restore = (fun ~scope requested -> + let* () = Execute.check_current scope in + let* view = inspect requested in + Ok { Dsl.keys = Crypto.status crypto + ; checkpoint = Option.map Types.sync_view_checkpoint view + ; recovery_pending = Option.fold ~none:false + ~some:(fun view -> Types.sync_view_submissions view <> []) view }) + ; worker_owner = (module Bound_worker) + ; dispatch = + { database = (fun ~owner:_ ~scope:_ ~identity:_ _ -> + Error (Dsl.Unexpected_failure "Database instructions belong to Execute")) + ; transport = (fun ~scope ~identity operation -> Transport.dispatch transport ~scope ~identity operation) + ; crypto = (fun ~scope ~identity operation -> Worker.crypto_dispatch worker ~scope ~identity operation) + ; assets = (fun ~scope ~identity operation -> + with_asset scope (fun () -> Worker.assets_dispatch worker ~scope ~identity operation)) } + ; observe = (fun requested -> + let* view = inspect requested in + Ok { Dsl.identity; keys = Crypto.status crypto; network = !gate + ; checkpoint = Option.map Types.sync_view_checkpoint view + ; pending_submissions = Option.fold ~none:0 + ~some:(fun view -> List.length (Types.sync_view_submissions view)) view + ; pending_assets = !active_assets; cleanup_pending = 0; last_failure = None }) + ; fence = (fun requested -> + let* () = if requested = identity then Ok () + else Error (Dsl.Identity_mismatch "Sync fence identity") in + gate := Dsl.Closed; + retired := true; + Assets.fence assets; + Worker.fence worker ~identity:requested) + ; retire = (fun requested -> + let* () = Worker.await_quiescent worker ~identity:requested + |> diagnose_cleanup "worker-quiescence" in + let* () = Assets.close assets |> diagnose_cleanup "asset-cache-close" in + Crypto.retire crypto; + release_subscription (); + Ok ()) + ; release_file = (fun requested file -> Assets.release_file assets ~identity:requested file + |> diagnose_cleanup "asset-file-release") + ; release_lease = (fun requested lease -> Assets.release_lease assets ~identity:requested lease + |> diagnose_cleanup "asset-lease-release") + ; release_snapshot = (fun requested snapshot -> Transport.release_snapshot transport ~identity:requested snapshot + |> diagnose_cleanup "snapshot-release") + } in + let caps = Transport.capabilities transport caps in + let caps = { caps with + authorize_network = (fun requested ~epoch -> + let* () = caps.authorize_network requested ~epoch in + gate := Dsl.Open { authorization_epoch = epoch }; + Ok ()) + ; fence_network = (fun requested -> + gate := Dsl.Closed; + caps.fence_network requested) } in + let bind_ingress ingress = + bound := Some ingress; + match !subscription with + | None -> () + | Some listener -> + (match activate_subscription listener with + | Ok () -> () + | Error error -> on_error error; + failwith (Dsl.error_message error)) in + match Service.start ~sw ~config + ~dependencies:{ clock; database_owner; capabilities = caps; bind_ingress } ~on_change with + | Ok service -> Ok { identity; worker; service; database_owner; bound; owns_database } + | Error error when !acquired -> Error error + | Error primary -> + let cleanup = List.filter_map (function Ok () -> None | Error error -> Some error) + [Transport.close_network transport ~identity; Assets.close assets] in + Crypto.retire crypto; + match cleanup with + | [] -> Error primary + | failures -> Error (Dsl.Cleanup_failure { primary = Some primary; failures = Rrbvec.of_list failures }) +;; + +let start_native ~sw ~clock ~network ~authenticator ~config ~deadlines + ~liveness ~cleanup_seconds ~root ~database_owner ~upload_store + ~namespace ~encrypted ~token ~on_change ~on_error = + start_native_assembled ~sw ~clock ~network ~authenticator ~config ~deadlines + ~liveness ~cleanup_seconds ~root ~make_owner:(fun _ -> Ok database_owner) + ~owns_database:false ~upload_store ~namespace ~encrypted ~token ~on_change ~on_error +;; + +let start_owned_native ~sw ~clock ~network ~authenticator ~(config : Service.config) ~deadlines + ~liveness ~cleanup_seconds ~root ~dependencies ~graph_name ~upload_store + ~namespace ~encrypted ~token ~on_change ~on_error = + let* graph_id = Logseq_db_types.Graph_types.Uuid.of_string config.identity.graph + |> Result.map_error (fun message -> Dsl.Invalid_input ("Graph identity: " ^ message)) in + let* inspection = Database.inspect_mirror ~application_support_directory:root ~graph_id + |> Result.map_error (function + | Types.Invalid_application_support_directory -> Dsl.Invalid_input "Application support directory" + | Mirror_location_unavailable message -> Dsl.Storage_failure ("Mirror location unavailable: " ^ message) + | Mirror_inspection_failed message -> Dsl.Storage_failure ("Mirror inspection failed: " ^ message)) in + let make_owner transport = + match Database.mirror_presence inspection with + | Types.Absent _ -> Execute.activation_owner ~sw ~dependencies ~inspection ~graph_name + ~snapshot_source:(Transport.snapshot_source transport) + | Available _ -> + Database.open_ ~sw dependencies inspection ~graph_name + |> Result.map Execute.database_owner + |> Result.map_error (function + | Types.Mirror_absent -> Dsl.Storage_failure "Mirror disappeared before open" + | Invalid_graph_name -> Dsl.Invalid_input "Graph name" + | Ownership_conflict -> Dsl.Storage_failure "Graph ownership conflict" + | Attachment_stale -> Dsl.Stale_context + | Restore_failed message -> Dsl.Storage_failure ("Graph restore failed: " ^ message) + | Corrupt_authoritative_store message -> Dsl.Storage_failure ("Corrupt authoritative store: " ^ message) + | Corrupt_outbox message -> Dsl.Storage_failure ("Corrupt outbox: " ^ message) + | Corrupt_mutation_receipt message -> Dsl.Storage_failure ("Corrupt mutation receipt: " ^ message) + | Open_resource_failed message -> Dsl.Storage_failure ("Graph open failed: " ^ message)) in + start_native_assembled ~sw ~clock ~network ~authenticator ~config ~deadlines + ~liveness ~cleanup_seconds ~root ~make_owner ~owns_database:true + ~upload_store ~namespace ~encrypted ~token ~on_change ~on_error +;; diff --git a/app/impl/sync_lifecycle/dune b/app/impl/sync_lifecycle/dune new file mode 100644 index 00000000..6e910fd3 --- /dev/null +++ b/app/impl/sync_lifecycle/dune @@ -0,0 +1,6 @@ +(library + (name journal_sync_lifecycle_impl) + (public_name logseq_journal.sync_lifecycle.impl) + (implements journal_sync_lifecycle) + (modules core account) + (libraries eio execute sync_dsl logseq_sync.sync_service logseq_sync.transport logseq_sync.native_transport yojson uri logseq_sync.crypto logseq_sync.assets logseq_sync.worker_owner logseq_db_storage logseq_overlay_db logseq_db_types sqlite3 rrbvec x509 unix)) diff --git a/app/journal_capture.ml b/app/journal_capture.ml index e0456d04..39d06dc9 100644 --- a/app/journal_capture.ml +++ b/app/journal_capture.ml @@ -102,6 +102,7 @@ type t = ; phase : phase ; pending : Journal_graph_request.t option ; submitted_source : string option + ; checkpoint_suppressed : bool ; retry_identity_locked : bool ; retained_chain : Journal_graph_mutation_confirmation.Pure_insertion.t option ; pending_attachments : Journal_asset_import.staged list @@ -113,6 +114,7 @@ let create ~session_number ~source = ; phase = Editing ; pending = None ; submitted_source = None + ; checkpoint_suppressed = false ; retry_identity_locked = false ; retained_chain = None ; pending_attachments = [] @@ -140,15 +142,17 @@ let rebind t ~session_number = let phase t = t.phase let retry_identity_locked t = t.retry_identity_locked +let can_checkpoint t = + not t.checkpoint_suppressed && not t.retry_identity_locked && t.phase <> Saving let lock_for_retry t = { t with retry_identity_locked = true } let release_rejected_attempt t = {t with retry_identity_locked=false;pending=None;submitted_source=None;retained_chain=None;phase=Editing} -let begin_submission t = {t with submitted_source=Some(source t);phase=Saving;retry_identity_locked=true} +let begin_submission t = {t with submitted_source=Some(source t);checkpoint_suppressed=true;phase=Saving;retry_identity_locked=true} let await_confirmation t = if t.submitted_source=None then t else {t with phase=Saving;retry_identity_locked=true} let finish_submission t = match t.submitted_source with |Some submitted when submitted<>source t-> - Some{t with submitted_source=None;phase=Editing;pending=None;retry_identity_locked=false; + Some{t with submitted_source=None;checkpoint_suppressed=false;phase=Editing;pending=None;retry_identity_locked=false; retained_chain=None;pending_attachments=[]} |_->None @@ -225,18 +229,20 @@ let add_attachment t staged = if can_attach t && not duplicate then ( let t = replace_attempt t in - { t with pending_attachments = t.pending_attachments @ [ staged ] }) + { t with pending_attachments = t.pending_attachments @ [ staged ]; checkpoint_suppressed = false }) else t ;; let remove_attachment t ~token = if t.submitted_source<>None then t else - { t with - pending_attachments = - List.filter - (fun staged -> - not (String.equal (Journal_asset_import.staged_token staged) token)) - t.pending_attachments + let pending_attachments = + List.filter + (fun staged -> + not (String.equal (Journal_asset_import.staged_token staged) token)) + t.pending_attachments + in + { t with pending_attachments; + checkpoint_suppressed = t.checkpoint_suppressed && pending_attachments = t.pending_attachments } ;; @@ -246,9 +252,9 @@ let clear_attachments t = let update_source t ~source:new_source = if String.equal (source t) new_source then t - else if t.submitted_source<>None then {t with editor=Editor.replace t.editor ~source:new_source} + else if t.submitted_source<>None then {t with editor=Editor.replace t.editor ~source:new_source;checkpoint_suppressed=false} else if t.retry_identity_locked || t.phase=Saving then t - else let t=replace_attempt t in {t with editor=Editor.replace t.editor ~source:new_source} + else let t=replace_attempt t in {t with editor=Editor.replace t.editor ~source:new_source;checkpoint_suppressed=false} ;; let with_task_state t task_state = @@ -273,7 +279,7 @@ let with_task_state t task_state = { t with task_state; pending }) else ( let t = replace_attempt t in - { t with task_state }) + { t with task_state; checkpoint_suppressed = false }) ;; let toggle_task_intent t = @@ -286,9 +292,11 @@ let toggle_task_intent t = let apply_text_edit t edit = match Editor.apply_text_edit t.editor edit with - | Some editor when t.submitted_source<>None -> {t with editor} + | Some editor when t.submitted_source<>None -> + {t with editor;checkpoint_suppressed=t.checkpoint_suppressed && String.equal (source t) (Editor.source editor)} | Some editor when t.phase <> Saving && not t.retry_identity_locked -> - { t with editor; phase = Editing; pending = None } + { t with editor; phase = Editing; pending = None; + checkpoint_suppressed = t.checkpoint_suppressed && String.equal (source t) (Editor.source editor) } | Some _ -> t | None -> t ;; diff --git a/app/journal_capture.mli b/app/journal_capture.mli index f5126adb..d9f0dd50 100644 --- a/app/journal_capture.mli +++ b/app/journal_capture.mli @@ -36,6 +36,7 @@ val rebind : t -> session_number:int64 -> t val phase : t -> phase val can_save : t -> bool +val can_checkpoint : t -> bool val update_source : t -> source:string -> t val with_task_state : t -> Journal_model.task_state -> t val toggle_task_intent : t -> t diff --git a/app/journal_platform.ml b/app/journal_platform.ml index b292a765..d9032c71 100644 --- a/app/journal_platform.ml +++ b/app/journal_platform.ml @@ -412,6 +412,8 @@ module Draft_store = struct (try on_cleanup_warning warning with _->report_cleanup_warning warning)) let save ?(on_cleanup_warning=report_cleanup_warning) t drafts = protect(fun()-> prepare t; + let drafts=List.filter(fun (draft:draft)-> + Journal_capture.can_checkpoint draft.capture)drafts in let copies=ref [] in let wire=List.map(fun (draft:draft)-> let capture=draft.capture in @@ -424,7 +426,7 @@ module Draft_store = struct |Journal_model.No_status->None |Default_status->Some "default" |Status id->Some id |Invalid_status _->fail "Draft task status is invalid." in Wire.{graph=draft.graph;parent=draft.parent;source=Journal_capture.source capture;task; - interrupted=Journal_capture.retry_identity_locked capture || Journal_capture.phase capture=Saving;assets})drafts in + interrupted=false;assets})drafts in validate wire; let encoded=Yojson.Safe.to_string(Wire.snapshot_to_yojson(snapshot t wire)) in if String.length encoded>maximum_metadata then fail "Draft metadata exceeds 2 MiB."; @@ -460,13 +462,15 @@ module Draft_store = struct let load t = protect(fun()-> if not(exists(directory t)) then [] else ( private_directory(parent t);private_directory(directory t); - let drafts=read_snapshot t in + let stored=read_snapshot t in + let drafts=List.filter(fun (draft:Wire.draft)->not draft.interrupted)stored in let total=ref 0 in List.iter(fun (draft:Wire.draft)->List.iter(fun(asset:Wire.asset)-> total:= !total+regular(Filename.concat(directory t)asset.name)maximum_file; if !total>maximum_media then fail "Draft attachments exceed 128 MiB.")draft.assets)drafts; let staged_files=ref [] in - try List.mapi(fun index (draft:Wire.draft)-> + try + let recovered=List.mapi(fun index (draft:Wire.draft)-> let capture=Journal_capture.create ~session_number:(Int64.of_int(index+1)) ~source:draft.source in let task=match draft.task with None->Journal_model.No_status |Some "default"->Default_status |Some id->Status id in @@ -482,11 +486,10 @@ module Draft_store = struct match Journal_asset_import.decode_event payload with |Ok(Picked(staged,_))->Journal_capture.add_attachment capture staged |_->fail "Draft attachment could not be recovered.")capture draft.assets in - let capture=if draft.interrupted then - Journal_capture.fail (Journal_capture.lock_for_retry capture) - ~message:"The previous save could not be confirmed. Check the graph before discarding this draft." - else capture in - {graph=draft.graph;parent=draft.parent;capture})drafts + {graph=draft.graph;parent=draft.parent;capture})drafts in + if List.length drafts<>List.length stored then ( + match save t recovered with Ok()->() |Error cause->fail cause); + recovered with error->List.iter(fun path->if exists path then Unix.unlink path)!staged_files;raise error)) module Writer = struct diff --git a/app/journal_platform.mli b/app/journal_platform.mli index f666e8b7..9275c8fc 100644 --- a/app/journal_platform.mli +++ b/app/journal_platform.mli @@ -76,7 +76,8 @@ val decode_authenticated_scope : bytes -> (Logseq_db_worker_lui.Logseq_db_worker publication, orphan cleanup is best-effort and cannot turn success into an error; a later save retries cleanup. Save/discard replace the snapshot; [clear] removes only this account's owned files. Caller serializes operations - and reports errors. Recovered interrupted saves remain locked. *) + and reports errors. Submitted drafts are omitted; loading removes legacy + interrupted drafts without restoring submission authority. *) module Draft_store : sig type t type draft = { graph : string; parent : string option; capture : Journal_capture.t } diff --git a/app/journal_startup.ml b/app/journal_startup.ml index d2e20511..3ba7862a 100644 --- a/app/journal_startup.ml +++ b/app/journal_startup.ml @@ -52,8 +52,10 @@ let startup_error ~awaiting_e2ee_password stage message = | During_local_restore -> Local_restore, Some Begin_online_recovery | During_bootstrap -> Bootstrap, Some Begin_online_recovery | During_e2ee -> - E2ee, Some (if awaiting_e2ee_password then Submit_e2ee_password - else Begin_online_recovery) + ( E2ee + , Some + (if awaiting_e2ee_password then Submit_e2ee_password else Begin_online_recovery) + ) in { owner; message; recovery } ;; @@ -70,7 +72,7 @@ let derive ~snapshot ~(graph : Logseq_db_worker.graph_state) = let facts = snapshot.Graph_service.startup in let graph_is_current = graph_matches snapshot graph in let ready = - (facts.authenticated || facts.local_access=Graph_service.Local_available) + (facts.authenticated || facts.local_access = Graph_service.Local_available) && graph_is_current && graph.phase = Logseq_db_worker.Graph_open && not snapshot.timeline_presentation_pending @@ -93,29 +95,57 @@ let derive ~snapshot ~(graph : Logseq_db_worker.graph_state) = | None -> assert false) else if ready then { phase = Ready; error = None } - else if facts.local_access=Graph_service.Local_unavailable Graph_service.No_local_account - then {phase=Signed_out;error=None} - else if facts.awaiting_selection then {phase=Awaiting_selection;error=None} - else if facts.awaiting_e2ee_password then - (match facts.failure with - | None -> {phase=Awaiting_e2ee_password;error=None} - | Some stage -> {phase=Failed;error=Some(startup_error ~awaiting_e2ee_password:true stage - (Option.value snapshot.last_error ~default:"Unable to unlock the graph."))}) - else if (match facts.local_access with Graph_service.Local_unavailable _ -> true | _ -> false) - then - let error=startup_error ~awaiting_e2ee_password:false - (Option.value facts.failure ~default:Graph_service.During_local_restore) - (Option.value snapshot.last_error ~default:"Local graph data is unavailable.") in - let error=if facts.local_access=Graph_service.Local_unavailable Graph_service.Local_restore_failed - then {error with recovery=Some Retry_graph_open} else error in - {phase=Failed;error=Some error} + else if + facts.local_access = Graph_service.Local_unavailable Graph_service.No_local_account + then { phase = Signed_out; error = None } + else if facts.awaiting_selection + then { phase = Awaiting_selection; error = None } + else if facts.awaiting_e2ee_password + then ( + match facts.failure with + | None -> { phase = Awaiting_e2ee_password; error = None } + | Some stage -> + { phase = Failed + ; error = + Some + (startup_error + ~awaiting_e2ee_password:true + stage + (Option.value snapshot.last_error ~default:"Unable to unlock the graph.")) + }) + else if + match facts.local_access with + | Graph_service.Local_unavailable _ -> true + | _ -> false + then ( + let error = + startup_error + ~awaiting_e2ee_password:false + (Option.value facts.failure ~default:Graph_service.During_local_restore) + (Option.value snapshot.last_error ~default:"Local graph data is unavailable.") + in + let error = + if + facts.local_access + = Graph_service.Local_unavailable Graph_service.Local_restore_failed + then { error with recovery = Some Retry_graph_open } + else error + in + { phase = Failed; error = Some error }) else if not facts.authenticated - then - (match facts.failure with - | Some stage -> { phase = Failed; error = Some (startup_error ~awaiting_e2ee_password:facts.awaiting_e2ee_password stage - (Option.value snapshot.last_error ~default:"Startup failed.")) } - | None when facts.restoring_local -> { phase = Restoring_local; error = None } - | None -> { phase = Signed_out; error = None }) + then ( + match facts.failure with + | Some stage -> + { phase = Failed + ; error = + Some + (startup_error + ~awaiting_e2ee_password:facts.awaiting_e2ee_password + stage + (Option.value snapshot.last_error ~default:"Startup failed.")) + } + | None when facts.restoring_local -> { phase = Restoring_local; error = None } + | None -> { phase = Signed_out; error = None }) else if graph_is_current && graph.phase = Graph_failed then { phase = Failed @@ -145,6 +175,15 @@ let derive ~snapshot ~(graph : Logseq_db_worker.graph_state) = | None when facts.awaiting_selection -> { phase = Awaiting_selection; error = None } | None when facts.awaiting_e2ee_password -> { phase = Awaiting_e2ee_password; error = None } + | None when facts.bootstrapping && snapshot.sync_phase = Graph_service.Failed -> + { phase = Failed + ; error = + Some + (startup_error + ~awaiting_e2ee_password:false + During_bootstrap + (Option.value snapshot.last_error ~default:"Graph bootstrap failed.")) + } | None when facts.bootstrapping -> { phase = Bootstrapping; error = None } | None -> { phase = Restoring_local; error = None }) ;; @@ -195,6 +234,7 @@ let decode bytes = module Presentation = struct type ticket = int64 + type observation = { manager : Graph_service.snapshot option ; graph : Logseq_db_worker.graph_state @@ -203,9 +243,25 @@ module Presentation = struct ; password_pending : bool ; graph_error : bool } - type account_result = Local | No_local | User | No_user - type screen = Waiting | Timeline | Interaction of startup_state | Account_failure - type account = Discovering | Local_available | User_available | User_required + + type account_result = + | Local + | No_local + | User + | No_user + + type screen = + | Waiting + | Timeline + | Interaction of startup_state + | Account_failure + + type account = + | Discovering + | Local_available + | User_available + | User_required + type t = { screen : screen ; serial : int64 @@ -217,6 +273,7 @@ module Presentation = struct ; account_failed : bool ; observed : observation option } + type event = | Observe of observation | Account of ticket * account_result @@ -227,86 +284,159 @@ module Presentation = struct | Elapsed of ticket * float let create () = - { screen = Waiting; serial = 0L; account_serial = 0L; indicator = false - ; foreground = true; account = Discovering; local_available = false - ; account_failed = false; observed = None } + { screen = Waiting + ; serial = 0L + ; account_serial = 0L + ; indicator = false + ; foreground = true + ; account = Discovering + ; local_available = false + ; account_failed = false + ; observed = None + } + ;; + let foreground t = t.foreground let screen t = t.screen let loading_indicator t = t.foreground && t.screen = Waiting && t.indicator + let loading_request t = if t.foreground && t.screen = Waiting && not t.indicator then Some t.serial else None + ;; + let account_ticket t = t.account_serial let initial_attempt t = t.serial = 0L + let equal_presentation left right = left.screen = right.screen - && (left.screen <> Waiting || - (loading_indicator left = loading_indicator right - && initial_attempt left = initial_attempt right - && left.foreground = right.foreground)) + && (left.screen <> Waiting + || (loading_indicator left = loading_indicator right + && initial_attempt left = initial_attempt right + && left.foreground = right.foreground)) + ;; + + let required phase = Interaction { phase; error = None } - let required phase = Interaction {phase;error=None} let classify t = - if t.account = User_required then required Signed_out - else match t.observed with - | Some o when o.password_pending -> required Awaiting_e2ee_password - | Some o -> - (match o.manager with - | Some manager when o.graph_ready && o.feed_loaded && not o.graph_error - && (manager.startup.authenticated || manager.startup.local_access=Graph_service.Local_available) - && o.graph.phase = Logseq_db_worker.Graph_open && graph_matches manager o.graph -> Timeline - | Some manager -> - let startup = derive ~snapshot:manager ~graph:o.graph in - (match startup.phase with - | Signed_out when manager.startup.local_access=Graph_service.Local_unavailable Graph_service.No_local_account -> Interaction startup - | Awaiting_selection | Awaiting_e2ee_password | Failed -> Interaction startup - | _ when o.graph_error -> required Failed - | _ when t.account_failed && not t.local_available -> Account_failure - | _ -> Waiting) - | None -> if t.account_failed && not t.local_available then Account_failure else Waiting) - | None -> if t.account_failed && not t.local_available then Account_failure else Waiting + if t.account = User_required + then required Signed_out + else ( + match t.observed with + | Some o when o.password_pending -> required Awaiting_e2ee_password + | Some o -> + (match o.manager with + | Some manager + when o.graph_ready + && o.feed_loaded + && (not o.graph_error) + && (manager.startup.authenticated + || manager.startup.local_access = Graph_service.Local_available) + && o.graph.phase = Logseq_db_worker.Graph_open + && graph_matches manager o.graph -> Timeline + | Some manager -> + let startup = derive ~snapshot:manager ~graph:o.graph in + (match startup.phase with + | Signed_out + when manager.startup.local_access + = Graph_service.Local_unavailable Graph_service.No_local_account -> + Interaction startup + | Awaiting_selection | Awaiting_e2ee_password | Failed -> Interaction startup + | _ when o.graph_error -> required Failed + | _ when t.account_failed && not t.local_available -> Account_failure + | _ -> Waiting) + | None -> + if t.account_failed && not t.local_available then Account_failure else Waiting) + | None -> + if t.account_failed && not t.local_available then Account_failure else Waiting) + ;; let project t = let next = classify t in - if next = t.screen then t - else if next = Waiting then - {t with screen=next;serial=Int64.succ t.serial;indicator=false} - else {t with screen=next;indicator=false} + if next = t.screen + then t + else if next = Waiting + then { t with screen = next; serial = Int64.succ t.serial; indicator = false } + else { t with screen = next; indicator = false } + ;; let step t event = - let next = match event with - | Observe o -> - let previous = Option.bind t.observed (fun o -> o.manager) in - let changed_account = match previous,o.manager with - | Some previous,Some next -> previous.startup.account_generation <> 0 - && previous.startup.account_generation <> next.startup.account_generation - | _ -> false in - let account = match o.manager with - | Some manager when changed_account && not manager.startup.authenticated - && not manager.startup.restoring_local && manager.startup.local_access=Graph_service.Local_pending -> User_required - | Some manager when manager.startup.authenticated && t.account <> User_required -> User_available - | _ -> t.account in - project {t with observed=Some o;account; - account_serial=(if changed_account then Int64.succ t.account_serial else t.account_serial)} - | Account (key,result) when key=t.account_serial -> - let account,local_available=match result with - | Local -> Local_available,true - | No_local -> (if t.account=Local_available then Discovering else t.account),false - | User -> User_available,t.local_available - | No_user -> User_required,false in - project {t with account;local_available;account_failed=false} - | Account_error key when key=t.account_serial -> project {t with account_failed=true} - | Retry -> - {t with screen=Waiting;serial=Int64.succ t.serial;account_serial=Int64.succ t.account_serial; - indicator=false;account=Discovering;local_available=false;account_failed=false;observed=None} - | Signed_out -> - {t with screen=required Signed_out;indicator=false;account=User_required; - account_serial=Int64.succ t.account_serial;local_available=false;account_failed=false} - | Foreground active when active<>t.foreground -> - {t with foreground=active;indicator=false; - serial=(if active && t.screen=Waiting then Int64.succ t.serial else t.serial)} - | Elapsed (key,elapsed) when key=t.serial && t.foreground && t.screen=Waiting - && elapsed>=0.1 && not t.indicator -> {t with indicator=true} - | Account _ | Account_error _ | Foreground _ | Elapsed _ -> t + let next = + match event with + | Observe o -> + let previous = Option.bind t.observed (fun o -> o.manager) in + let changed_account = + match previous, o.manager with + | Some previous, Some next -> + previous.startup.account_generation <> 0 + && previous.startup.account_generation <> next.startup.account_generation + | _ -> false + in + let account = + match o.manager with + | Some manager + when changed_account + && (not manager.startup.authenticated) + && (not manager.startup.restoring_local) + && manager.startup.local_access = Graph_service.Local_pending -> + User_required + | Some manager when manager.startup.authenticated && t.account <> User_required + -> User_available + | _ -> t.account + in + project + { t with + observed = Some o + ; account + ; account_serial = + (if changed_account then Int64.succ t.account_serial else t.account_serial) + } + | Account (key, result) when key = t.account_serial -> + let account, local_available = + match result with + | Local -> Local_available, true + | No_local -> + (if t.account = Local_available then Discovering else t.account), false + | User -> User_available, t.local_available + | No_user -> User_required, false + in + project { t with account; local_available; account_failed = false } + | Account_error key when key = t.account_serial -> + project { t with account_failed = true } + | Retry -> + { t with + screen = Waiting + ; serial = Int64.succ t.serial + ; account_serial = Int64.succ t.account_serial + ; indicator = false + ; account = Discovering + ; local_available = false + ; account_failed = false + ; observed = None + } + | Signed_out -> + { t with + screen = required Signed_out + ; indicator = false + ; account = User_required + ; account_serial = Int64.succ t.account_serial + ; local_available = false + ; account_failed = false + } + | Foreground active when active <> t.foreground -> + { t with + foreground = active + ; indicator = false + ; serial = + (if active && t.screen = Waiting then Int64.succ t.serial else t.serial) + } + | Elapsed (key, elapsed) + when key = t.serial + && t.foreground + && t.screen = Waiting + && elapsed >= 0.1 + && not t.indicator -> { t with indicator = true } + | Account _ | Account_error _ | Foreground _ | Elapsed _ -> t in - if next=t then t else next + if next = t then t else next + ;; end diff --git a/app/spec/journal_execution_flow.mli b/app/spec/journal_execution_flow.mli index 8f0e850b..55c4b4ef 100644 --- a/app/spec/journal_execution_flow.mli +++ b/app/spec/journal_execution_flow.mli @@ -1,650 +1,31 @@ -(** {1 TLDR} - - Explain the important Journal execution paths using existing specification - types: App coordination, DB Worker execution, Sync, durable writes, observation - and resource retirement. This file contains documentation only. It exports no - types or values, adds no runtime owner and is not a new Dune virtual library. - - The examples use these aliases for existing modules, not new interfaces: - {[ - module A = Journal_worker_coordination.Worker_api - module AC = Journal_worker_coordination.Worker_pure_coordinator - module AR = Journal_worker_coordination.Worker_effect_runner - module W = Logseq_db_worker_pure_reducer.Core - module WR = Logseq_db_worker_effect_runner.Effect_runner - module S = Logseq_sync_pure_reducer.Core - module SR = Logseq_sync_effect_runner.Effect_runner - ]} - - Open the corresponding module contracts here; the aliases inside code and - trace blocks remain literal OCaml names: - {ul - {- [A]: {!Journal_worker_coordination.Worker_api}.} - {- [AC]: {!Journal_worker_coordination.Worker_pure_coordinator}.} - {- [AR]: {!Journal_worker_coordination.Worker_effect_runner}.} - {- [W]: {!Logseq_db_worker_pure_reducer.Core}.} - {- [WR]: {!Logseq_db_worker_effect_runner.Effect_runner}.} - {- [S]: {!Logseq_sync_pure_reducer.Core}.} - {- [SR]: {!Logseq_sync_effect_runner.Effect_runner}.}} - - There are three policy boundaries, not three independent App event loops: - {ul - {- App coordination owns typed tickets, sessions, presentation checks and - request/resource retirement. The host owns UI/business state.} - {- Worker owns graph lifecycle, scoped DB execution and the embedded Sync - reducer. Reads and durable writes take different execution paths.} - {- Sync decides network/bootstrap/outbox work. Its network runner performs - I/O; database work is delegated back to Worker.}} - - Each reducer returns a next state plus ordered work. Install the state before - interpreting the work. I/O callbacks enqueue later serialized events; they do - not recursively call a reducer or directly update the UI. - - Admission, local durable commitment, confirmation for the current presentation - and server synchronization are separate facts. Cancellation is not rollback. - A cleanup request is not a physical release acknowledgement. - - OCaml blocks below are illustrative local functions built from the existing - spec API; their helper names are not exported by this file or the repository. - Trace blocks describe values produced/consumed by owners, not instructions to - manufacture private constructors or sealed capabilities. Host-only operations - are stated explicitly rather than replaced by fictional APIs. - - {1 Minimal usage example} - - Read one Journal page after real bootstrap has supplied an Open [A.session]. - The caller supplies the serialized coordinator state and its existing runner. - This request never creates a missing Journal page. - {[ - let request_journal - (state : AC.t ref) - (runner : AR.t) - (session : A.session) - ~yyyymmdd = - match A.journal_day yyyymmdd with - | Error error -> Error error - | Ok day -> - match A.read_budget session - ~maximum_items:1 ~maximum_bytes:4096 with - | Error error -> Error error - | Ok budget -> - match AC.request !state session (A.Query (budget, A.Journal day)) with - | Error error -> Error error - | Ok (ticket, transition) -> - state := transition.next; - List.iter (AR.submit runner) transition.instructions; - Ok ticket - ]} - - The result is an [A.page option AC.ticket], not the page itself. The existing - mailbox pump later calls [AR.on_event]; the runner queues [AC.Completed] and - the host applies it through the same serialized coordinator turn: - {[ - let app_turn (state : AC.t ref) (runner : AR.t) (event : AC.event) = - let transition = AC.step !state event in - state := transition.next; - List.iter (AR.submit runner) transition.instructions - - let journal_reply - (state : AC.t ref) - (ticket : A.page option AC.ticket) = - AC.reply (AC.view !state) ticket - - let forget_consumed_reply state runner ticket = - let transition = AC.step !state (AC.Forget ticket) in - state := transition.next; - List.iter (AR.submit runner) transition.instructions - ]} - - [None] from [AC.reply] means pending; [Answered (Ok None)] means the page is - absent; [Answered (Ok (Some page))] carries the actual page. Handle - [Not_admitted], [Answered (Error error)] and [Interrupted] separately. The host - consumes the reply before calling the local [forget_consumed_reply] example. - If consumption is deferred to another App turn, revalidate the reply with - [AC.revalidate_reply] against that turn's view before applying it. Forgetting - a ticket does not release an observation, demand or file lease. - - {1 Construction and validation} - - {2 Host startup and the first usable session} - - Native bridge startup, authenticated-user/token exchange, graph catalog, - selection/unlock policy, the shared client and the single mailbox pump remain - host responsibilities. They are not APIs of this document or the App - coordination spec. The current integration starts the Worker runtime from - [app/application.ml] and borrows that client in - [app/journal_worker_coordination_host.ml]. - - The following example accepts an already verified client/account. [post] is - a host-supplied queue callback of type [AC.event -> unit]; [publish] consumes - [AC.view]. Neither may synchronously re-enter a coordinator turn. Store the - returned state/runner before draining the queued binding event. - {[ - let create_coordination - (client : AR.client) - ~client_id ~managed_origin ~user_id ~post ~publish = - let state = ref (AC.initial ~client_id) in - match AR.create ~client ~managed_origin ~user_id ~post ~publish with - | Error error -> Error error - | Ok runner -> Ok (state, runner) - ]} - - The actual sequence is: - {v - AR.create - -> queued AC.Worker_bound binding - -> AC.step - -> private AC.Bootstrap instruction - -> AR.submit - -> existing Service registration + authoritative baseline/replay - -> queued AC.Bootstrap_received (binding, A.Ready projection) - -> AC.step - v} - - [AR.create] success is not [Ready]. [Ready] with no resident graphs is valid - and different from [Bootstrap_not_ready]. A usable session comes from an - Open resident in [A.residents projection]; selected presentation is obtained - with [A.selected projection], and its session with [A.presentation_session]. - [AC.validate_session] and request admission still check current authority. - Do not construct a session from a UUID or optimistic UI selection. - - This App baseline is distinct from Sync graph bootstrap, which may restore a - local mirror or download a snapshot. Registration can succeed while a graph - is still Opening. [Bootstrap_failed] and [Binding_revoked] remain observable; - an explicit [AC.Retry_bootstrap] respects old-registration release before - replacement. Old binding callbacks cannot authorize the replacement. - - {2 Scope and data validation} - - [A.session] is execution authority; [A.presentation] is current-view authority. - Focus changes can preserve a resident session while replacing presentation. - Close/reopen, account replacement and rebinding invalidate execution identity. - [A.read_budget], [A.journal_day], [A.append_intent] and [A.mutation_intent] - validate input; DTOs, UUIDs and successful shape checks alone grant no authority. - - Some spec signatures deliberately refer to contracts outside [spec/], notably - [AR.client] in {!Journal_worker_coordination.Worker_effect_runner}, - [packed_query]/[answer] in {!Worker_pure_read}, - [t] in {!Logseq_db_worker_contract.Config} and - [durable_intent]/[outcome] in - {!Logseq_db_worker_contract.Worker_pure_mutation}. - Those are existing boundary inputs, not types defined here. The examples keep - them opaque and never invent their representation or a packing function. - - {1 Capability details} - - {2 App request, Service transport and Worker read} - - [AC.request] returns the exact typed ticket together with its transition. - Apply that transition before planning another request, including an identical - concurrent request. Only the coordinator produces private [AC.Send]. - {v - A.Query (budget, query) - -> AC.request - -> AC.Send (ticket, session, request) - -> AR.submit - -> Service.Coordination_query [implementation boundary] - -> scoped Worker read wrapper [implementation boundary] - -> WR.await_reply - -> queued W.Graph_request_for_scope { scope; id; read } - -> W.step - -> W.Run_worker - (W.Request (ticket, W.Execute_request { database; read })) - -> WR.submit and DB I/O - -> queued W.Runner_completed (W.Execute_request_completed ...) - -> W.step - -> W.Publish (W.Reply (id, answer)) - -> resolve the original await_reply waiter - -> Service response and existing mailbox - -> AR.on_event - -> queued AC.Completed (ticket, reply) - -> AC.step, AC.reply, business projection and UI - v} - - App tickets, actual accepted mailbox IDs and Worker request/effect tickets are - different identities. The transport associates real accepted IDs; it does not - infer ownership from a response's shape or success. Worker validates pending - identity/generation before publishing a reply. [W.Publish (W.Reply ...)] has a - dedicated waiter-resolution path, not an ordinary broadcast-only callback. - - This boundary example accepts an existing packed read and a host queue callback; - packing the App query is not an API in the selected spec files: - {[ - let worker_read - (runner : WR.t) - (scope : S.graph_scope) - (id : W.request_id) - (read : Worker_pure_read.packed_query) - ~(post : W.event -> unit) = - WR.await_reply runner ~id ~read - ~post:(fun () -> post (W.Graph_request_for_scope { scope; id; read })) - ]} - - The shared App pump calls [AR.on_event] before legacy response dispatch: - {ul - {- [Consumed]: this owner handled the response, including owned failures, - cancellation or malformed receipts. Never also run a generic failure path.} - {- [Unowned_response]: forward only to another registered request owner.} - {- [Broadcast]: other interested owners receive it once; this runner already - processed its share.} - {- [Dropped_envelope]: discard a foreign/stale envelope.}} - An error in [delivery.error] does not erase its routing disposition. - - {2 Timeline, detail and bounded continuation} - - Business reducers still decide date ranges, feed order, Favorites, navigation, - detail expansion and Capture editing. They translate those decisions into - existing typed queries, rather than becoming part of Worker policy: - {ul - {- [A.Journals] reads a bounded date range and initial revision.} - {- [A.Tree], [A.Children] and [A.Child_members] read bounded structure.} - {- [A.Page] and [A.Outline_block] preserve present/missing results.} - {- [A.Block_summary] reads a target, owning page and child slice atomically.} - {- [A.Favorites], [A.Task_statuses] and [A.Task_status_definitions] retain their - real DTOs, ordering, revisions and missing slots.}} - - A returned continuation is bound to the session, original query parameters, - revision and budget. Use [A.continuation_budget] and the same query when - continuing; a continuation does not renew a stale session or snapshot. Empty - data is a successful observation. A partial slice is not a complete tree or an - exact total child count. The host applies [page_data], [block_data] and other - canonical DTOs through its existing graph projection, not a new UI model here. - - {2 Capture Append and durable mutation} - - Create stable mutation/block identities and the complete bounded intent before - first send. The following local helper makes the host's persistence obligation - explicit. [persist] is a caller-supplied [bytes -> (unit, A.error) result]; it - returns [Ok ()] only after durable storage. It is not a spec module function. - A failed [persist] must not send the request. - {[ - let append_to_journal - (state : AC.t ref) (runner : AR.t) (session : A.session) - ~mutation_id ~block_id ~day ~title ~task_status ~children - ~(persist : bytes -> (unit, A.error) result) = - match A.append_intent session ~mutation_id ~block_id ~day ~title - ~task_status ~children with - | Error error -> Error error - | Ok intent -> - let durable = A.Append_intent intent in - match persist (A.encode_durable_intent durable) with - | Error error -> Error error - | Ok () -> - match AC.request !state session (A.Append intent) with - | Error error -> Error error - | Ok (ticket, transition) -> - state := transition.next; - List.iter (AR.submit runner) transition.instructions; - Ok (durable, ticket) - ]} - - Editing an existing block uses [A.mutation_intent session ~mutation_id - (A.Save_block { block; title; preconditions })], persists the corresponding - [A.Mutation_intent], then requests [A.Mutate intent]. Other mutations use the - existing Insert/Delete/Create_journal/status/asset-reference constructors. - Append is atomic: Worker resolves or creates the journal by day and commits - the complete root/child/status payload. Separate create/insert/status calls - are not an implementation of the same contract. - - Durable writes do not travel through [W.Execute_request]: - {v - A.Append / A.Mutate - -> AC.request / AR.submit - -> Service.Coordination_mutation [implementation boundary] - -> scoped durable-mutation wrapper [implementation boundary] - -> WR.durable_mutation - -> persist immutable local intent - -> commit local intent / outbox state - -> queued W.Sync_event (S.Local_graph_outbox_changed scope) - -> local committed proof returned through Service to App - v} - - Posting the outbox notification precedes returning the local proof. Its later - consumption and the App reply remain separate: neither proves completion of - the other or server synchronization. - - The Worker runner API can be shown without inventing its referenced intent: - {[ - let durable_write - (runner : WR.t) - (scope : S.graph_scope) - (intent : Logseq_db_worker_contract.Worker_pure_mutation.durable_intent) - ~(current : unit -> bool) = - WR.durable_mutation runner ~scope ~intent ~current - ]} - - [current] is a host-supplied check of the actual account/graph/binding owner, - not a constant [true]. The DB owner deduplicates by immutable identity and - bytes. The same ID with a different payload conflicts. On success it posts - [Local_graph_outbox_changed]; a failed/unknown operation cannot invent that - commit proof. Source callbacks and durable outcomes retain their real owner. - - {2 Local commitment versus presentation confirmation} - - [A.submission_outcome] is separate from [A.mutation_progress]: - {ul - {- [Submission_committed receipt] proves local durability, not current UI - presentation or server acceptance.} - {- [Submission_unknown] retains the original immutable intent for recovery. - [Submission_unknown_with_cause cause] and [Outcome_unknown_with_cause cause] - additionally retain an allowlisted, at-most-1024-byte diagnostic from that - exact request. The same owner/generation checks apply; a global latest - diagnostic must not supply another graph's cause. This remains unknown, - never rejection or rollback, and adds no local retry. Error info renders - the safe classification without raw exception text, journal fields, UUIDs - or credentials. Generic recovery must not erase a retained specific cause.} - {- [Submission_rejected error] requires evidence that the intent did not apply.}} - - [Not_admitted] concerns this attempt only. After an earlier admitted attempt, - queue refusal, interruption, reply loss or an unvalidated error cannot prove - rollback. A retry must not replace an unknown intent with a fresh ID. - - Confirm performs bounded canonical readback, not another mutation: - {[ - let confirm_commit - (state : AC.t ref) (runner : AR.t) - (presentation : A.presentation) (receipt : A.receipt) = - let session = A.presentation_session presentation in - match AC.request !state session (A.Confirm (presentation, receipt)) with - | Error error -> Error error - | Ok (ticket, transition) -> - state := transition.next; - List.iter (AR.submit runner) transition.instructions; - Ok ticket - - let confirmation_data (confirmation : A.confirmation) = - A.confirmation_primary confirmation, - A.confirmation_additional confirmation - ]} - - The business owner applies that readback only to the still-current - presentation. It need not query the same evidence again. Additional fragments - are bounded: omitted targets are not absent, and child continuations retain - their snapshot/budget. Readback failure preserves the committed receipt. - Focus away/back creates a new presentation even if the session survives. - [AC.reconcile_mutation] returns a [Submission_ticket] for unresolved intent or - a [Confirmation_ticket] for known commitment. After actual UI application, - [AC.settle_applied_mutation] determines acknowledgement versus reconciliation; - neither a transport ACK nor merely reading a reply proves UI consumption. - - {2 Worker execution and its embedded Sync reducer} - - [W.initial config] validates the initial state. Its host creates [WR.t] from - real dependencies and binds [WR.create]'s [post] to one serial event channel. - Recovery/upload/asset predicates read the installed Worker state. The host - then submits [W.Start] through the same turn function used for later events: - {[ - let worker_turn - (state : W.state ref) (runner : WR.t) (event : W.event) = - let transition = W.step !state event in - state := transition.next; - List.iter (WR.submit runner) transition.effects - ]} - - Worker owns the Sync reducer state. [W.Sync_event event] causes an internal - [S.step] and translates its output; App must not create a second Sync state: - {v - S.Run network_effect - -> W.Run_sync network_effect - -> WR.submit -> configured Sync runner -> SR.submit - - S.Delegate worker_effect - -> W.Run_worker - (W.Request (ticket, W.Handle_sync_worker_effect worker_effect)) - -> WR.submit -> DB I/O - -> W.Runner_completed (W.Sync_worker_effect_completed ...) - -> apply the returned lifecycle_result - -> feed sync_worker_result.event, if present, to the owned Sync reducer - - S.Publish output - -> ordinary output: W.Publish (W.Sync_output output) - -> protected/asset results: their existing dedicated delivery paths - v} - - Detach/reset can wait for resource retirement rather than execute immediately. - Network callbacks produce [S.event], queued as [W.Sync_event]. Database - completions keep the original ticket; stale generations do not acquire fresh - authority. [S.runner_effect] is private and its effect tickets are opaque. - These traces describe reducer output, not legal external construction. - - {2 Graph selection, local restore and snapshot bootstrap} - - The host supplies authenticated-user, selection, online-recovery and unlock - input through the existing Service owner. Sync chooses a local or online path: - {v - S.Graph_selected graph_id - -> Delegate (Inspect_mirror mirror_request) - -> Mirror_inspected (Mirror_available mirror_request) - -> Delegate (Attach_graph mirror_request) - -> Graph_attached { scope; sync } - - For an absent mirror when online bootstrap is permitted: - -> Run (Request (ticket, Fetch_snapshot_metadata scope)) - -> Runner_completed (Completion (ticket, Ok metadata)) - -> Run (Request (ticket, Download_snapshot download)) - -> Runner_completed (Completion (ticket, Ok artifact)) - -> Delegate (Activate_snapshot activation_request) - -> Snapshot_activated activation - -> Delegate (Inspect_mirror mirror_request) - -> Mirror_inspected (Mirror_available mirror_request) - -> Delegate (Attach_graph mirror_request) - -> Graph_attached { scope; sync } - v} - - Activation is followed by a new mirror inspection, not an unconditional direct - attach. Encrypted graphs additionally use Load_and_unlock_graph_key and, when - needed, Fetch_e2ee_graph_key/Fetch_e2ee_user_keys and E2ee_password_submitted. - Snapshot download/artifact/activation values are opaque, request-bound - capabilities. Their owner uses the specified accessors and completion - constructors; a fabricated path/UUID record is not an activation receipt. - [Fetch_snapshot_baseline] exists but is not a mandatory step in the current - normal metadata/download bootstrap path. Failure and explicit recovery remain - distinct from successful local restore; an offline local graph can still be - usable without network progress. - - {2 Local outbox, server acknowledgement and authoritative pull} - - The typical success path starts only after local commit. Exact request fields, - keys and scopes are reducer-produced values; the trace omits their payloads: - {v - S.Local_graph_outbox_changed graph_scope - -> Delegate (Inspect_sync graph_scope) - -> Sync_inspected { scope; sync } - -> Delegate (Apply_outbox_transition request) - request.transition = Submit_group mutation_ids - -> Outbox_transition_applied { scope; commit; sync } - -> Run (Send_websocket ...) - message = Sync_protocol.Client.Tx_batch ... - -> Websocket_message (..., Sync_protocol.Server.Tx_batch_ok ...) - -> Delegate (Apply_outbox_transition request) - request.transition = Accept_group { batch_id; barrier } - -> Outbox_transition_applied ... - -> Run (Send_websocket ...) - message = Sync_protocol.Client.Pull { since } - -> Websocket_message (..., Sync_protocol.Server.Pull_ok ...) - -> Delegate (Apply_authoritative_batch batch) - -> Authoritative_batch_applied { scope; commit; sync } - v} - - [Submit_group] is persisted before Tx_batch transmission. Server acceptance - does not itself mean the authoritative batch is applied locally. Disconnects - and rejection can require Retry_group/Reject_group or a blocked/deferred state; - the arrows above are not unconditional retry policy. The opaque - [sync_view] in {!Logseq_overlay_db.Types} is inspected through its accessors, - such as [sync_view_token], never through invented pending/token fields. - - [Websocket_opened], Server.Hello and Server.Changed can initiate Client.Pull. - The current path does not require an invented initial Client.Hello send. - Successful authoritative DB application produces the existing projection and - change/observation facts; a network callback does not directly render UI. - - {2 Observations, changefeed and visible refresh} - - [A.Observe (budget, query)] registers and samples with a gap-free baseline or - replay boundary. Check [A.supports_observation session query]; the coarse - capability flag does not support every query. Observe Changes is rejected so - changefeed pull/ACK retains one owner. The coordinator receives registration, - invalidation and sampled facts through [AC.Observation_registered], - [AC.Observation_notice] and [AC.Observation_received]. - - [A.Refresh observation] retains its registered handle. Failed reads do not - release the interest or discard retained reconciliation. Empty data remains a - real result. [A.Release observation] requires its actual release ACK; late - registration/acquisition racing cancellation remains owned for cleanup. - - The changefeed alternative uses [A.Query (budget, A.Changes { generation; - after })], then [AC.begin_change_application] and its exact consumer tokens. - Business owners report [Change_consumer_expanded], [Change_consumer_applied] - or [Change_consumer_retired] as they actually create/apply/retire dependent - work. Only after all affected state is applied or explicitly retired may - [A.Acknowledge_changes batch] advance that exact generation/window chain. - A read failure is not an ACK. One migrated interest cannot be independently - rehydrated or acknowledged by both the legacy runtime and the new owner. - - {2 Media, import and retained files} - - Picker UI, editing and source staging remain host/business operations. Worker - consumes existing descriptors and complete [A.asset_import_intent] values: - {v - Visible media demand: - asset_consumer -> Replace_asset_demand -> checked asset_demand - -> asset notice -> ready_asset_file -> Acquire_asset_file - -> optional file_lease -> host consumes file_path/file_type - -> Release_file -> actual release ACK + File_lease_returned - -> Release_asset_demand -> actual demand cleanup ACK - - Durable import: - persist encode_asset_import_intent before first Import - -> Import_rejected / Import_unknown / Imported - -> optional Acquire_imported_file -> tracked file_lease - -> existing upload/Sync owners report later upload status - v} - - A demand ACK is not download completion; an import receipt is not cloud upload - completion. Source retirement requires a checked durable import/adoption proof - or known rejection; Unknown retains the original source obligation. File paths - alone grant no acquisition authority. [AC.File_lease_returned] and - [AC.Observation_returned] mean the presentation no longer retains a capability, - not that physical I/O cleanup finished. Keep physical releases, creator - completion and finalization separately tracked; no retry may leak an orphan - retain or silently replay a previously completed physical release. - - {1 Lifecycle and recovery} - - {2 Retry without changing identity} - - Preserve complete immutable intent bytes before first send. On restart, - [A.restore_durable_intent] validates the same logical graph and original - payload; restoration itself is read-only. [A.Recover_mutations] and import - recovery expose persistent evidence. [AC.reconcile_mutation] does not authorize - changing the intent, silently replacing an unknown ID or treating equal text - after A -> B -> A as confirmation of the old action. - - [AC.Cancel ticket] requests cooperative stop, not rollback. [AC.Forget ticket] - retires result presentation, not actual in-flight ownership or durable intent. - Full/not-ready/stopping are typed admission results; keep retained logical - dependencies and retry only under their specified capacity/recovery events. - A remote rejection cannot erase an already locally committed receipt. - - {2 Graph close, account replacement and shutdown} - - Host authentication, deletion policy/confirmation and client lifetime remain - outside coordination. Before retiring an execution owner, plan its exact - change, apply the returned transition and retain the returned handoff ID: - {[ - let prepare_change - (state : AC.t ref) (runner : AR.t) (change : A.host_change) = - match AC.prepare_host_change !state change with - | Error error -> Error error - | Ok (handoff, transition) -> - state := transition.next; - List.iter (AR.submit runner) transition.instructions; - Ok handoff - - let consume_safe_receipt - (state : AC.t ref) (runner : AR.t) (receipt : A.handoff_receipt) = - match AC.authorize_host_change !state receipt with - | Error error -> Error error - | Ok transition -> - state := transition.next; - List.iter (AR.submit runner) transition.instructions; - Ok () - ]} - - These local helpers are only the coordination portion. The host must join - other existing owners, use the exact current handoff, and respect required - confirmation. [Safe_to_proceed] is a scoped cleanup receipt; the consume call - above does not itself prove action completion. Only the matching admitted - completion publishes [Host_change_applied]. Do not send a second legacy - Close/Delete after that action ACK. Unknown outcomes retain their original - owner and permission; a stale receipt cannot authorize a reopened graph. - - [AR.shutdown runner] is nonblocking: [Draining] requires continued delivery of - real cleanup callbacks and ACKs. Revisit after pump progress until [Stopped] - or an observable failure; no busy loop and no premature shared-client close. - All owners, including retired hosts, must reach their own fence. Worker - [W.Shutdown] and Sync [S.Shutdown] still produce work that must be interpreted. - [SR.shutdown] alone is not a checked join receipt. [Websocket_closed] reports - connection closure; [Websocket_retired] reports actual resource/callback - retirement. A shutdown flag or empty mailbox cannot substitute for these facts. - - {1 Observation and helpers} - - {2 Reading progress without creating another owner} - - [AC.view], [AC.bootstrap], [AC.projection], [AC.reply], [AC.intent_progress], - [AC.import_progress] and [AC.handoff] expose coordination facts. [AC.query_origin] - retains the exact session/budget actually sent, not a later selected session. - [AC.current_presentation] and [AC.revalidate_reply] guard deferred business - consumption. [AC.changefeed_updates] and [AC.asset_updates] report accepted - original facts; they do not grant a second invalidation or cleanup owner. - - [W.view] exposes Worker graph state and the Sync projection; [S.state] is an - observation of the owned Sync reducer, not a mutable replacement for it. - Preserve concrete [A.Db_error] causes and typed failure/unknown distinctions - across boundaries instead of turning every result into a generic string. - - {2 Contract and implementation reading map} - - Public contracts used above: - {ul - {- {{!Journal_worker_coordination.Worker_api}[app/spec/worker_coordination/worker_api.mli]}: - typed requests, authority, durable intent, canonical readback, - observations/resources and handoff.} - {- {{!Journal_worker_coordination.Worker_pure_coordinator}[app/spec/worker_coordination/worker_pure_coordinator.mli]}: - events, private transitions, request planning, reconciliation and ownership.} - {- {{!Journal_worker_coordination.Worker_effect_runner}[app/spec/worker_coordination/worker_effect_runner.mli]}: - borrowed client, actual response routing, interpretation and nonblocking - retirement.} - {- {{!Logseq_db_worker_pure_reducer.Core}[logseq_db_worker/spec/pure_reducer/core.mli]} and - {{!Logseq_db_worker_effect_runner.Effect_runner}[logseq_db_worker/spec/effect_runner/effect_runner.mli]}: - Worker events, execution, scoped reads/durable writes and Sync delegation.} - {- {{!Logseq_sync_pure_reducer.Core}[logseq_sync/spec/pure_reducer/core.mli]}, - {{!Logseq_sync_pure_reducer.Sync_protocol}[sync_protocol.mli]} in the same - directory, and - {{!Logseq_sync_effect_runner.Effect_runner}[logseq_sync/spec/effect_runner/effect_runner.mli]}: - Sync policy, network protocol and I/O ownership.} - {- {{!Logseq_overlay_db.Types}[logseq_overlay_db/spec/types.mli]} and - {{!Logseq_overlay_db.Database}[database.mli]} in the same directory: - immutable local intent, outbox transitions and authoritative application.}} - - Current implementation seams, intentionally not presented as spec APIs: - {ul - {- [app/application.ml]: bridge/runtime startup, shared pump and business owners.} - {- [app/journal_worker_coordination_host.ml]: serial state installation, - runner binding and delivery of typed replies to business callbacks.} - {- [app/impl/worker_coordination/worker_effect_authority.ml]: checked mapping - between App typed requests and concrete Service requests.} - {- [logseq_db_worker/lui/logseq_db_worker_lui_service.ml]: scoped Service - routing and real coordination registration.} - {- [logseq_db_worker/lib/logseq_db_worker.ml]: lifecycle, event channel, - await-reply wrapper and direct durable-mutation wrapper.} - {- Worker and Sync [lib/pure_reducer/core.ml] and - [lib/effect_runner/effect_runner.ml]: the ordered transitions and effects - described above.}} - - This document introduces no initialize/begin_sync/post_events/sync_now API. - All control flow must continue through the owners and public boundaries that - actually exist. It explains the major paths, not every UI branch, protocol - error, asset codec or storage implementation detail. +(** Journal execution owners. + + {!Journal_worker_coordination.Worker_pure_coordinator} owns typed App requests, + presentation authority, receipts, observations and resource handoffs. Its + runner borrows the shared Worker client; one App pump routes each response. + Install each transition before interpreting its instructions. Deferred UI + consumption must revalidate the original presentation. + + {!Journal_sync_lifecycle.Account} owns account admission and replacement. + {!Journal_sync_lifecycle.Core} assembles each graph's Service, Execute, + transport, crypto, asset and direct database owners. Worker does not embed a + second Sync policy loop. {!Logseq_db_worker_effect_runner.Effect_runner} + obtains the current owner through its services boundary. + + {!Logseq_sync_sync_service.Core} schedules {!Sync_dsl.Core} policy through + {!Execute.Core}. Database work uses the checked database owner; native I/O + uses the transport and asset owners. Local commit, server acceptance, + authoritative application and current-presentation confirmation are separate + facts. Cancellation neither rolls back a durable mutation nor proves cleanup. + + Persist immutable mutation/import identity and payload before first send. + Unknown outcomes retain that original intent. Confirmation is bounded + canonical readback, not another mutation. Observation/changefeed ACKs follow + actual application or explicit retirement of every dependent consumer. + + Graph and account changes retain their exact handoff permission through the + matching completion. A cleanup request, empty mailbox or shutdown flag is not + a physical join receipt. Sync retires before an owned database closes; a + borrowed upload store remains owned by its caller. Retained files and late + acquisitions require their actual release acknowledgements. *) diff --git a/app/spec/sync_lifecycle/account.mli b/app/spec/sync_lifecycle/account.mli new file mode 100644 index 00000000..67cd6d86 --- /dev/null +++ b/app/spec/sync_lifecycle/account.mli @@ -0,0 +1,78 @@ +module Dsl = Sync_dsl.Core +module Graph = Logseq_db_types.Managed_graph + +type account = + { origin : string + ; account : string + ; root : string + } + +type service + +(** Close returns only after physical retirement; failures retain ownership. *) +val service + : set_network:(Dsl.network_gate -> (unit, Dsl.error) result) + -> close:(unit -> (unit, Dsl.error) result) + -> service + +val of_core : Core.t -> service + +type catalog = + account:account + -> check_current:(unit -> (unit, Dsl.error) result) + -> gate:(unit -> Dsl.network_gate) + -> (Graph.t list, Dsl.error) result + +val http_catalog + : backend:Logseq_sync_transport.Http_client.backend + -> token:(account:account -> epoch:int64 -> refresh:bool -> (string, Dsl.error) result) + -> catalog + +val native_catalog + : sw:Eio.Switch.t + -> clock:_ Eio.Time.Mono.t + -> network:_ Eio.Net.t + -> authenticator:X509.Authenticator.t + -> deadlines:Execute.Core.network_deadlines + -> token:(account:account -> epoch:int64 -> refresh:bool -> (string, Dsl.error) result) + -> catalog + +type t + +(** Calls are confined to one Eio domain. Mutations reject reentry; refresh may overlap. + Start errors must leave no upload-store borrowers; success owns graph resources + through checked close. *) +val create + : maximum_services:int + -> catalog:catalog + -> start: + (account:account + -> account_generation:int64 + -> graph_generation:int64 + -> graph:Graph.t + -> upload_store:Logseq_sync_worker_owner.Core.store + -> network:Dsl.network_gate + -> on_change:(Core.Service.status -> unit) + -> (service, Dsl.error) result) + -> delete_local:(account:account -> graph:Graph.t -> (unit, Dsl.error) result) + -> on_change:(Graph.t -> Core.Service.status -> unit) + -> (t, Dsl.error) result + +(** Replacement/logout revoke delivery first. Failed retirement blocks reuse and + keeps the upload store open; retry logout before replacing the account. *) +val replace : t -> account -> (unit, Dsl.error) result + +val logout : t -> (unit, Dsl.error) result +val refresh : t -> (Graph.t list, Dsl.error) result +val catalog : t -> Graph.t list +val restore_catalog : t -> Graph.t list -> (unit, Dsl.error) result +val release_graph : t -> Logseq_db_types.Graph_types.Uuid.t -> (unit, Dsl.error) result + +(** At capacity, physically retires the oldest service before starting another. *) +val select : t -> Logseq_db_types.Graph_types.Uuid.t -> (unit, Dsl.error) result + +val deselect : t -> (unit, Dsl.error) result +val selected : t -> Logseq_db_types.Graph_types.Uuid.t option +val service_count : t -> int +val set_network : t -> Dsl.network_gate -> (unit, Dsl.error) result +val delete_local : t -> Logseq_db_types.Graph_types.Uuid.t -> (unit, Dsl.error) result diff --git a/app/spec/sync_lifecycle/core.mli b/app/spec/sync_lifecycle/core.mli new file mode 100644 index 00000000..3736347f --- /dev/null +++ b/app/spec/sync_lifecycle/core.mli @@ -0,0 +1,56 @@ +module Dsl = Sync_dsl.Core +module Execute = Execute.Core +module Service = Logseq_sync_sync_service.Core + +type t + +(** Borrows an open or activation database owner and upload store through checked close. + Subscribes after activation. Ingress binds asynchronously; callbacks must not block. *) +val start_native + : sw:Eio.Switch.t + -> clock:_ Eio.Time.Mono.t + -> network:_ Eio.Net.t + -> authenticator:X509.Authenticator.t + -> config:Service.config + -> deadlines:Execute.network_deadlines + -> liveness:Execute.websocket_liveness + -> cleanup_seconds:float + -> root:string + -> database_owner:Execute.database_owner + -> upload_store:Logseq_sync_worker_owner.Core.store + -> namespace:string + -> encrypted:bool + -> token:(epoch:int64 -> refresh:bool -> (string, Dsl.error) result) + -> on_change:(Service.status -> unit) + -> on_error:(Dsl.error -> unit) + -> (t, Dsl.error) result + +(** Owns an existing mirror or activates an absent one from the native snapshot ledger. + Close retires Sync before closing the database; the upload store remains borrowed. *) +val start_owned_native + : sw:Eio.Switch.t + -> clock:_ Eio.Time.Mono.t + -> network:_ Eio.Net.t + -> authenticator:X509.Authenticator.t + -> config:Service.config + -> deadlines:Execute.network_deadlines + -> liveness:Execute.websocket_liveness + -> cleanup_seconds:float + -> root:string + -> dependencies:Logseq_overlay_db.Database.dependencies + -> graph_name:string + -> upload_store:Logseq_sync_worker_owner.Core.store + -> namespace:string + -> encrypted:bool + -> token:(epoch:int64 -> refresh:bool -> (string, Dsl.error) result) + -> on_change:(Service.status -> unit) + -> on_error:(Dsl.error -> unit) + -> (t, Dsl.error) result + +val identity : t -> Dsl.identity +val database_owner : t -> Execute.database_owner +val worker_owner : t -> Logseq_sync_worker_owner.Core.t +val ingress : t -> (Service.ingress, Dsl.error) result +val projection_changed : t -> (unit, Dsl.error) result +val set_network : t -> Dsl.network_gate -> (unit, Dsl.error) result +val close : t -> unit diff --git a/app/spec/sync_lifecycle/dune b/app/spec/sync_lifecycle/dune new file mode 100644 index 00000000..f37c9bb2 --- /dev/null +++ b/app/spec/sync_lifecycle/dune @@ -0,0 +1,7 @@ +(library + (name journal_sync_lifecycle) + (public_name logseq_journal.sync_lifecycle) + (modules core account) + (virtual_modules core account) + (default_implementation journal_sync_lifecycle_impl) + (libraries eio execute sync_dsl logseq_sync.sync_service logseq_sync.transport logseq_sync.worker_owner logseq_overlay_db logseq_db_types x509)) diff --git a/apple-tests/application-platform/JournalApplicationPlatformTests.swift b/apple-tests/application-platform/JournalApplicationPlatformTests.swift index 599bdaed..58a65d43 100644 --- a/apple-tests/application-platform/JournalApplicationPlatformTests.swift +++ b/apple-tests/application-platform/JournalApplicationPlatformTests.swift @@ -9,16 +9,74 @@ import Foundation enum JournalCognitoSession { static let changed = Notification.Name("journal.tests.auth.changed") } struct JournalEnvironmentSample: Equatable, Codable {} @MainActor final class PlatformTestAuth: JournalAuthCapability { - func currentUserID() async throws -> String? { "fixture-user" } + var identityFailure: Error? + func currentUserID() async throws -> String? { + if let identityFailure { throw identityFailure } + return "fixture-user" + } + var signOutStarted = false + var signOutCompletion: CheckedContinuation? + var holdSignOut = false func freshIDToken() async throws -> String { "fixture-token" } var signOutFailure: Error? - func signOut() async throws { if let signOutFailure { throw signOutFailure } } + func signOut() async throws { + signOutStarted = true + NotificationCenter.default.post(name: JournalCognitoSession.changed, object: self) + if holdSignOut { await withCheckedContinuation { signOutCompletion = $0 } } + if let signOutFailure { throw signOutFailure } + } } @main struct JournalApplicationPlatformTests { @MainActor static func main() async throws { func packet(_ tag: UInt8) -> Data { var bytes = Data("LJP2".utf8); bytes.append(contentsOf: [2,0,tag,0]); bytes.append(Data(repeating:0,count:24)); return bytes } + for failSignOut in [false, true] { + let auth = PlatformTestAuth() + auth.holdSignOut = true + if failSignOut { auth.signOutFailure = NSError(domain: "test.signout", code: 9) } + var cleared = false + let owner = JournalPlatformServices(auth: auth, account: JournalAccountStore( + load: { nil }, save: { _ in JournalAccountSave(commit: {}, rollback: {}) }, + clear: { cleared = true }), managedSyncOrigin: "https://api.logseq.io") + let bridge = JournalApplicationPlatform(services: owner) + let port = JournalRuntime() + bridge.connect(to: port) + _ = try await owner.response(for: .timelinePresented) + let (notificationCompletions, notificationCompletion) = AsyncStream.makeStream( + bufferingPolicy: .bufferingNewest(1)) + bridge.authenticationNotificationCompletionObserver = { notificationCompletion.yield(()) } + let logout = Task { await bridge.request(packet(10)) } + var notifications = notificationCompletions.makeAsyncIterator() + let notificationCompleted: Void? = await notifications.next() + precondition(notificationCompleted != nil) + bridge.authenticationNotificationCompletionObserver = nil + notificationCompletion.finish() + precondition(auth.signOutStarted && auth.signOutCompletion != nil, + "Notification refresh must complete while native sign-out is suspended") + let notificationIssue = bridge.accountIssue + auth.signOutCompletion?.resume() + let result = await logout.value + precondition(notificationIssue == nil, + "Sign-out notification must not publish identityLookup.unavailable while native revocation is pending") + if failSignOut { + precondition(result?.prefix(4) == Data("LJF1".utf8) && !cleared, + "Real native sign-out failure must still complete as failure without clearing the binding") + auth.identityFailure = NSError(domain: "test.identity", code: 7) + let refresh = bridge.refreshAuthentication() + precondition(refresh != nil) + await refresh?.value + precondition(bridge.accountIssue?.kind == .unavailable, + "Real current identity lookup errors after failed sign-out must remain visible") + } else { + precondition(result?.prefix(4) == Data("LJP2".utf8) && cleared, + "Successful sign-out must acknowledge native and local cleanup") + precondition(bridge.authenticationRequired && bridge.localAccountAvailable == false + && bridge.accountIssue == nil && !owner.onlineAllowed, + "Completed sign-out must remain a clean signed-out state") + } + bridge.disconnect() + } let faultAuth = PlatformTestAuth() faultAuth.signOutFailure = NSError(domain:"SECRET token path",code:1) let faultServices = JournalPlatformServices(auth:faultAuth, account:JournalAccountStore( diff --git a/apple-tests/image-preview/README.md b/apple-tests/image-preview/README.md index 61ed48ff..0facd464 100644 --- a/apple-tests/image-preview/README.md +++ b/apple-tests/image-preview/README.md @@ -33,19 +33,13 @@ Reuse the warm-start fixture's binary PPX compilation pipeline for with the generated `.cmo` file. Avoid pretty-printing a generated AST back into source: compiler-only identifiers may not survive that conversion. Pass the copied support directory, PNG directory and, for an older fixture, its existing journal-page UUID. The -generator uses public storage sessions and `Core.step` → `Effect_runner.submit` -asset requests. It starts a disposable Python HTTPS server on an ephemeral -loopback port, generates a temporary certificate with OpenSSL, and fetches the -three PNGs through the production asset runner. Its network provider routes -asset connections to that server while retaining the fixture's -`https://api.logseq.io` account identity; the production resolver may resolve -that hostname, but no production server receives an HTTP request. The temporary -certificate is trusted only by this runner, and the server and certificate files -are removed when preparation ends. Python 3 with `ssl` and OpenSSL with -`req -addext` must be available on `PATH`. +generator uses public storage sessions, `Logseq_sync_native_crypto.Asset_codec` +verification and `Logseq_sync_assets.Cache` publication. It retains the fixture's +`https://api.logseq.io` account identity without opening network connections or +using a temporary server, certificate or trust override. -The generator releases each lease, closes the cache scope, and reads the -persisted asset back through the public cache requests before emitting PPX +The generator releases each handle, closes and reopens the cache, and checks the +persisted asset's path and exact bytes through the public cache API before emitting PPX metadata. It adds three ordered direct image children to the known fixture root and updates only the synthetic journal day and title. Regenerate the synthetic date after midnight. Run it only against the separate disposable support copy. diff --git a/apple-tests/image-preview/generate_assets.ml b/apple-tests/image-preview/generate_assets.ml index 6ca3615e..3688144d 100644 --- a/apple-tests/image-preview/generate_assets.ml +++ b/apple-tests/image-preview/generate_assets.ml @@ -1,12 +1,10 @@ -(* Extend an isolated warm-start fixture through public storage and Core/Effect_runner APIs only. +(* Extend an isolated warm-start fixture through public storage and asset-cache APIs. Arguments: SUPPORT_ROOT PNG_DIRECTORY [JOURNAL_PAGE_UUID]. *) module G = Logseq_db_types.Graph_types module Storage = Logseq_db_storage.Logseq_sqlite_storage module Session = Logseq_db_storage.Storage_session -module Runner = Logseq_sync_effect_runner.Effect_runner -module Codec = Logseq_sync_effect_runner.Asset_codec +module Codec = Logseq_sync_native_crypto.Asset_codec module Asset = Logseq_db_types.Asset_descriptor -module Core = Logseq_sync_pure_reducer.Core let get = function | Ok value -> value @@ -56,358 +54,58 @@ let session = ~callbacks:(Storage.connection_callbacks connection) ;; -(* A disposable HTTPS endpoint keeps the warm fixture's real cache identity while - routing asset connections to loopback. Production DNS may resolve the origin; - no production server receives a request and no trust store changes. *) -let rec wait_process pid = - try snd (Unix.waitpid [] pid) with - | Unix.Unix_error (Unix.EINTR, _, _) -> wait_process pid -;; - let with_asset_publication publish = - let directory = Filename.temp_file "journal-image-assets-" "" in - Sys.remove directory; - Unix.mkdir directory 0o700; - let certificate = Filename.concat directory "certificate.pem" - and private_key = Filename.concat directory "private-key.pem" in - let server_pid = ref None in + let module Cache = Logseq_sync_assets.Cache in + let scope : Logseq_sync_assets.Namespace.graph_scope = + { account = + { managed_sync_origin = Uri.of_string "https://api.logseq.io" + ; user_id = "user-1" + } + ; graph_id + } + in + let open_cache () = + Cache.create + ~root:support + ~scope + ~budget_bytes:268435456L + ~maximum_file_bytes:(8 * 1024 * 1024) + |> get + in + let cache = ref (open_cache ()) in Fun.protect - ~finally:(fun () -> - Option.iter - (fun pid -> - (try Unix.kill pid Sys.sigterm with - | Unix.Unix_error (Unix.ESRCH, _, _) -> ()); - ignore (wait_process pid)) - !server_pid; - Sys.readdir directory - |> Array.iter (fun name -> Sys.remove (Filename.concat directory name)); - Unix.rmdir directory) + ~finally:(fun () -> Cache.close !cache) (fun () -> - let log = - Unix.openfile - (Filename.concat directory "server.log") - [ Unix.O_CREAT; Unix.O_WRONLY ] - 0o600 - in - Fun.protect - ~finally:(fun () -> Unix.close log) - (fun () -> - let args = - [| "openssl" - ; "req" - ; "-x509" - ; "-newkey" - ; "rsa:2048" - ; "-nodes" - ; "-keyout" - ; private_key - ; "-out" - ; certificate - ; "-days" - ; "1" - ; "-subj" - ; "/CN=api.logseq.io" - ; "-addext" - ; "subjectAltName=DNS:api.logseq.io" - |] - in - let pid = Unix.create_process "openssl" args Unix.stdin log log in - if wait_process pid <> Unix.WEXITED 0 - then failwith "Fixture TLS certificate failed"; - let server = - {|import http.server, pathlib, ssl, sys -images = pathlib.Path(sys.argv[1]) -class Handler(http.server.BaseHTTPRequestHandler): - def do_GET(self): - prefix = '/assets/60000000-0000-4000-8000-000000000001/92000000-0000-4000-8000-' - if not self.path.startswith(prefix) or self.headers.get('Authorization') != 'Bearer image-fixture': - self.send_error(404); return - names = {'000000000001.png':'red.png', '000000000002.png':'blue.png', '000000000003.png':'green.png'} - name = names.get(self.path[len(prefix):]) - if name is None: - self.send_error(404); return - data = (images/name).read_bytes() - self.send_response(200) - self.send_header('Content-Type', 'application/octet-stream') - self.send_header('Content-Length', str(len(data))) - self.end_headers() - self.wfile.write(data) - def log_message(self, *_): pass -server = http.server.HTTPServer(('127.0.0.1', 0), Handler) -context = ssl.SSLContext(ssl.PROTOCOL_TLS_SERVER) -context.load_cert_chain(sys.argv[2], sys.argv[3]) -server.socket = context.wrap_socket(server.socket, server_side=True) -print(server.server_port, flush=True) -server.serve_forever() -|} - in - let read_port, write_port = Unix.pipe () in - let pid = - Unix.create_process - "python3" - [| "python3" - ; "-B" - ; "-c" - ; server - ; Sys.argv.(2) - ; certificate - ; private_key - |] - Unix.stdin - write_port - log - in - server_pid := Some pid; - Unix.close write_port; - let channel = Unix.in_channel_of_descr read_port in - let port = - Fun.protect - ~finally:(fun () -> close_in channel) - (fun () -> input_line channel |> int_of_string) - in - let cert = - In_channel.with_open_bin certificate In_channel.input_all - |> X509.Certificate.decode_pem - |> get - in - Eio_main.run (fun environment -> - Eio.Switch.run (fun sw -> - let real_network = Eio.Stdenv.net environment in - let module Network = struct - type t = unit - type tag = [ `Generic ] - - let listen _ ~reuse_addr:_ ~reuse_port:_ ~backlog:_ ~sw:_ _ = - failwith "Fixture network cannot listen" - ;; - - let connect () ~sw = function - | `Tcp (_, 443) -> - (Eio.Net.connect - ~sw - real_network - (`Tcp (Eio.Net.Ipaddr.V4.loopback, port)) - :> tag Eio.Net.stream_socket_ty Eio.Resource.t) - | _ -> failwith "Fixture network rejects nonlocal destinations" - ;; - - let datagram_socket _ ~reuse_addr:_ ~reuse_port:_ ~sw:_ _ = - failwith "Fixture network cannot use datagrams" - ;; - - let getaddrinfo () ~service host = - if host <> "api.logseq.io" || service <> "443" - then failwith "Fixture network rejects external DNS"; - [ `Tcp (Eio.Net.Ipaddr.V4.loopback, port) ] - ;; - - let getnameinfo _ _ = failwith "Fixture network cannot reverse DNS" - end - in - let network = Eio.Resource.T ((), Eio.Net.Pi.network (module Network)) in - let tasks = Queue.create () - and posted = Queue.create () in - let dependencies = - Runner.dependencies - ~runtime: - (Runner.runtime - ~fork:(fun ~sw:_ task -> Queue.add task tasks) - ~sleep:(fun _ -> ()) - |> get) - ~transport: - (Runner.transport - ~tls_authenticator: - (Runner.tls_authenticator - (X509.Authenticator.chain_of_trust - (* Unix.time truncates to the certificate's not-before second. *) - ~time:(fun () -> Ptime.of_float_s (Unix.gettimeofday ())) - [ cert ])) - ~network - ~clock:(Eio.Stdenv.clock environment) - ~websocket_liveness:Runner.Disabled - |> get) - ~local_store: - (Runner.local_store - ~asset_cache_budget_bytes:268435456L - ~asset_maximum_file_bytes:(8 * 1024 * 1024) - ~application_support_directory:support - () - |> get) - ~artifact_store: - (Runner.artifact_store - ~staging_directory:(Filename.concat directory "snapshot-staging") - |> get) - ~secrets: - (Runner.secrets - ~unlock_private_key: - (fun - ~managed_sync_origin:_ - ~user_id:_ - ~password:_ - ~private_key_package:_ -> Error "unused") - ~unlock_graph_key: - (fun - ~managed_sync_origin:_ ~user_id:_ ~encrypted_graph_key:_ -> - Error "unused") - ~load_wrapped_graph_key: - (fun - ~managed_sync_origin:_ ~user_id:_ ~graph_id:_ -> - Error (Runner.Wrapped_graph_key_unavailable "unused")) - ~verify_and_save_wrapped_graph_key: - (fun - ~managed_sync_origin:_ - ~user_id:_ - ~graph_id:_ - ~encrypted_graph_key:_ -> Error "unused") - ~delete_account_secrets:(fun ~managed_sync_origin:_ ~user_id:_ -> - Ok ()) - |> get) - ~crypto: - (Runner.crypto - ~encrypt_aes_gcm:(fun ~key:_ ~plaintext:_ -> Error "unused") - ~decrypt_aes_gcm:(fun ~key:_ ~iv:_ ~ciphertext:_ -> - Error "unused") - |> get) - ~id_token_provider: - (Runner.id_token_provider - ~acquire:(fun _ -> Ok "image-fixture") - ~invalidate:(fun _ ~token:_ -> ())) - |> get - in - let runner = - Runner.create ~sw dependencies ~post:(fun event -> - Queue.add event posted) - |> get - in - Fun.protect - ~finally:(fun () -> Runner.shutdown runner) - (fun () -> - let origin = Uri.of_string "https://api.logseq.io" in - let initial = - Core.config - ~managed_sync_origin:origin - ~limits: - (Core.limits - ~maximum_response_bytes: - Logseq_db_types.Limits.maximum_response_bytes - ~maximum_artifact_bytes:(8 * 1024 * 1024) - ~submission_batch_size:32 - |> get) - |> get - |> Core.initial - |> get - in - let initial = (Core.step initial (Core.Sign_in_requested - {account_generation=0; presentation_generation=0})).next in - let authenticated = - Core.step - initial - (Core.Account_authenticated { user_id = Some "user-1" }) - in - let graph : Core.graph = - { graph_id - ; name = "Image fixture" - ; schema = { major = 65; minor = 33; exact = true } - ; encrypted = false - } - in - let catalog = - List.find_map - (function - | Core.Run (Core.Request (ticket, Core.Fetch_catalog _)) -> - Some - (Core.step - authenticated.next - (Core.Runner_completed - (Core.Completion (ticket, Ok [ graph ])))) - | _ -> None) - authenticated.effects - |> Option.get - in - let state = - ref (Core.step catalog.next (Core.Graph_selected graph_id)).next - in - let scope = Core.admitted_graph_scope !state |> Option.get in - let consume outputs effects = - List.fold_left - (fun outputs -> function - | Core.Run runnable -> - Runner.submit runner runnable; - outputs - | Core.Publish (Core.Asset_finished output) -> - output :: outputs - | _ -> outputs) - outputs - effects - in - let rec pump outputs = - match Queue.take_opt tasks with - | Some task -> - task (); - pump outputs - | None -> - (match Queue.take_opt posted with - | None -> List.rev outputs - | Some event -> - let transition = Core.step !state event in - state := transition.next; - pump (consume outputs transition.effects)) - in - let sequence = ref 0 in - let request action = - incr sequence; - let transition = - Core.step - !state - (Core.Asset_requested - { scope - ; operation = "image-" ^ string_of_int !sequence - ; action - }) - in - state := transition.next; - match pump (consume [] transition.effects) with - | [ output ] -> get output.Core.result - | _ -> failwith "Fixture asset request did not complete" - in - publish (fun ~asset ~version ~bytes -> - let handle = - match - request - (Core.Fetch_asset - { asset - ; version - ; maximum_plaintext_bytes = String.length bytes - }) - with - | Core.Asset_downloaded handle -> handle - | _ -> failwith "Fixture asset download failed" - in - let lease, path = - match request (Core.Retain_asset_file handle) with - | Core.Asset_retained (Some resource) -> resource - | _ -> failwith "Fixture asset lease failed" - in - if In_channel.with_open_bin path In_channel.input_all <> bytes - then failwith "Fixture asset cache bytes changed"; - ignore (request (Core.Release_asset_file lease)); - ignore (request (Core.Release_asset_file handle)); - ignore (request Core.Close_asset_scope); - let restored = - match request (Core.Check_asset_cache (asset, version)) with - | Core.Asset_cached (Some handle) -> handle - | _ -> failwith "Fixture asset cache did not survive reopening" - in - let lease, restored_path = - match request (Core.Retain_asset_file restored) with - | Core.Asset_retained (Some resource) -> resource - | _ -> failwith "Fixture restored asset lease failed" - in - if restored_path <> path - then failwith "Fixture asset path changed after reopening"; - ignore (request (Core.Release_asset_file lease)); - ignore (request (Core.Release_asset_file restored)); - path)))))) + publish (fun ~asset ~version ~bytes -> + let crypto : Codec.crypto = + { encrypt = (fun ~key:_ ~plaintext:_ -> Error "Unencrypted fixture") + ; decrypt = (fun ~key:_ ~iv:_ ~ciphertext:_ -> Error "Unencrypted fixture") + } + in + let verified = + Codec.decode + ~maximum_plaintext_bytes:(String.length bytes) + ~crypto + ~key:None + ~expected:version + bytes + |> get + in + let handle = + Cache.publish !cache ~asset ~current:(fun () -> true) ~verified |> get + in + let path = Cache.path !cache handle |> Option.get in + if In_channel.with_open_bin path In_channel.input_all <> bytes + then failwith "Fixture asset cache bytes changed"; + Cache.release !cache handle; + Cache.close !cache; + cache := open_cache (); + let restored = Cache.lookup !cache ~asset ~version |> get |> Option.get in + let restored_path = Cache.path !cache restored |> Option.get in + if restored_path <> path + then failwith "Fixture asset path changed after reopening"; + Cache.release !cache restored; + path)) ;; type metadata = diff --git a/apple-tests/warm-start/generate_fixture.ml b/apple-tests/warm-start/generate_fixture.ml index 32abfdcc..e65060c3 100644 --- a/apple-tests/warm-start/generate_fixture.ml +++ b/apple-tests/warm-start/generate_fixture.ml @@ -1,5 +1,5 @@ module Graph = Logseq_db_types.Graph_types -module Core = Logseq_sync_pure_reducer.Core +module Core = Logseq_sync_domain.Core module Storage = Logseq_db_storage.Logseq_sqlite_storage module Session = Logseq_db_storage.Storage_session diff --git a/logseq_db_storage/lib/asset_upload_store.ml b/logseq_db_storage/lib/asset_upload_store.ml index 2132c81e..8cbcc74e 100644 --- a/logseq_db_storage/lib/asset_upload_store.ml +++ b/logseq_db_storage/lib/asset_upload_store.ml @@ -36,6 +36,9 @@ let initialize_database db = db "CREATE INDEX IF NOT EXISTS asset_upload_scope ON asset_upload_intents(origin, \ account, graph, operation)") in + let* () = check db (Sqlite3.exec db "DROP INDEX IF EXISTS asset_upload_original") in + let* () = check db (Sqlite3.exec db + "CREATE INDEX IF NOT EXISTS asset_upload_original_v2 ON asset_upload_intents(origin,account,graph,json_extract(CASE WHEN json_valid(payload) THEN payload ELSE '{}' END,'$.asset'),json_extract(CASE WHEN json_valid(payload) THEN payload ELSE '{}' END,'$.version.checksum'),operation) WHERE json_extract(CASE WHEN json_valid(payload) THEN payload ELSE '{}' END,'$.phase[0]') <> 'Cancelled'") in Mutation_effect_store.initialize_database db |> Result.map_error input_store_error) ;; @@ -150,6 +153,21 @@ let list db ~origin ~account ~graph ~after ~limit = rows []) ;; +let find_original db ~origin ~account ~graph ~asset ~checksum = + statement db + "SELECT operation,payload FROM asset_upload_intents INDEXED BY asset_upload_original_v2 WHERE origin=? AND account=? AND graph=? AND json_extract(CASE WHEN json_valid(payload) THEN payload ELSE '{}' END,'$.asset')=? AND json_extract(CASE WHEN json_valid(payload) THEN payload ELSE '{}' END,'$.version.checksum')=? AND json_extract(CASE WHEN json_valid(payload) THEN payload ELSE '{}' END,'$.phase[0]') <> 'Cancelled' ORDER BY operation LIMIT 1" + (fun stmt -> + bind stmt [TEXT origin; TEXT account; TEXT (Uuid.to_string graph); TEXT (Uuid.to_string asset); TEXT checksum]; + match Sqlite3.step stmt with + | DONE -> Ok None + | ROW -> + let* intent = decode (Sqlite3.column_text stmt 1) in + if intent.origin = origin && intent.account = account && intent.graph = graph && intent.asset = asset && intent.version.checksum = checksum && intent.phase <> Intent.Cancelled && Uuid.to_string intent.operation_id = Sqlite3.column_text stmt 0 + then Ok (Some intent) + else Error "Original upload checkpoint scope mismatch" + | rc -> let* () = check db rc in Error "Unexpected original upload query result") +;; + let transact db action = let* () = check db (Sqlite3.exec db "BEGIN IMMEDIATE") in match protect action with diff --git a/logseq_db_storage/lib/asset_upload_store.mli b/logseq_db_storage/lib/asset_upload_store.mli index 6c4c1c2d..cf37f52f 100644 --- a/logseq_db_storage/lib/asset_upload_store.mli +++ b/logseq_db_storage/lib/asset_upload_store.mli @@ -21,6 +21,15 @@ val list -> limit:int -> (Logseq_db_types.Asset_upload_intent.t list, string) result +val find_original + : Sqlite3.db + -> origin:string + -> account:string + -> graph:Logseq_db_types.Graph_types.Uuid.t + -> asset:Logseq_db_types.Graph_types.Uuid.t + -> checksum:string + -> (Logseq_db_types.Asset_upload_intent.t option, string) result + (** Retires checkpoints and complete inputs atomically. The caller supplies namespaces derived from this account and its existing graph identities; source bytes cannot establish account ownership on their own. *) diff --git a/logseq_db_types.opam b/logseq_db_types.opam index 3da66a39..3fa85c50 100644 --- a/logseq_db_types.opam +++ b/logseq_db_types.opam @@ -12,5 +12,6 @@ depends: [ "dune" {= "3.23.1"} "ppx_deriving_yojson" {>= "3.9.1"} "yojson" {= "3.0.0"} + "uri" ] build: [["dune" "build" "-p" name "-j" jobs]] diff --git a/logseq_db_types/lib/dune b/logseq_db_types/lib/dune index fcc0072a..63566190 100644 --- a/logseq_db_types/lib/dune +++ b/logseq_db_types/lib/dune @@ -1,6 +1,6 @@ (library (name logseq_db_types) (public_name logseq_db_types) - (libraries ppx_deriving_yojson.runtime yojson) + (libraries ppx_deriving_yojson.runtime uri yojson) (preprocess (pps ppx_deriving_yojson))) diff --git a/logseq_db_types/lib/sync_account.ml b/logseq_db_types/lib/sync_account.ml new file mode 100644 index 00000000..895345c7 --- /dev/null +++ b/logseq_db_types/lib/sync_account.ml @@ -0,0 +1,199 @@ +type graph_id = Graph_types.Uuid.t [@@deriving yojson] + +type cache_schema = Managed_graph.schema = + { major : int + ; minor : int + ; exact : bool + } +[@@deriving yojson] + +type graph = Managed_graph.t = + { graph_id : graph_id + ; name : string + ; schema : cache_schema + ; encrypted : bool + } +[@@deriving yojson] + +type account_generation = int + +type graph_generation = int + +type connection_generation = int + +type presentation_generation = int + +type lifecycle_generation = int64 + +type account_scope = + { managed_sync_origin : Uri.t + ; user_id : string + ; account_generation : account_generation + ; presentation_generation : presentation_generation + ; lifecycle_generation : lifecycle_generation + } + +type graph_scope = + { account : account_scope + ; graph_id : graph_id + ; graph_generation : graph_generation + } + +type sync_phase = + | Offline + | Connecting + | Pulling + | Submitting + | Current + | Paused + | Failed + +type startup_failure_stage = + | During_authentication + | During_catalog + | During_local_restore + | During_bootstrap + | During_e2ee + +type local_unavailability = + | No_local_account + | Local_catalog_unavailable + | Local_mirror_unavailable + | Local_key_unavailable + | Local_restore_failed + +type local_access = + | Local_pending + | Local_available + | Local_unavailable of local_unavailability + +type startup_network_phase = + | Local_only + | Online_after_timeline + | Online_by_user + +type startup_facts = + { authenticated : bool + ; local_access : local_access + ; network_phase : startup_network_phase + ; catalog_loading : bool + ; awaiting_selection : bool + ; restoring_local : bool + ; bootstrapping : bool + ; awaiting_e2ee_password : bool + ; failure : startup_failure_stage option + ; account_generation : account_generation + ; graph_generation : graph_generation + ; presentation_generation : presentation_generation + } + +type catalog_cache = + { cache_user_id : string + ; cache_graphs : graph list + ; cache_selected_graph : graph_id option + ; cache_resident_graphs : graph_id list option [@default None] + } +[@@deriving yojson] + +type startup_scope = + { account_generation : account_generation + ; presentation_generation : presentation_generation + } + +type timeline_scope = + { startup : startup_scope + ; graph_id : graph_id + ; graph_generation : graph_generation + } + +type local_deletion_stage = + | Closing_graph + | Deleting_mirror + | Clearing_selection + +type local_deletion = + | Deletion_in_progress of local_deletion_stage + | Deletion_failed of local_deletion_stage + +type unlock_challenge = + { challenge_scope : graph_scope + ; challenge_id : int + ; encrypted_graph_key : string + ; private_key_package : string + } + +type unlock_stage = Private_key of int | Graph_key of int | Attachment + +type unlock_submission = + { challenge : unlock_challenge + ; operation_id : int + ; stage : unlock_stage + } + +type unlock_attached = { submission : unlock_submission; attached_scope : graph_scope } + +type unlock_recovery = + | Retry_password of unlock_challenge * string + | Retry_preparation of string + +type unlock_phase = + | Unlock_inactive + | Unlock_challenge of unlock_challenge + | Unlock_running of unlock_submission + | Unlock_bootstrapping of unlock_submission + | Unlock_presentable of unlock_attached + | Unlock_failed of unlock_recovery + | Unlock_cancelled of unlock_submission + +let unlock_challenge = function + | Unlock_challenge challenge | Unlock_failed (Retry_password (challenge, _)) -> Some challenge + | _ -> None +;; + +let equal_unlock_challenge a b = + a.challenge_id = b.challenge_id && a.challenge_scope = b.challenge_scope +let unlock_challenge_scope challenge = challenge.challenge_scope +let unlock_submission_challenge submission = submission.challenge +let equal_unlock_submission a b = + a.operation_id = b.operation_id && equal_unlock_challenge a.challenge b.challenge +;; +let unlock_attached_submission receipt = receipt.submission +let unlock_attached_scope receipt = receipt.attached_scope + +(** Presentation only: retry eligibility is still determined by the original outcome. + Unknown and security failures require attention even when transport retries them. *) +type error_attention = Transient_connection | Requires_attention + +type snapshot = + { sync_phase : sync_phase + ; catalog : graph list + ; selected_graph : graph_id option + ; resident_graphs : graph_id list + ; applied_server_t : Server_cursor.t option + ; timeline_presentation_pending : bool + ; unlock : unlock_phase + ; startup : startup_facts + ; last_error : string option + ; last_error_attention : error_attention option + ; local_deletion : local_deletion option + } + +type diagnostic_group = + { title : string + ; entries : (string * string) list + } + +type diagnostics = { groups : diagnostic_group list } + +type state = + { snapshot : snapshot + ; diagnostics : diagnostics + } + +type coordination_state = + { account : account_scope option + ; desired_residents : graph_id list + ; residents : (graph_scope * state) list + ; selected : timeline_scope option + ; deletion : (graph_scope * local_deletion) option + } diff --git a/logseq_db_types/lib/sync_account.mli b/logseq_db_types/lib/sync_account.mli new file mode 100644 index 00000000..37a50954 --- /dev/null +++ b/logseq_db_types/lib/sync_account.mli @@ -0,0 +1,192 @@ +type graph_id = Graph_types.Uuid.t [@@deriving yojson] + +type cache_schema = Managed_graph.schema = + { major : int + ; minor : int + ; exact : bool + } +[@@deriving yojson] + +type graph = Managed_graph.t = + { graph_id : graph_id + ; name : string + ; schema : cache_schema + ; encrypted : bool + } +[@@deriving yojson] + +type account_generation = int + +type graph_generation = int + +type connection_generation = int + +type presentation_generation = int + +type lifecycle_generation = int64 + +type account_scope = + { managed_sync_origin : Uri.t + ; user_id : string + ; account_generation : account_generation + ; presentation_generation : presentation_generation + ; lifecycle_generation : lifecycle_generation + } + +type graph_scope = + { account : account_scope + ; graph_id : graph_id + ; graph_generation : graph_generation + } + +type sync_phase = + | Offline + | Connecting + | Pulling + | Submitting + | Current + | Paused + | Failed + +type startup_failure_stage = + | During_authentication + | During_catalog + | During_local_restore + | During_bootstrap + | During_e2ee + +type local_unavailability = + | No_local_account + | Local_catalog_unavailable + | Local_mirror_unavailable + | Local_key_unavailable + | Local_restore_failed + +type local_access = + | Local_pending + | Local_available + | Local_unavailable of local_unavailability + +type startup_network_phase = + | Local_only + | Online_after_timeline + | Online_by_user + +type startup_facts = + { authenticated : bool + ; local_access : local_access + ; network_phase : startup_network_phase + ; catalog_loading : bool + ; awaiting_selection : bool + ; restoring_local : bool + ; bootstrapping : bool + ; awaiting_e2ee_password : bool + ; failure : startup_failure_stage option + ; account_generation : account_generation + ; graph_generation : graph_generation + ; presentation_generation : presentation_generation + } + +type catalog_cache = + { cache_user_id : string + ; cache_graphs : graph list + ; cache_selected_graph : graph_id option + ; cache_resident_graphs : graph_id list option [@default None] + } +[@@deriving yojson] + +type startup_scope = + { account_generation : account_generation + ; presentation_generation : presentation_generation + } + +type timeline_scope = + { startup : startup_scope + ; graph_id : graph_id + ; graph_generation : graph_generation + } + +type local_deletion_stage = + | Closing_graph + | Deleting_mirror + | Clearing_selection + +type local_deletion = + | Deletion_in_progress of local_deletion_stage + | Deletion_failed of local_deletion_stage + +type unlock_challenge = + { challenge_scope : graph_scope + ; challenge_id : int + ; encrypted_graph_key : string + ; private_key_package : string + } + +type unlock_stage = Private_key of int | Graph_key of int | Attachment + +type unlock_submission = + { challenge : unlock_challenge + ; operation_id : int + ; stage : unlock_stage + } + +type unlock_attached = { submission : unlock_submission; attached_scope : graph_scope } + +type unlock_recovery = + | Retry_password of unlock_challenge * string + | Retry_preparation of string + +type unlock_phase = + | Unlock_inactive + | Unlock_challenge of unlock_challenge + | Unlock_running of unlock_submission + | Unlock_bootstrapping of unlock_submission + | Unlock_presentable of unlock_attached + | Unlock_failed of unlock_recovery + | Unlock_cancelled of unlock_submission + +val unlock_challenge : unlock_phase -> unlock_challenge option +val equal_unlock_challenge : unlock_challenge -> unlock_challenge -> bool +val unlock_challenge_scope : unlock_challenge -> graph_scope +val unlock_submission_challenge : unlock_submission -> unlock_challenge +val equal_unlock_submission : unlock_submission -> unlock_submission -> bool +val unlock_attached_submission : unlock_attached -> unlock_submission +val unlock_attached_scope : unlock_attached -> graph_scope + +(** Presentation only: retry eligibility is still determined by the original outcome. + Unknown and security failures require attention even when transport retries them. *) +type error_attention = Transient_connection | Requires_attention + +type snapshot = + { sync_phase : sync_phase + ; catalog : graph list + ; selected_graph : graph_id option + ; resident_graphs : graph_id list + ; applied_server_t : Server_cursor.t option + ; timeline_presentation_pending : bool + ; unlock : unlock_phase + ; startup : startup_facts + ; last_error : string option + ; last_error_attention : error_attention option + ; local_deletion : local_deletion option + } + +type diagnostic_group = + { title : string + ; entries : (string * string) list + } + +type diagnostics = { groups : diagnostic_group list } + +type state = + { snapshot : snapshot + ; diagnostics : diagnostics + } + +type coordination_state = + { account : account_scope option + ; desired_residents : graph_id list + ; residents : (graph_scope * state) list + ; selected : timeline_scope option + ; deletion : (graph_scope * local_deletion) option + } diff --git a/logseq_db_worker/lib/dune b/logseq_db_worker/lib/dune index 0e579ad9..87662133 100644 --- a/logseq_db_worker/lib/dune +++ b/logseq_db_worker/lib/dune @@ -13,5 +13,8 @@ logseq_db_worker.effect_runner logseq_db_worker.pure_reducer logseq_overlay_db - logseq_sync.pure_reducer + execute + sync_dsl + logseq_sync.worker_owner + logseq_sync.domain logseq_sync.asset_transfer uri)) diff --git a/logseq_db_worker/lib/effect_runner/dune b/logseq_db_worker/lib/effect_runner/dune index 071b5493..ec50fedc 100644 --- a/logseq_db_worker/lib/effect_runner/dune +++ b/logseq_db_worker/lib/effect_runner/dune @@ -17,7 +17,10 @@ logseq_db_worker.contract logseq_db_worker.pure_reducer logseq_overlay_db - logseq_sync.pure_reducer + execute + sync_dsl + logseq_sync.worker_owner + logseq_sync.domain logseq_sync.asset_transfer melange-transit-core melange-transit-native uri diff --git a/logseq_db_worker/lib/effect_runner/effect_runner.ml b/logseq_db_worker/lib/effect_runner/effect_runner.ml index b4e764b9..bc6b195a 100644 --- a/logseq_db_worker/lib/effect_runner/effect_runner.ml +++ b/logseq_db_worker/lib/effect_runner/effect_runner.ml @@ -9,7 +9,7 @@ module Core = Logseq_db_worker_pure_reducer.Core module Database = Logseq_overlay_db.Database module Overlay = Logseq_overlay_db.Types module Graph = Logseq_db_types.Graph_types -module Sync = Logseq_sync_pure_reducer.Core +module Sync = Logseq_sync_domain.Core module Transit = Transit_core.Json module Transit_codec = Transit_native.Transit.Json @@ -308,17 +308,20 @@ type runtime = ; sleep : float -> unit } -type sync_runner = - { submit : Sync.runner_effect -> unit +type services = + { post : Core.lifecycle_intent -> unit + ; owner : scope:Sync.graph_scope -> (Sync_dsl.Core.identity * Logseq_sync_worker_owner.Core.t, string) result + ; asset : Sync.asset_request -> Sync.asset_result + ; await_io : scope:Sync.effect_scope -> (unit, string) result ; shutdown : unit -> unit - ; retire_account : account:Sync.account_scope -> (unit,string)result + ; retire_account : account:Sync.account_scope -> (unit, string) result } type dependencies = { runtime : runtime ; config : Logseq_db_worker_contract.Config.t ; overlay : Database.dependencies - ; sync_runner : sync_runner + ; services : services ; publish : Core.output -> unit } @@ -330,12 +333,8 @@ type waiter = ; resolve : Read.answer Eio.Promise.u } -type asset_waiter = +type asset_diagnostic = { request : Sync.asset_request - ; resolve : Sync.asset_output Eio.Promise.u - ; mutable delivered : Sync.asset_output option - ; mutable accepted : bool - ; mutable reclaimed : bool ; mutable diagnostic : Logseq_db_types.Failure_diagnostic.t option } @@ -348,11 +347,6 @@ type notice_diagnostic = ; ticket_id : int } -type protected_waiter = - { protected_request : Sync.protected_request - ; protected_resolve : Sync.protected_output Eio.Promise.u - } - type change_window = { encoded_bytes : int ; id : string @@ -366,6 +360,8 @@ type change_window = type database_session = { graph_scope : Sync.graph_scope ; database : Database.t + ; database_owner : Execute.Core.database_owner + ; borrowed : bool ; subscription : Database.subscription ; mutable generation : string ; mutable next_window : int @@ -405,22 +401,28 @@ type host_copy={mutable copy_handoff:Graph.Uuid.t;copy_registration:int64;copy_o copy_preparation:Database.prepared_mirror_deletion;copy_generation:Overlay.mirror_generation; mutable copy_bound:Sync.mirror_deletion option} +type imported_resource = + { imported_scope : Sync.graph_scope + ; imported_retain : unit -> ((string * string) option, string) result + ; imported_release : unit -> (unit, string) result + } + type t = { sw : Eio.Switch.t + ; imported_resources : (string, imported_resource) Hashtbl.t + ; mutable next_imported_resource : int64 ; dependencies : dependencies ; post : Core.event -> unit ; upload_operations : (Upload.ticket, unit Eio.Promise.u) Hashtbl.t - ; protected_waiters : (string, protected_waiter) Hashtbl.t - ; asset_waiters : (string, asset_waiter) Hashtbl.t + ; asset_waiters : (string, asset_diagnostic) Hashtbl.t ; mutable notice_diagnostics : notice_diagnostic list - ; asset_operations : (Logseq_sync_pure_reducer.Asset_transfer.ticket, string) Hashtbl.t + ; asset_operations : (Logseq_sync_asset_transfer.Core.ticket, string) Hashtbl.t ; mutable next_asset_operation : int64 ; upload_staging_lock : Eio.Mutex.t ; mutable staging_reconciled : Sync.graph_scope list - ; local_imports : (Sync.graph_scope * Graph.Uuid.t * string, Graph.Uuid.t) Hashtbl.t ; recovery_current : Core.upload_recovery_ticket -> bool ; upload_current : Upload.ticket -> bool - ; asset_current : Logseq_sync_pure_reducer.Asset_transfer.ticket -> bool + ; asset_current : Logseq_sync_asset_transfer.Core.ticket -> bool ; observation_slots : (int64 * Graph.Uuid.t, observation_slot) Hashtbl.t ; mutable observation_registry : Core.Observation_registry.t ; mutable orphan_cleanup : Core.Orphan_cleanup.t @@ -429,9 +431,8 @@ type t = ; mutable next_asset_barrier : int64 ; asset_cleanup_jobs : (int64,asset_cleanup_job) Hashtbl.t ; asset_cleanup_barriers : (int64,unit Eio.Promise.u) Hashtbl.t - ; sync_io_waiters : (int64,Sync.effect_scope * (unit,string)result Eio.Promise.u)Hashtbl.t - ; cancelled_asset_jobs : (Logseq_sync_pure_reducer.Asset_transfer.ticket,asset_cleanup_job)Hashtbl.t - ; asset_result_sources : ((Sync.graph_scope*string),Logseq_sync_pure_reducer.Asset_transfer.ticket)Hashtbl.t + ; cancelled_asset_jobs : (Logseq_sync_asset_transfer.Core.ticket,asset_cleanup_job)Hashtbl.t + ; asset_result_sources : ((Sync.graph_scope*string),Logseq_sync_asset_transfer.Core.ticket)Hashtbl.t ; asset_cleanup_stopped : unit Eio.Promise.t * unit Eio.Promise.u ; resource_cleanup_owners : ((Sync.graph_scope * string),asset_cleanup_job) Hashtbl.t ; resource_release_waiters : ((Sync.graph_scope * string),unit Eio.Promise.u list) Hashtbl.t @@ -453,32 +454,31 @@ type t = ; mutable authorized_graphs : Authorized_graphs.t ; mutable host_copy : host_copy option ; mutable deletion_owner : Sync.mirror_deletion option + ; mutable sync_store : Sqlite3.db option } let runtime ~fork ~sleep = Ok { fork; sleep } -let sync_runner ~submit ~shutdown ~retire_account () = { submit; shutdown;retire_account } - -let dependencies ~runtime ~config ~overlay ~sync_runner ~publish = - Ok { runtime; config; overlay; sync_runner; publish } +let dependencies ~runtime ~config ~overlay ~services ~publish = + Ok { runtime; config; overlay; services; publish } ;; let create ~sw dependencies ~post ~recovery_current ~upload_current ~asset_current = Ok { sw + ; imported_resources = Hashtbl.create 16 + ; next_imported_resource = 0L ; dependencies ; post ; asset_current ; upload_current ; recovery_current ; upload_operations = Hashtbl.create 32 - ; protected_waiters = Hashtbl.create 32 ; asset_waiters = Hashtbl.create 32 ; notice_diagnostics = [] ; asset_operations = Hashtbl.create 8 ; next_asset_operation = 0L ; upload_staging_lock = Eio.Mutex.create () ; staging_reconciled = [] - ; local_imports = Hashtbl.create 32 ; observation_slots = Hashtbl.create 32 ; observation_registry = Core.Observation_registry.empty ; orphan_cleanup = Core.Orphan_cleanup.empty @@ -487,7 +487,6 @@ let create ~sw dependencies ~post ~recovery_current ~upload_current ~asset_curre ; next_asset_barrier = 0L ; asset_cleanup_jobs = Hashtbl.create 16 ; asset_cleanup_barriers = Hashtbl.create 16 - ; sync_io_waiters = Hashtbl.create 4 ; cancelled_asset_jobs = Hashtbl.create 16 ; asset_result_sources = Hashtbl.create 16 ; asset_cleanup_stopped = Eio.Promise.create () @@ -511,6 +510,7 @@ let create ~sw dependencies ~post ~recovery_current ~upload_current ~asset_curre ; authorized_graphs = Authorized_graphs.empty ; host_copy = None ; deletion_owner = None + ; sync_store = None } ;; @@ -520,14 +520,17 @@ let with_upload_store t action = try let fd = Unix.openfile path [ Unix.O_RDWR; Unix.O_CREAT ] 0o600 in Unix.close fd; - let db = Sqlite3.db_open path in - Fun.protect - ~finally:(fun () -> (if not (Sqlite3.db_close db) then report_warning "journal: phase=cleanup operation=closeSqlite scope=temporaryStorageHandle code=closePending recovery=reopen\n" )) - (fun () -> - Unix.chmod path 0o600; - Result.bind - (Logseq_db_storage.Asset_upload_store.initialize_database db) - (fun () -> action db)) + let opened = match t.sync_store with + | Some db -> Ok db + | None -> + let db = Sqlite3.db_open path in + Unix.chmod path 0o600; + (match Logseq_db_storage.Asset_upload_store.initialize_database db with + | Ok () -> t.sync_store <- Some db; Ok db + | Error message -> + if Sqlite3.db_close db then Error message + else (t.sync_store <- Some db; Error (message ^ "; upload store close failed"))) in + Result.bind opened action |> Result.map (fun value -> let fd = Unix.openfile root [ Unix.O_RDONLY ] 0 in Fun.protect ~finally:(fun () -> Unix.close fd) (fun () -> Unix.fsync fd); @@ -536,6 +539,23 @@ let with_upload_store t action = | exn -> Error (Printexc.to_string exn) ;; +let sync_upload_store t = + if t.stopped then Error "The Worker upload store is retired." + else with_upload_store t (fun db -> Ok (Logseq_sync_worker_owner.Core.store db)) +;; + +let worker_owner t ~(scope : Sync.graph_scope) = + if t.stopped then Error "The Worker is retired." + else Result.bind (t.dependencies.services.owner ~scope) (fun ((identity, _) as owner) -> + if identity.Sync_dsl.Core.origin = Uri.to_string scope.account.managed_sync_origin + && identity.account = scope.account.user_id + && identity.graph = Graph.Uuid.to_string scope.graph_id + && identity.account_generation = Int64.of_int scope.account.account_generation + && identity.graph_generation = Int64.of_int scope.graph_generation + then Ok owner + else Error "The direct Sync owner belongs to a different graph incarnation.") +;; + let asset_request t ~scope action = Hashtbl.replace t.resource_scopes scope (); let operation = "worker-asset:" ^ Int64.to_string t.next_asset_operation in @@ -543,7 +563,17 @@ let asset_request t ~scope action = { Sync.scope; operation; action } ;; -let post_asset_request t request = t.post (Core.Sync_event (Sync.Asset_requested request)) +let post_asset_request t request = + t.dependencies.runtime.fork ~sw:t.sw (fun () -> + match t.dependencies.services.asset request with + | Ok _ -> () + | Error failure -> + let detail = Sync.asset_failure_cause failure in + t.dependencies.publish (Core.Diagnostic + (match detail with + | Some detail -> Logseq_db_types.Failure_diagnostic.signal_prefix + ^ Logseq_db_types.Failure_diagnostic.encode detail + | None -> "Direct Sync asset cleanup failed"))) let send_asset t ~scope action = Eio.Cancel.protect (fun () -> post_asset_request t (asset_request t ~scope action)) @@ -563,13 +593,6 @@ let reclaim_asset_output t (output : Sync.asset_output) = Option.iter (send_asset t ~scope:output.request.scope) action ;; -let reclaim_waiter t waiter = - if (not waiter.accepted) && not waiter.reclaimed - then ( - waiter.reclaimed <- true; - Option.iter (reclaim_asset_output t) waiter.delivered) -;; - let current_graph_scope t (scope : Sync.graph_scope) = match t.latest_snapshot with | Some snapshot -> not t.stopped @@ -589,56 +612,125 @@ let publish_scoped_asset_detail t scope detail = try t.dependencies.publish (Core.Diagnostic (D.signal_prefix ^ D.encode detail)) with _ -> ()) ;; +let native_asset_failure = function + | Sync_dsl.Core.Network_failure _ -> Sync.Asset_network + | Authentication_failure _ -> Asset_authentication + | Locked -> Asset_locked + | Storage_failure source -> + Sync.Asset_with_cause (Sync.Asset_invalid_content source, + Logseq_db_types.Failure_diagnostic.of_source + ~operation:"assetCache" ~reason:"persistenceFailed" source) + | Cancelled | Stale_context | Network_closed -> Asset_cancelled + | error -> Asset_invalid_content (Sync_dsl.Core.error_message error) +;; + +let native_asset_call t ~scope action = + let module Owner = Logseq_sync_worker_owner.Core in + let module Dsl = Sync_dsl.Core in + let ( let* ) = Result.bind in + let* identity, owner = worker_owner t ~scope + |> Result.map_error (fun message -> Sync.Asset_invalid_content message) in + let* operation = Graph.Uuid.of_string + (Printf.sprintf "00000000-0000-4000-8000-%012Lx" t.next_asset_operation) + |> Result.map_error (fun message -> Sync.Asset_invalid_content message) in + let run : type value. value Dsl.assets -> (value, string) result = fun action -> + Owner.run_assets owner ~identity ~operation action |> Result.map_error Dsl.error_message in + let capacity () = + if Hashtbl.length t.imported_resources >= 2048 || t.next_imported_resource = Int64.max_int + then Error "Consumer resource capacity exceeded." else Ok () in + let register ~path ~retain ~release = + t.next_imported_resource <- Int64.succ t.next_imported_resource; + let handle = "worker-file:" ^ Int64.to_string t.next_imported_resource in + Hashtbl.add t.imported_resources handle + { imported_scope = scope; imported_retain = retain; imported_release = release }; + handle, path in + let file_result file = + let* path = Owner.asset_file_path owner file |> Result.map_error Dsl.error_message in + let rec retain () = + let* () = capacity () in + let* lease = run (Dsl.Retain file) in + Ok (Some (register ~path ~retain ~release:(fun () -> run (Dsl.Release_lease lease)))) in + Ok (register ~path ~retain ~release:(fun () -> run (Dsl.Release_file file))) in + let descriptor asset version = + Logseq_db_types.Asset_descriptor.create ~uuid:asset ~source:(Managed (Some version)) + ~current_checksum:None ~size:None ~dimensions:None + |> Result.map_error (fun message -> Sync.Asset_invalid_content message) in + match action with + | Sync.Prune_asset_staging _ -> + Owner.run_assets owner ~identity ~operation Dsl.Reconcile_staging + |> Result.map (fun count -> Sync.Asset_pruned count) + |> Result.map_error native_asset_failure + | Sync.Check_asset_cache (asset,version) -> + let* () = capacity () |> Result.map_error (fun message -> Sync.Asset_invalid_content message) in + let* descriptor = descriptor asset version in + let* file = Owner.run_assets owner ~identity ~operation (Dsl.Lookup descriptor) + |> Result.map_error native_asset_failure in + (match file with + | None -> Ok (Sync.Asset_cached None) + | Some file -> file_result file |> Result.map (fun (handle,_) -> Sync.Asset_cached (Some handle)) + |> Result.map_error (fun message -> Sync.Asset_invalid_content message)) + | Fetch_asset {asset;version;_} -> + let* () = capacity () |> Result.map_error (fun message -> Sync.Asset_invalid_content message) in + let* descriptor = descriptor asset version in + let* result = Owner.run_assets owner ~identity ~operation (Dsl.Download descriptor) + |> Result.map_error native_asset_failure in + (match result with + | Dsl.Available file -> file_result file |> Result.map (fun (handle,_) -> Sync.Asset_downloaded handle) + |> Result.map_error (fun message -> Sync.Asset_invalid_content message) + | Waiting_remote | Missing_local -> Error Sync.Asset_not_found) + | _ -> Error (Sync.Asset_invalid_content "Unsupported native asset action") +;; + let asset_call ?(started = fun _ -> ()) ?on_diagnostic t ~scope action = let request = asset_request t ~scope action in - let promise, resolve = Eio.Promise.create () in - let waiter = - { request; resolve; delivered = None; accepted = false; reclaimed = false; diagnostic = None } - in + let waiter = { request; diagnostic = None } in Hashtbl.add t.asset_waiters request.operation waiter; Fun.protect - ~finally:(fun () -> - Hashtbl.remove t.asset_waiters request.operation; - if not waiter.accepted - then ( - reclaim_waiter t waiter; - send_asset t ~scope (Sync.Cancel_asset_operation request.operation))) + ~finally:(fun () -> Hashtbl.remove t.asset_waiters request.operation) (fun () -> - started request.operation; - post_asset_request t request; - let output = Eio.Promise.await promise in - waiter.accepted <- true; - (match output.Sync.result, waiter.diagnostic with - | Error Sync.Asset_cancelled, _ | Ok _, _ | _, None -> () - | Error _, Some detail -> - (try (Option.value on_diagnostic ~default:(publish_scoped_asset_detail t scope)) detail with _ -> ())); - waiter.diagnostic <- None; - output.Sync.result) + started request.operation; + let result = match action with + | Sync.Release_asset_file handle when Hashtbl.mem t.imported_resources handle -> + let resource = Hashtbl.find t.imported_resources handle in + if resource.imported_scope <> scope then Error Sync.Asset_cancelled else + (match resource.imported_release () with + | Error message -> Error (Sync.Asset_invalid_content message) + | Ok () -> Hashtbl.remove t.imported_resources handle; Ok Sync.Asset_unit) + | Sync.Retain_asset_file handle when Hashtbl.mem t.imported_resources handle -> + let resource = Hashtbl.find t.imported_resources handle in + if resource.imported_scope <> scope then Error Sync.Asset_cancelled else + resource.imported_retain () |> Result.map (fun file -> Sync.Asset_retained file) + |> Result.map_error (fun message -> Sync.Asset_invalid_content message) + | Sync.Check_asset_cache _ | Fetch_asset _ -> native_asset_call t ~scope action + | Sync.Prune_asset_staging _ -> + native_asset_call t ~scope action + | _ -> t.dependencies.services.asset request in + (match result, waiter.diagnostic with + | Error Sync.Asset_cancelled, _ | Ok _, _ | _, None -> () + | Error _, Some detail -> + (Option.value on_diagnostic ~default:(publish_scoped_asset_detail t scope)) detail); + result) ;; let protected_call t action = - let key = - match action with - | Sync.Encrypt_values (key, _) | Decrypt_value (key, _) -> key - in - let operation = "worker-protected:" ^ Int64.to_string t.next_asset_operation in - t.next_asset_operation <- Int64.succ t.next_asset_operation; - let request = - { Sync.protected_operation = operation - ; protected_scope = Sync.graph_key_handle_scope key - ; protected_action = action - } - in - let promise, resolve = Eio.Promise.create () in - Hashtbl.add - t.protected_waiters - operation - { protected_request = request; protected_resolve = resolve }; - Fun.protect - ~finally:(fun () -> Hashtbl.remove t.protected_waiters operation) - (fun () -> - t.post (Core.Sync_event (Sync.Protected_requested request)); - (Eio.Promise.await promise).Sync.protected_result) + let key = match action with + | Sync.Encrypt_values (key, _) | Decrypt_value (key, _) -> key in + let scope = Sync.graph_key_handle_scope key in + let result = Result.bind (worker_owner t ~scope) (fun (identity, owner) -> + let module Owner = (val Logseq_sync_worker_owner.Core.worker_owner owner : Execute.Core.WORKER_OWNER) in + let operation = Graph.Uuid.of_string + (Printf.sprintf "00000000-0000-4000-8000-%012Lx" t.next_asset_operation) in + t.next_asset_operation <- Int64.succ t.next_asset_operation; + Result.bind operation (fun operation -> Owner.with_scope ~identity ~operation (fun owned -> + match action with + | Sync.Encrypt_values (_, plaintexts) -> + Result.map (fun values -> Sync.Encrypted_values (Rrbvec.to_list values)) + (Owner.crypto owned (Sync_dsl.Core.Encrypt_values (Rrbvec.of_list plaintexts))) + | Sync.Decrypt_value (_, ciphertext) -> + Result.map (fun plaintext -> Sync.Decrypted_value plaintext) + (Owner.crypto owned (Sync_dsl.Core.Decrypt_value ciphertext))) + |> Result.map_error Sync_dsl.Core.error_message)) in + Result.map_error (fun message -> Sync.Effect_failed message) result ;; let encrypt_protected_values t key plaintexts = @@ -779,62 +871,15 @@ let reconcile_staging t (scope : Sync.graph_scope) ~current = else if List.mem scope t.staging_reconciled then Ok () else ( - Hashtbl.filter_map_inplace (fun (owner,_,_) operation -> if owner=scope then None else Some operation) t.local_imports; - let keep = - with_upload_store t (fun db -> - let rec read after reversed = - if (not (current ())) || t.stopped - then Error "Stale staging recovery" - else - Result.bind - (Logseq_db_storage.Asset_upload_store.list - db - ~origin:(Uri.to_string scope.account.managed_sync_origin) - ~account:scope.account.user_id - ~graph:scope.graph_id - ~after - ~limit:128) - (fun intents -> - let reversed = - List.fold_left - (fun files intent -> - let module I = Logseq_db_types.Asset_upload_intent in - if intent.I.phase <> I.Cancelled then - Hashtbl.replace t.local_imports - (scope, intent.asset, intent.version.checksum) intent.operation_id; - intent.staged_file :: files) - reversed - intents - in - match List.rev intents with - | [] -> Ok (List.rev reversed) - | last :: _ when List.length intents = 128 -> - read - (Some last.Logseq_db_types.Asset_upload_intent.operation_id) - reversed - | _ -> Ok (List.rev reversed)) - in - read None []) - in - Result.bind keep (fun keep -> - if (not (current ())) || t.stopped - then Error "Stale staging recovery" - else ( - match asset_call t ~scope (Sync.Prune_asset_staging keep) with - | Ok (Sync.Asset_pruned _) -> - if current () - then ( - t.staging_reconciled <- scope :: t.staging_reconciled; - Ok ()) - else Error "Stale staging recovery" - | Error failure -> Error (asset_failure_message failure) - | Ok _ -> Error "Unexpected staging recovery completion"))) -;; - -let save_upload t intent expected = - with_upload_store t (fun db -> - Logseq_db_storage.Asset_upload_store.save db ~expected intent) - |> Result.map_error (fun message -> Upload.Persistence_failed message) + match asset_call t ~scope (Sync.Prune_asset_staging []) with + | Ok (Sync.Asset_pruned _) -> + if current () + then ( + t.staging_reconciled <- scope :: t.staging_reconciled; + Ok ()) + else Error "Stale staging recovery" + | Error failure -> Error (asset_failure_message failure) + | Ok _ -> Error "Unexpected staging recovery completion") ;; let worker_error ~code message = @@ -1109,6 +1154,29 @@ let publish_projection_change t session = function (session.graph_scope, Worker_pure_data.Resync_required {generation=generation;reason=resync_reason reason})) ;; +let attach_database_owner t ~(scope:Sync.graph_scope) owner = + match Hashtbl.find_opt t.attachments scope with + | Some database_id -> Ok(Core.database_opened ~database_id ~graph_id:scope.graph_id) + | None -> + Result.bind (Execute.Core.database_if_open owner |> Result.map_error Sync_dsl.Core.error_message) (function + | None -> Error "The Sync database is not active yet." + | Some database -> + match Database.listen database with + | Error error -> Error(Logseq_db_worker_contract.Error.message(listen_error error)) + | Ok(subscription,snapshot) -> + let version=Database.snapshot_version snapshot in + Database.release_snapshot snapshot; + let database_id="lifecycle-" ^ Int64.to_string t.next_database_id in + t.next_database_id<-Int64.succ t.next_database_id; + let session={graph_scope=scope;database;database_owner=owner;borrowed=true;subscription; + generation=Overlay.Generation.to_string version.generation;next_window=1; + acknowledged_through="change-window:v1:0";windows=Rrbvec.empty;changefeed=Core.Changefeed_owner.empty} in + match Database.activate_subscription subscription ~notify:(publish_projection_change t session) with + | Error error -> Database.unlisten subscription;Error(Logseq_db_worker_contract.Error.message(listen_error error)) + | Ok () -> Hashtbl.add t.databases database_id session;Hashtbl.add t.attachments scope database_id; + Ok(Core.database_opened ~database_id ~graph_id:scope.graph_id)) +;; + let report_current_graph t (scope : Sync.graph_scope) code = if current_graph_scope t scope then t.dependencies.publish (Core.Diagnostic code) @@ -1203,8 +1271,8 @@ let publish_asset_notice t scope notice output = (match status with Upload.Uploaded | Cancelled -> true | _ -> false) | Asset_availability {asset;availability;_} -> Some (Download_notice asset), (match availability with - Logseq_sync_pure_reducer.Asset_transfer.Failed {retry_scheduled=false;_} -> true | _ -> false), - (match availability with Logseq_sync_pure_reducer.Asset_transfer.Ready _ -> true | _ -> false) + Logseq_sync_asset_transfer.Core.Failed {retry_scheduled=false;_} -> true | _ -> false), + (match availability with Logseq_sync_asset_transfer.Core.Ready _ -> true | _ -> false) | Asset_demand_accepted _ | Asset_backpressure _ | Asset_capacity_available -> None, false, false in let detail = Option.bind key (fun key -> let found = List.find_opt (fun item -> item.scope=scope && item.key=key) t.notice_diagnostics in @@ -1310,7 +1378,7 @@ let close_database_by_id ?scope t database_id = |Some error->Error error |None-> Database.unlisten session.subscription; - let result = Database.close session.database in + let result = if session.borrowed then Ok () else Database.close session.database in (match result with | Ok () -> () | Error error -> @@ -1338,6 +1406,13 @@ let close_database_by_id ?scope t database_id = result) ;; +let detach_database_owner t ~scope = + match Hashtbl.find_opt t.attachments scope with + | None -> Ok () + | Some database_id -> close_database_by_id ~scope t database_id + |> Result.map_error Logseq_db_worker_contract.Error.message +;; + let close_attached t = match t.attached with | None -> Ok () @@ -1910,6 +1985,13 @@ let attached_session t scope = Option.bind (Hashtbl.find_opt t.attachments scope) (Hashtbl.find_opt t.databases) ;; +let database_owner t ~scope = + if t.stopped then Error "The Worker is retired." + else match attached_session t scope with + | None -> Error "The scoped database is no longer attached." + | Some session -> Ok session.database_owner +;; + let retire_changefeed t ~registration = Hashtbl.iter(fun _ session->session.changefeed<-Core.Changefeed_owner.retire session.changefeed ~registration) t.databases ;; @@ -2390,7 +2472,6 @@ let deletion_authorized t (request : Sync.mirror_deletion) = snapshot.startup.authenticated && snapshot.startup.account_generation = request.account.account_generation && snapshot.selected_graph <> Some request.graph_id - && snapshot.local_deletion = Some (Sync.Deletion_in_progress Sync.Deleting_mirror) && Authorized_graphs.mem (Graph.Uuid.to_string request.graph_id) t.authorized_graphs | None -> false ;; @@ -2407,7 +2488,7 @@ let report_asset_diagnostic t ~scope ~operation ~request_operation detail = && request.graph_id=scope.graph_id) t.deletion_owner in match Hashtbl.find_opt t.asset_waiters request_operation with | Some waiter when current && waiter.request.scope=scope - && not waiter.accepted && waiter.delivered=None && D.operation detail=operation -> + && D.operation detail=operation -> waiter.diagnostic <- Some detail | Some _ | None -> () ;; @@ -2794,6 +2875,8 @@ let handle_sync_worker_effect t = function let session = { graph_scope = request.scope ; database + ; database_owner = Execute.Core.database_owner database + ; borrowed = false ; subscription ; generation = Overlay.Generation.to_string version.generation ; next_window = 1 @@ -2970,6 +3053,21 @@ let run_request t (type a) (request : a Core.runner_request) | Handle_sync_worker_effect worker_effect -> handle_sync_worker_effect t worker_effect ;; +let delete_local_copy t ?prepared_handoff ~(account:Sync.account_scope) ~graph_id () = + let request:Sync.mirror_deletion={account;graph_id;scope=Sync.effect_scope_of_account account} in + if t.deletion_owner<>None then Error "Another local deletion is in progress." else + let bound=match prepared_handoff with + | None -> Ok () + | Some handoff_id -> bind_host_copy t ~handoff_id request |> Result.map_error Logseq_db_worker_contract.Error.message in + Result.bind bound(fun()-> + t.deletion_owner<-Some request; + Fun.protect ~finally:(fun()->t.deletion_owner<-None)(fun()-> + match handle_sync_worker_effect t (Sync.Delete_mirror request) with + | Ok {event=Some(Sync.Mirror_deleted(_,result));_}->result + | Ok _ -> Error "Local deletion returned an invalid completion." + | Error error -> Error(Logseq_db_worker_contract.Error.message error))) +;; + let request_operation : type a. a Core.runner_request -> string = function | Core.Execute_request _ -> "workerOperation" | Close_database _ -> "closeDatabase" @@ -3038,157 +3136,98 @@ let submit_worker t (Core.Request(ticket,request) as packed) = | Error message -> post_worker_result t ticket request (Error (effect_error message)) ;; -let rec upload_failure = function - | Sync.Asset_with_cause (failure, _) -> upload_failure failure - | Asset_network -> Upload.Network - | Asset_authentication | Asset_locked -> Authentication - | Asset_missing_source | Asset_not_found -> Missing_source - | Asset_size_rejected -> Size_rejected - | Asset_revoked_access -> Revoked_access - | Asset_checksum_mismatch | Asset_storage_full | Asset_cancelled - | Asset_invalid_content _ -> Invalid_content -;; - -let run_upload t (context : Sync.asset_context) instruction = - let current ticket = (not t.stopped) && t.upload_current ticket - && not(Core.Writer_fence.fenced t.writer_fence context.scope) in - let on_diagnostic ticket detail = - if current ticket then queue_notice_diagnostic t context.scope - (Upload_notice ticket.Upload.operation) ticket.id detail in - let database ticket f = - if not (current ticket) - then Error Upload.Invalid_content - else ( - match attached_session t ticket.Upload.scope with - | None -> - on_diagnostic ticket (failure_detail ~operation:"assetUpload" ~reason:"databaseClosed" ()); - Error Upload.Invalid_content - | Some session -> f session.database) - in - let finish ticket result success = - if current ticket - then ( - if Result.is_ok result then clear_notice_diagnostic t context.scope (Upload_notice ticket.Upload.operation); - t.post - (Core.Upload_completed - ( ticket - , match result with - | Ok () -> success - | Error error -> Upload.Failed error ))) - in - let changed () = t.post (Core.Sync_event Sync.Local_outbox_changed) in - match instruction with - | Upload.Cancel_operation ticket -> clear_notice_diagnostic ~ticket_id:ticket.id t context.scope (Upload_notice ticket.operation) - | Release_staging intent -> - let detail = ref None in - let on_failure failure = detail := Some (asset_failure_detail ~operation:"assetRelease" failure) in - let on_diagnostic value = detail := Some value in - (match - let can_release = - if intent.Logseq_db_types.Asset_upload_intent.phase = Cancelled - then Ok () - else ( - match - asset_call - t - ~scope:context.scope - (Sync.Check_asset_cache (intent.asset, intent.version)) - with - | Ok (Sync.Asset_cached (Some handle)) -> - asset_unit ~on_failure ~on_diagnostic t ~scope:context.scope (Sync.Release_asset_file handle) - | _ -> - asset_unit ~on_failure ~on_diagnostic - t - ~scope:context.scope - (Sync.Publish_staged_asset - { asset = intent.asset - ; version = intent.version - ; file = intent.staged_file - })) - in - Result.bind can_release (fun () -> - asset_unit ~on_failure ~on_diagnostic t ~scope:context.scope (Sync.Release_staged_file intent.staged_file)) - with - | Ok () -> () - | Error source -> - let detail = Option.value !detail ~default:(failure_detail ~source ~operation:"assetRelease" ~reason:"cleanupFailed" ()) in - if Logseq_db_types.Failure_diagnostic.reason detail <> "cancelled" then - publish_failure (report_current_graph t context.scope) detail; - report_warning "journal: operation=releaseUploadStaging code=cleanupPending\n" ) - | Persist (ticket, intent, expected) -> - if current ticket then finish ticket (save_upload t intent expected) Upload.Persisted - | Inspect (ticket, intent) -> - if current ticket - then ( - let result = database ticket (fun db -> Upload_database.inspect ~on_diagnostic:(on_diagnostic ticket) db intent) in - if current ticket - then - t.post - (Core.Upload_completed - ( ticket - , match result with - | Ok observation -> Upload.Inspected observation - | Error error -> Upload.Failed error ))) - | Apply_local (ticket, intent) -> - if current ticket - then ( - let result = database ticket (fun db -> Upload_database.apply_local ~on_diagnostic:(on_diagnostic ticket) db intent) in - (match result with - | Ok () -> changed () - | Error _ -> ()); - finish ticket result Upload.Local_applied) - | Apply_metadata (ticket, intent) -> - if current ticket - then ( - let result = database ticket (fun db -> Upload_database.apply_metadata ~on_diagnostic:(on_diagnostic ticket) db intent) in - (match result with - | Ok () -> changed () - | Error _ -> ()); - finish ticket result Upload.Metadata_applied) - | Put (ticket, intent) -> - if current ticket - then ( - let detail = ref None in - finish - ticket - (match - asset_call - ~on_diagnostic:(fun value -> detail := Some value) - t - ~scope:context.scope - (Sync.Put_asset_file - { asset = intent.asset - ; version = intent.version - ; file = intent.staged_file - ; maximum_plaintext_bytes = 8 * 1024 * 1024 - }) - with - | Ok Sync.Asset_unit -> Ok () - | Ok _ -> - on_diagnostic ticket (failure_detail ~operation:"assetUpload" ~reason:"invalidContent" ()); - Error Upload.Invalid_content - | Error failure -> - if current ticket && Sync.asset_failure_kind failure <> Sync.Asset_cancelled then - on_diagnostic ticket (Option.value !detail - ~default:(asset_failure_detail ~operation:"assetUpload" failure)); - Error (upload_failure failure)) - Upload.Put_succeeded) - | Await_publication (ticket, intent) -> - let rec await () = - if current ticket - then ( - match database ticket (fun db -> Upload_database.inspect ~on_diagnostic:(on_diagnostic ticket) db intent) with - | Ok Upload.Publication_acknowledged -> - finish ticket (Ok ()) Upload.Publication_acked - | Ok (Local_present | Metadata_present) -> - t.dependencies.runtime.sleep 0.5; - await () - | Ok (Absent | Entity_cancelled) -> - on_diagnostic ticket (failure_detail ~operation:"assetUpload" ~reason:"missingPrecondition" ()); - finish ticket (Error Upload.Invalid_content) Upload.Publication_acked - | Error error -> finish ticket (Error error) Upload.Publication_acked) - in - await () +let rec upload_owner_diagnostic error = + let module Dsl = Sync_dsl.Core in + let module D = Logseq_db_types.Failure_diagnostic in + let detail ?source reason = failure_detail ?source ~operation:"assetUpload" ~reason () in + match error with + | Dsl.Invalid_input source -> detail ~source "invalidRequest" + | Identity_mismatch source -> detail ~source "ownershipConflict" + | Stale_context -> detail "attachmentStale" + | Network_closed -> detail "networkFailed" + | Network_failure source -> detail ~source "networkFailed" + | Authentication_failure source -> detail ~source "authenticationFailed" + | Locked -> detail "locked" + | Crypto_failure source -> detail ~source "cryptoInvalid" + | Protocol_failure source -> detail ~source "invalidContent" + | Storage_failure source -> detail ~source "persistenceFailed" + | Cancelled -> detail "cancelled" + | Outcome_unknown source -> detail ~source "stateError" + | Cleanup_failure {primary; failures} -> + let causes = Rrbvec.to_seq failures in + (match primary, causes () with + | Some primary, Seq.Cons (cleanup, _) -> + D.with_related (upload_owner_diagnostic primary) (upload_owner_diagnostic cleanup) + | Some primary, Seq.Nil -> upload_owner_diagnostic primary + | None, Seq.Cons (cleanup, rest) -> + let primary = upload_owner_diagnostic cleanup in + (match rest () with + | Seq.Nil -> primary + | Seq.Cons (cleanup, _) -> D.with_related primary (upload_owner_diagnostic cleanup)) + | None, Seq.Nil -> detail "cleanupFailed") + | Capacity_exceeded source -> detail ~source "limitExceeded" + | Sync_read_failure _ -> detail ~source:(Dsl.error_message error) "inspectionFailed" + | Outbox_failure _ -> detail ~source:(Dsl.error_message error) "storageOutboxFailed" + | Authoritative_failure _ -> detail ~source:(Dsl.error_message error) "integrityFailed" + | Activation_failure _ -> detail ~source:(Dsl.error_message error) "restoreFailed" + | Asset_failure diagnostic -> D.of_source ~operation:"assetUpload" ~reason:(D.reason diagnostic) (D.encode diagnostic) + | Unexpected_failure source -> detail ~source "resourceFailed" +;; + +let run_direct_upload t (context : Sync.asset_context) instruction = + let module Dsl = Sync_dsl.Core in + let current ticket = not t.stopped && t.upload_current ticket + && not (Core.Writer_fence.fenced t.writer_fence context.scope) in + let ticket, intent = match instruction with + | Upload.Cancel_operation ticket -> Some ticket, None + | Release_staging intent -> None, Some intent + | Persist (ticket, intent, _) | Inspect (ticket, intent) + | Apply_local (ticket, intent) | Apply_metadata (ticket, intent) + | Put (ticket, intent) | Await_publication (ticket, intent) -> Some ticket, Some intent in + match intent with + | None -> () + | Some _ when Option.fold ~none:false ~some:(fun ticket -> not (current ticket)) ticket -> () + | Some intent -> + let result = Result.bind + (worker_owner t ~scope:context.scope |> Result.map_error (fun message -> Dsl.Storage_failure message)) + (fun (identity, owner) -> + let module Owner = (val Logseq_sync_worker_owner.Core.worker_owner owner : Execute.Core.WORKER_OWNER) in + Owner.with_scope ~identity ~operation:intent.operation_id (fun scope -> + match instruction with + | Upload.Cancel_operation _ -> Owner.cancel scope |> Result.map (fun () -> Upload.Persisted) + | Release_staging _ -> Owner.release_import_staging scope intent |> Result.map (fun () -> Upload.Persisted) + | Persist (_, _, expected) -> Owner.persist_import scope intent ~expected |> Result.map (fun () -> Upload.Persisted) + | Inspect _ -> Owner.inspect_import scope intent |> Result.map (fun observed -> + Upload.Inspected (match observed with + | `Import_absent -> Upload.Absent + | `Local_present -> Local_present + | `Metadata_present -> Metadata_present + | `Publication_acknowledged -> Publication_acknowledged + | `Entity_cancelled -> Entity_cancelled)) + | Apply_local _ -> Owner.apply_local_import scope intent |> Result.map (fun () -> Upload.Local_applied) + | Apply_metadata _ -> Owner.apply_import_metadata scope intent |> Result.map (fun () -> Upload.Metadata_applied) + | Put _ -> Owner.put_import scope intent |> Result.map (fun () -> Upload.Put_succeeded) + | Await_publication _ -> Owner.await_import_publication scope intent |> Result.map (fun () -> Upload.Publication_acked))) in + match ticket, result with + | Some ticket, Ok completion when current ticket -> + clear_notice_diagnostic t context.scope (Upload_notice ticket.operation); + t.post (Core.Upload_completed (ticket, completion)) + | Some ticket, Error error when current ticket -> + let detail = upload_owner_diagnostic error in + queue_notice_diagnostic t context.scope (Upload_notice ticket.operation) ticket.id detail; + let failure = match error with + | Dsl.Network_closed | Network_failure _ -> Upload.Network + | Authentication_failure _ | Locked -> Upload.Authentication + | _ -> Upload.Invalid_content in + t.post (Core.Upload_completed (ticket, Upload.Failed failure)) + | None, Error error -> + publish_failure (report_current_graph t context.scope) + (failure_detail ~source:(Dsl.error_message error) ~operation:"assetRelease" ~reason:"cleanupFailed" ()) + | Some _, (Ok _ | Error _) | None, Ok _ -> () +;; + +let run_upload t context instruction = + run_direct_upload t context instruction ;; let submit_upload t (context : Sync.asset_context) instruction = @@ -3225,7 +3264,7 @@ let submit_upload t (context : Sync.asset_context) instruction = let rec transfer_failure = function | Sync.Asset_with_cause (failure, _) -> transfer_failure failure - | Sync.Asset_network -> Logseq_sync_pure_reducer.Asset_transfer.Network + | Sync.Asset_network -> Logseq_sync_asset_transfer.Core.Network | Asset_not_found | Asset_missing_source -> Not_found | Asset_checksum_mismatch -> Checksum_mismatch | Asset_authentication | Asset_revoked_access -> Authentication @@ -3237,29 +3276,15 @@ let rec transfer_failure = function ;; let local_import t ~(scope : Sync.graph_scope) ~asset ~checksum ~current = - with_staging_lock t (fun () -> - let ( let* ) = Result.bind in - let* () = reconcile_staging t scope ~current in - match Hashtbl.find_opt t.local_imports (scope, asset, checksum) with - | None -> Ok None - | Some operation -> - let* intent = - with_upload_store t (fun db -> - Logseq_db_storage.Asset_upload_store.read db ~operation) - in - Ok - (Option.bind intent (fun (i : Logseq_db_types.Asset_upload_intent.t) -> - if - current () - && (not t.stopped) - && i.origin = Uri.to_string scope.account.managed_sync_origin - && i.account = scope.account.user_id - && i.graph = scope.graph_id - && i.asset = asset - && i.version.checksum = checksum - && i.phase <> Logseq_db_types.Asset_upload_intent.Cancelled - then Some i - else None))) + if t.stopped || not (current ()) then Error "Import authority retired." + else Result.bind (worker_owner t ~scope) (fun (_, owner) -> + Result.bind + (Logseq_db_types.Asset_descriptor.create ~uuid:asset ~source:(Managed None) + ~current_checksum:(Some checksum) ~size:None ~dimensions:None) + (fun descriptor -> + let result = Logseq_sync_worker_owner.Core.original_import owner descriptor + |> Result.map_error Sync_dsl.Core.error_message in + if current () && not t.stopped then result else Error "Import authority retired.")) ;; let lookup_local_asset t ~scope ~asset ~checksum ~current ~diagnostic call = @@ -3408,20 +3433,14 @@ let asset_callback_barrier t = (fun()->Eio.Promise.await promise) (fun()->Eio.Promise.await(fst t.asset_cleanup_stopped))) ;; -let await_sync_io t ~scope = +let await_sync_io t ~(scope : Sync.effect_scope) = if t.stopped then Error "The Worker stopped before Sync I/O settlement." - else if t.next_asset_barrier=Int64.max_int then Error "I/O fence counter exhausted." else - let promise,resolve=Eio.Promise.create() in - t.next_asset_barrier<-Int64.succ t.next_asset_barrier; - let id=t.next_asset_barrier in - Hashtbl.add t.sync_io_waiters id(scope,resolve); - Fun.protect ~finally:(fun()->Hashtbl.remove t.sync_io_waiters id)(fun()-> - t.post(Core.Sync_event(Sync.IO_fence_requested(id,scope))); - Eio.Fiber.first - (fun()->let result=Eio.Promise.await promise in - if t.stopped then Error "The Worker stopped before Sync I/O settlement." else result) - (fun()->Eio.Promise.await(fst t.asset_cleanup_stopped); - Error "The Worker stopped before Sync I/O settlement.")) + else if Hashtbl.fold (fun _ resource live -> live || + (Option.fold ~none:true ~some:((=)resource.imported_scope.account.account_generation) scope.Sync.account_generation + && Option.fold ~none:true ~some:((=)resource.imported_scope.graph_generation) scope.graph_generation)) + t.imported_resources false + then Error "Imported consumer files have not physically released." + else t.dependencies.services.await_io ~scope ;; let quiesce_writers t ~scope = @@ -3480,7 +3499,7 @@ let settle_host_scope t ~(scope:Sync.graph_scope) = let retire_account t ~account= Result.bind (cleanup_account_uploads t account) (fun () -> - t.dependencies.sync_runner.retire_account ~account) + t.dependencies.services.retire_account ~account) ;; let start_asset_cleanup_job ?parent ?(prepare=(fun _->())) t ~scope work = @@ -3503,7 +3522,7 @@ let asset_cleanup_parent t scope handle= ;; let run_asset t (context : Sync.asset_context) instruction = - let module Transfer = Logseq_sync_pure_reducer.Asset_transfer in + let module Transfer = Logseq_sync_asset_transfer.Core in let perform ticket action success = let scope = match ticket with @@ -3646,26 +3665,8 @@ let run_asset t (context : Sync.asset_context) instruction = ;; let submit t = function - | Core.Sync_io_fence_reached(id,scope,result)-> - (match Hashtbl.find_opt t.sync_io_waiters id with - |Some(expected,resolve) when expected=scope->ignore(Eio.Promise.try_resolve resolve result:bool) - |_->()) - | Core.Deliver_protected_result output -> - (match - Hashtbl.find_opt - t.protected_waiters - output.Sync.protected_request.protected_operation - with - | Some waiter when waiter.protected_request = output.protected_request -> - ignore (Eio.Promise.try_resolve waiter.protected_resolve output : bool) - | Some _ | None -> ()) | Core.Deliver_asset_result output -> - (match Hashtbl.find_opt t.asset_waiters output.Sync.request.operation with - | Some waiter when waiter.request = output.request && waiter.delivered = None -> - waiter.delivered <- Some output; - if not (Eio.Promise.try_resolve waiter.resolve output) then reclaim_waiter t waiter - | Some waiter when waiter.request = output.request -> () - | Some _ | None -> reclaim_asset_output t output) + reclaim_asset_output t output | Core.Read_uploads ticket -> if (not t.stopped) && t.recovery_current ticket then ( match reserve_writer t ticket.scope with @@ -3701,25 +3702,8 @@ let submit t = function | Core.Run_worker request -> if not t.stopped then submit_worker t request - | Run_sync (Sync.Request (ticket, Sync.Delete_account_secrets account) as sync_effect) - -> - if not t.stopped - then ( - match - (* This is the serial consumer: never await a staging owner whose asset - completion must return through this same consumer. *) - cleanup_account_uploads ~wait:false t account - with - | Ok () -> t.dependencies.sync_runner.submit sync_effect - | Error message -> - t.post - (Core.Sync_event - (Sync.Runner_completed - (Sync.Completion (ticket, Error (Sync.Effect_failed message)))))) - | Run_sync (Sync.Asset_io _ as sync_effect) -> - t.dependencies.sync_runner.submit sync_effect - | Run_sync sync_effect -> - if not t.stopped then t.dependencies.sync_runner.submit sync_effect + | Publish (Core.Lifecycle_intent intent) -> + if not t.stopped then t.dependencies.services.post intent | Publish (Core.Sync_output (Sync.Session_state_changed (scope,state)) as output) -> if state.snapshot.selected_graph=Some scope.graph_id then Hashtbl.replace t.session_snapshots scope state.snapshot else Hashtbl.remove t.session_snapshots scope; @@ -3765,16 +3749,20 @@ let submit t instruction = let shutdown t = if not t.stopped then ( - t.post (Core.Sync_event Sync.Shutdown); t.stopped <- true; ignore(Eio.Promise.try_resolve(snd t.asset_cleanup_stopped)():bool); t.notice_diagnostics <- []; Hashtbl.iter (fun _ resolve -> ignore (Eio.Promise.try_resolve resolve () : bool)) t.upload_operations; + t.dependencies.services.shutdown (); ignore (close_attached t); Hashtbl.iter (fun id _ -> ignore (close_database_by_id t id)) t.databases; - t.dependencies.sync_runner.shutdown (); + (match t.sync_store with + | None -> () + | Some db -> + if Sqlite3.db_close db then t.sync_store <- None + else t.dependencies.publish (Core.Diagnostic "Worker upload store close failed")); Eio.Mutex.use_rw ~protect:true t.waiter_lock (fun () -> Hashtbl.iter (fun _ (waiter : waiter) -> @@ -3804,146 +3792,23 @@ let release_asset_file t ~scope ~handle = | Ok _ -> report_source_failure t scope "assetRelease" "invalidContent" "Unexpected asset completion" ;; -let prepare_import_unlocked - ~on_diagnostic - t - ~(context : Sync.asset_context) - (request : Logseq_db_types.Asset_import.t) - ~current - = - let module I = Logseq_db_types.Asset_upload_intent in - let module A = Logseq_db_types.Asset_descriptor in - let ( let* ) = Result.bind in - let source_error reason result = Result.map_error (fun source -> - on_diagnostic (failure_detail ~source ~operation:"assetMetadata" ~reason ()); source) result in - let prepare version size staged_file = - I.prepare - ~replace_reference:request.replace_reference - ~operation_id:request.operation - ~origin:(Uri.to_string context.scope.account.managed_sync_origin) - ~account:context.scope.account.user_id - ~graph:context.scope.graph_id - ~asset:request.asset - ~target:request.target - ~local_mutation:request.local_mutation - ~metadata_mutation:request.metadata_mutation - ~title:request.title - ~version - ~size - ~staged_file - in - let* version = source_error "invalidRequest" (A.version ~checksum:(String.make 64 '0') ~file_type:request.file_type) in - let* _ = source_error "invalidRequest" (prepare version 0L "validation.bin") in - let namespace=Logseq_db_worker_contract.Worker_pure_mutation.namespace_for_owner - ~origin:context.scope.account.managed_sync_origin ~user:context.scope.account.user_id ~graph:context.scope.graph_id in - let input_owner:Logseq_db_storage.Asset_upload_store.Inputs.owner={namespace; - origin=Uri.to_string context.scope.account.managed_sync_origin;account=context.scope.account.user_id;graph=context.scope.graph_id} in - let* complete_input=Logseq_db_types.Asset_import.Pure_intent.create ~namespace request - |> Result.map_error(fun _->"Invalid complete import input") |> source_error "invalidRequest" in - if (not (current ())) || t.stopped - then Error "The selected graph is no longer active" - else if not(Core.Orphan_cleanup.acquisition_open t.orphan_cleanup) - then Error "Attachment cleanup is pending. Retry after cleanup succeeds." - else - let* reserved=source_error "persistenceFailed" (Eio.Cancel.protect(fun()-> - with_upload_store t(fun db->Logseq_db_storage.Asset_upload_store.Inputs.reserve db ~owner:input_owner ~input:complete_input))) in - let* () = source_error "restoreFailed" (reconcile_staging t context.scope ~current) in - let* existing = - source_error "persistenceFailed" (with_upload_store t (fun db -> - Logseq_db_storage.Asset_upload_store.read db ~operation:request.operation)) - in - match existing,reserved.prepared with - | None,Some _->source_error "restoreFailed" (Error "The original input was Prepared but its local upload checkpoint is unavailable; it cannot be replayed") - | Some _,None->source_error "restoreFailed" (Error "The upload checkpoint has no atomic original input proof") - | Some intent,Some _ - when intent.origin = Uri.to_string context.scope.account.managed_sync_origin - && intent.account = context.scope.account.user_id - && intent.graph = context.scope.graph_id - && intent.asset = request.asset - && intent.target = request.target - && intent.title = request.title - && intent.version.file_type = request.file_type - && intent.local_mutation = request.local_mutation - && intent.replace_reference = request.replace_reference - && intent.metadata_mutation = request.metadata_mutation -> Ok intent - | Some _,Some _ -> source_error "identityConflict" (Error "This import identity is already in use") - | None,None -> - let detail = ref None in - let* file, checksum, size = - match - asset_call - ~on_diagnostic:(fun value -> detail := Some value) - t - ~scope:context.scope - (Sync.Stage_asset_file - { operation = request.operation - ; file_type = request.file_type - ; source_file = request.source_file - }) - with - | Ok (Sync.Asset_staged { file; checksum; size }) -> Ok (file, checksum, size) - | Error failure -> - if Sync.asset_failure_kind failure <> Sync.Asset_cancelled then - on_diagnostic (Option.value !detail ~default:(asset_failure_detail ~operation:"assetMetadata" failure)); - Error (asset_failure_message failure) - | Ok _ -> source_error "invalidContent" (Error "Unexpected staging completion") - in - let persistence_attempted = ref false in - let result = - let* version = source_error "invalidRequest" (A.version ~checksum ~file_type:request.file_type) in - let* intent = source_error "invalidRequest" (prepare version size file) in - if (not (current ())) || t.stopped - then Error "The selected graph is no longer active" - else ( - persistence_attempted := true; - let* () = - source_error "persistenceFailed" (Eio.Cancel.protect (fun () -> - with_upload_store t (fun db -> - Logseq_db_storage.Asset_upload_store.Inputs.commit_prepared db ~owner:input_owner ~input:complete_input intent |> Result.map(fun _->())))) - in - Ok intent) - in - (match result with - | Ok _ -> () - | Error _ when not !persistence_attempted -> - Eio.Cancel.protect (fun () -> - ignore (asset_unit - ~on_failure:(fun failure -> if current () then report_asset_failure t context.scope "assetRelease" failure) - t ~scope:context.scope (Sync.Release_staged_file file))) - | Error _ -> ()); - result -;; - -let prepare_import_unfenced t ~(context : Sync.asset_context) request ~current = - with_staging_lock t (fun () -> - let on_diagnostic detail = if current () then - publish_failure (report_current_graph t context.scope) detail in - Result.map - (fun (intent : Logseq_db_types.Asset_upload_intent.t) -> - Hashtbl.replace - t.local_imports - (context.scope, intent.asset, intent.version.checksum) - intent.operation_id; - if intent.phase <> Logseq_db_types.Asset_upload_intent.Cancelled && current () - then - (let detail = ref None in - match asset_call ~on_diagnostic:(fun value -> detail := Some value) - t - ~scope:context.Sync.scope - (Sync.Publish_staged_asset - { asset = intent.asset - ; version = intent.version - ; file = intent.staged_file - }) with - | Ok Sync.Asset_unit | Error Sync.Asset_cancelled -> () - | Error failure -> if current () then report_asset_failure ?detail:!detail t context.scope "assetPreview" failure - | Ok _ -> if current () then report_source_failure t context.scope "assetPreview" "invalidContent" "Unexpected asset completion"); - intent) - (prepare_import_unlocked ~on_diagnostic t ~context request ~current)) -;; - let prepare_import t ~(context:Sync.asset_context) request ~current = - with_writer t context.scope(fun()->prepare_import_unfenced t ~context request ~current) + with_writer t context.scope (fun () -> + if not (current ()) then Error "Import authority retired." + else Result.bind (worker_owner t ~scope:context.scope) (fun (identity, owner) -> + let module Owner = (val Logseq_sync_worker_owner.Core.worker_owner owner : Execute.Core.WORKER_OWNER) in + Owner.with_scope ~identity ~operation:request.Logseq_db_types.Asset_import.operation + (fun owned -> + let current () = not t.stopped && current () in + Result.map (fun intent -> + if current () then + (match Owner.publish_import_preview owned intent with + | Ok () -> () + | Error error -> report_source_failure t context.scope "assetPreview" "ownerFailed" + (Sync_dsl.Core.error_message error)); + intent) + (Owner.prepare_import owned request ~current)) + |> Result.map_error Sync_dsl.Core.error_message)) ;; let retain_imported_file t ~(scope : Sync.graph_scope) ~operation = @@ -3960,12 +3825,33 @@ let retain_imported_file t ~(scope : Sync.graph_scope) ~operation = && intent.account = scope.account.user_id && intent.graph = scope.graph_id && intent.phase <> Logseq_db_types.Asset_upload_intent.Cancelled -> - (let detail = ref None in - match asset_call ~on_diagnostic:(fun value -> detail := Some value) t ~scope (Sync.Retain_staged_file intent.staged_file) with - | Ok (Sync.Asset_retained file) -> file - | Error Sync.Asset_cancelled -> None - | Error failure -> report_asset_failure ?detail:!detail t scope "assetPreview" failure; None - | Ok _ -> report_source_failure t scope "assetPreview" "invalidContent" "Unexpected asset completion"; None) + (let result = Result.bind (worker_owner t ~scope) (fun (identity, owner) -> + let module Owner = (val Logseq_sync_worker_owner.Core.worker_owner owner : Execute.Core.WORKER_OWNER) in + let rec acquire () = + if t.stopped || Core.Writer_fence.fenced t.writer_fence scope + || not (Core.Orphan_cleanup.acquisition_open t.orphan_cleanup) + then Error "Import acquisition is retired." + else if Hashtbl.length t.imported_resources >= 2048 || t.next_imported_resource = Int64.max_int + then Error "Imported resource capacity exceeded." + else + Owner.with_scope ~identity ~operation (fun owned -> + Result.map (Option.map (fun file -> + t.next_imported_resource <- Int64.succ t.next_imported_resource; + let handle = "worker-import:" ^ Int64.to_string t.next_imported_resource in + Hashtbl.add t.imported_resources handle + { imported_scope = scope + ; imported_retain = acquire + ; imported_release = (fun () -> Owner.release_imported_file file + |> Result.map_error Sync_dsl.Core.error_message) + }; + handle, Owner.imported_file_path file)) + (Owner.retain_imported_file owned intent)) + |> Result.map_error Sync_dsl.Core.error_message + in + acquire ()) in + match result with + | Ok file -> file + | Error message -> report_source_failure t scope "assetPreview" "ownerFailed" message; None) | Ok _ -> None | Error source -> report_source_failure t scope "assetMetadata" "persistenceFailed" source; None) ;; diff --git a/logseq_db_worker/lib/logseq_db_worker.ml b/logseq_db_worker/lib/logseq_db_worker.ml index 4514d06c..763f0859 100644 --- a/logseq_db_worker/lib/logseq_db_worker.ml +++ b/logseq_db_worker/lib/logseq_db_worker.ml @@ -3,7 +3,7 @@ module Error = Logseq_db_worker_contract.Error module Pure_reducer = Logseq_db_worker_pure_reducer.Core module Effect_runner = Logseq_db_worker_effect_runner.Effect_runner -module Sync = Logseq_sync_pure_reducer.Core +module Sync = Logseq_sync_domain.Core let unlock_log_count = Atomic.make 0 let unlock_attempt_count = Atomic.make 0 @@ -35,8 +35,6 @@ let observe_unlock before (transition : Pure_reducer.transition) event = | Pure_reducer.Sync_event (Sync.E2ee_password_submitted _) -> let attempt = Atomic.fetch_and_add unlock_attempt_count 1 in let operation = List.find_map (function - | Pure_reducer.Run_sync (Sync.Request (ticket, Sync.Unlock_private_key _)) -> - Some (Sync.effect_ticket_id ticket |> Sync.effect_id_to_string) | _ -> None) transition.effects in unlock_log "attempt=%d op=%s phase=password decision=%s challenge_advertised=%b challenge_ready=%b failure=%s account_generation=%d graph_generation=%d" attempt (Option.value operation ~default:"none") @@ -102,6 +100,9 @@ type coordination_snapshot = type t = { state : Pure_reducer.state ref ; runner : Effect_runner.t + ; mutable lifecycle : Pure_reducer.Lifecycle.t option + ; lifecycle_wake : Eio.Condition.t + ; lifecycle_retired : (Sync.graph_scope, unit) Hashtbl.t ; events : queued_event Eio.Stream.t ; followups : queued_event Queue.t ; mutable next_request_id : int64 @@ -141,10 +142,50 @@ let host_error=function |Not_settled->"The physical host change has not settled." ;; +let lifecycle_step t event = + let module L=Pure_reducer.Lifecycle in + match t.lifecycle with + | None -> [] + | Some state -> + let transition=L.step state event in + t.lifecycle<-Some transition.next; + List.filter_map(function + | L.Worker work -> Some(Pure_reducer.Run_worker work) + | L.Publish output -> Some(Pure_reducer.Publish output) + | L.Attached _ -> None + | L.Retired scope -> Hashtbl.replace t.lifecycle_retired scope ();Eio.Condition.broadcast t.lifecycle_wake;None) transition.effects +;; let dispatch t event = let refused_deletion=ref None in let before = !(t.state) in - let transition = Pure_reducer.step before event in + let transition = match t.lifecycle,event with + | Some _,Pure_reducer.Lifecycle_snapshot(sync,coordination,attachments) -> + let transition=Pure_reducer.step before (Pure_reducer.Lifecycle_snapshot(sync,coordination,attachments)) in + let admitted=Pure_reducer.graph_scopes transition.next in + let attachments=List.filter(fun(scope,_)->List.mem scope admitted)attachments in + let revision=Int64.succ (Pure_reducer.Lifecycle.snapshot(Option.get t.lifecycle)).revision in + let selected=Option.bind (Pure_reducer.coordination_state transition.next).sync.selected(fun selected->List.find_opt(fun(scope:Sync.graph_scope)->scope.graph_id=selected.graph_id)(List.map fst attachments)) in + let effects=lifecycle_step t (Projection{revision;scopes=List.map fst attachments;selected}) in + let effects=effects @ List.concat_map(fun(scope,opened)->lifecycle_step t (Attach(scope,opened)))attachments in + {transition with effects=effects @ transition.effects} + | Some _,Pure_reducer.Sync_event(Sync.Host_fence_started _) -> + let transition=Pure_reducer.step before event in + let current=Pure_reducer.Lifecycle.snapshot(Option.get t.lifecycle) in + let scopes=List.filter(fun scope->List.mem scope (Pure_reducer.graph_scopes transition.next))current.scopes in + let selected=Option.bind current.selected(fun scope->if List.mem scope scopes then Some scope else None) in + let effects=lifecycle_step t (Projection{revision=Int64.succ current.revision;scopes;selected}) in + {transition with effects=effects @ transition.effects} + | Some _,Pure_reducer.Graph_request_for_scope{scope;id;read} -> + {Pure_reducer.next=before;effects=lifecycle_step t (Read{scope;id;read})} + | Some _,Pure_reducer.Graph_request{id;read} -> + (match (Pure_reducer.Lifecycle.snapshot(Option.get t.lifecycle)).selected with + | Some scope -> {Pure_reducer.next=before;effects=lifecycle_step t (Read{scope;id;read})} + | None -> Pure_reducer.step before event) + | Some _,Pure_reducer.Projection_push_for_scope(scope,push) -> + {Pure_reducer.next=before;effects=lifecycle_step t (Push(scope,push))} + | Some _,Pure_reducer.Runner_completed completion -> + {Pure_reducer.next=before;effects=lifecycle_step t (Completed completion)} + | _ -> Pure_reducer.step before event in observe_unlock before transition event; t.state := transition.next; let state=Pure_reducer.coordination_state transition.next in @@ -212,9 +253,10 @@ let dispatch t event = ;; let accepts_after_shutdown = function + | Pure_reducer.Runner_completed (Execute_request_completed _) -> true | Pure_reducer.Asset_cleanup_barrier _->true | Pure_reducer.Sync_event - ( Logseq_sync_pure_reducer.Core.Asset_requested _ | Protected_requested _ + ( Logseq_sync_domain.Core.Asset_requested _ | Protected_requested _ | Runner_completed (Asset_completion _ | Protected_completion _) | IO_fence_completed _ | Shutdown ) -> true @@ -308,10 +350,12 @@ let fresh_host_lifetime () = |> Result.get_ok ;; -let create ~sw ~config ~runner_dependencies = +let create_internal ~lifecycle ~sw ~config ~runner_dependencies = match Pure_reducer.initial config with | Error (Pure_reducer.Invalid_create message) -> Error (Invalid_create message) | Ok state -> + let state=if lifecycle then let sync=(Pure_reducer.view state).sync in + Pure_reducer.install_lifecycle state ~sync ~coordination:(Pure_reducer.coordination_state state).sync ~attachments:[] else state in let events = Eio.Stream.create 256 in let state_ref = ref state in let event_sink = ref (fun event -> Eio.Stream.add events (Event event)) in @@ -330,7 +374,8 @@ let create ~sw ~config ~runner_dependencies = | Error (Effect_runner.Invalid_create message) -> Error (Invalid_create message) | Ok runner -> let t = - { state = state_ref; runner; events;followups=Queue.create(); next_request_id = 0L; stopped = false + { state = state_ref; runner; lifecycle=(if lifecycle then Some Pure_reducer.Lifecycle.empty else None); + lifecycle_wake=Eio.Condition.create();lifecycle_retired=Hashtbl.create 8; events;followups=Queue.create(); next_request_id = 0L; stopped = false ; local_deletion_waiter = None ; coordination={version=0L;presentation=0L;state=Pure_reducer.coordination_state state} ; coordination_registration=0L;coordination_notify=None;coordination_retirement=Pure_reducer.Registration_retirement.empty @@ -347,12 +392,35 @@ let create ~sw ~config ~runner_dependencies = Ok t) ;; +let create ~sw ~config ~runner_dependencies = create_internal ~lifecycle:false ~sw ~config ~runner_dependencies +let create_lifecycle ~sw ~config ~runner_dependencies = create_internal ~lifecycle:true ~sw ~config ~runner_dependencies +let attach_database_owner t ~scope owner = Effect_runner.attach_database_owner t.runner ~scope owner +let detach_database_owner t ~scope = + List.iter (Effect_runner.submit t.runner) (lifecycle_step t (Retire scope)); + Eio.Condition.loop_no_mutex t.lifecycle_wake (fun () -> + if Hashtbl.mem t.lifecycle_retired scope then Some () else None); + Effect_runner.detach_database_owner t.runner ~scope let post t event = route_event t event let report_asset_diagnostic t ~scope ~operation ~request_operation diagnostic = if not t.stopped then Effect_runner.report_asset_diagnostic t.runner ~scope ~operation ~request_operation diagnostic ;; +let database_owner t ~scope = + if t.stopped then Error "Worker stopped" + else Effect_runner.database_owner t.runner ~scope +;; + +let sync_upload_store t = + if t.stopped then Error "Worker stopped" + else Effect_runner.sync_upload_store t.runner +;; + +let worker_owner t ~scope = + if t.stopped then Error "Worker stopped" + else Effect_runner.worker_owner t.runner ~scope +;; + let view t = Pure_reducer.view !(t.state) let graph_state t = (view t).graph @@ -400,7 +468,7 @@ let replace_coordinated_asset_demand t ~registration ~scope ~consumer ~priority | R.Start-> Eio.Cancel.protect(fun()-> run_asset_command_barrier t ~owner ~data:true - (Pure_reducer.Asset_requested_for_scope(scope,Logseq_sync_pure_reducer.Asset_transfer.Replace{consumer;priority;assets})) + (Pure_reducer.Asset_requested_for_scope(scope,Logseq_sync_asset_transfer.Core.Replace{consumer;priority;assets})) |> Result.map(fun _->())) ;; @@ -427,7 +495,7 @@ let release_coordinated_asset_demand t ~registration ~scope ~consumer = | R.Release-> let actual=R.pending t.demand_registry ~registration |> Rrbvec.find_opt(fun owner->owner.R.consumer=consumer) |> Option.get in Result.bind(run_asset_command_barrier t ~owner:actual ~data:false - (Pure_reducer.Asset_requested_for_scope(actual.scope,Logseq_sync_pure_reducer.Asset_transfer.Release consumer))) + (Pure_reducer.Asset_requested_for_scope(actual.scope,Logseq_sync_asset_transfer.Core.Release consumer))) (fun snapshot->Result.map(fun()->t.demand_registry<-R.settled t.demand_registry actual) (Effect_runner.await_asset_cleanup t.runner snapshot)) ;; @@ -493,6 +561,10 @@ let delete_local_copy_internal ?prepared_handoff t ~account_generation ~graph_id || List.exists (fun (scope:Sync.graph_scope)->scope.graph_id=graph_id) (Pure_reducer.graph_scopes !(t.state)) || not (List.exists (fun (graph : Sync.graph) -> graph.graph_id = graph_id) snapshot.catalog) then Error "This local copy cannot be deleted now." + else if Option.is_some t.lifecycle || Option.is_some prepared_handoff then + (match t.coordination.state.sync.account with + | None -> Error "The graph account is no longer current." + | Some account -> Effect_runner.delete_local_copy t.runner ?prepared_handoff ~account ~graph_id ()) else ( let outcome, resolve = Eio.Promise.create () in let waiter = { graph_id; account_generation; prepared_handoff;request = None; resolve } in @@ -570,7 +642,7 @@ let unsettled_orphan_retains t=Effect_runner.unsettled_orphan_retains t.runner type import_receipt = { operation : Logseq_db_types.Graph_types.Uuid.t ; graph_generation : int - ; scope : Logseq_sync_pure_reducer.Core.graph_scope + ; scope : Logseq_sync_domain.Core.graph_scope ; target : Logseq_db_types.Graph_types.Uuid.t ; asset : Logseq_db_types.Asset_descriptor.t ; file_type : string diff --git a/logseq_db_worker/lib/logseq_db_worker.mli b/logseq_db_worker/lib/logseq_db_worker.mli index 1ce9b37b..15f0a093 100644 --- a/logseq_db_worker/lib/logseq_db_worker.mli +++ b/logseq_db_worker/lib/logseq_db_worker.mli @@ -26,13 +26,30 @@ val create -> runner_dependencies:Effect_runner.dependencies -> (t, create_error) result +val create_lifecycle + : sw:Eio.Switch.t -> config:Pure_reducer.config + -> runner_dependencies:Effect_runner.dependencies -> (t, create_error) result +val attach_database_owner : t -> scope:Logseq_sync_domain.Core.graph_scope + -> Execute.Core.database_owner -> (Pure_reducer.database_opened, string) result +val detach_database_owner : t -> scope:Logseq_sync_domain.Core.graph_scope -> (unit, string) result val post : t -> Pure_reducer.event -> unit +val database_owner + : t -> scope:Logseq_sync_domain.Core.graph_scope + -> (Execute.Core.database_owner, string) result + +(** Borrowed until Worker shutdown; checked Sync retirement precedes graph close. *) +val sync_upload_store : t -> (Logseq_sync_worker_owner.Core.store, string) result + +val worker_owner + : t -> scope:Logseq_sync_domain.Core.graph_scope + -> (Sync_dsl.Core.identity * Logseq_sync_worker_owner.Core.t, string) result + (** Diagnostic-only receipt for an exact active asset operation. Retired or unknown operations are ignored; the normal completion owns publication. *) val report_asset_diagnostic : t - -> scope:Logseq_sync_pure_reducer.Core.graph_scope + -> scope:Logseq_sync_domain.Core.graph_scope -> operation:string -> request_operation:string -> Logseq_db_types.Failure_diagnostic.t @@ -54,40 +71,40 @@ val shutdown : t -> unit val retain_asset_file : t - -> scope:Logseq_sync_pure_reducer.Core.graph_scope + -> scope:Logseq_sync_domain.Core.graph_scope -> handle:string -> (string * string) option val release_asset_file : t - -> scope:Logseq_sync_pure_reducer.Core.graph_scope + -> scope:Logseq_sync_domain.Core.graph_scope -> handle:string -> unit val release_asset_file_checked - : t -> scope:Logseq_sync_pure_reducer.Core.graph_scope -> handle:string + : t -> scope:Logseq_sync_domain.Core.graph_scope -> handle:string -> (unit, string) result (** Backend ownership of an unpublished retained resource, including retries. *) -val abandon_asset_file : t -> scope:Logseq_sync_pure_reducer.Core.graph_scope -> handle:string -> unit +val abandon_asset_file : t -> scope:Logseq_sync_domain.Core.graph_scope -> handle:string -> unit val unsettled_orphan_retains : t -> int (** Typed demand registration and checked retirement on the existing serial Worker entry. Release joins actual retiring IO/callbacks before success. *) val replace_coordinated_asset_demand : t -> registration:int64 - -> scope:Logseq_sync_pure_reducer.Core.graph_scope -> consumer:string - -> priority:Logseq_sync_pure_reducer.Asset_transfer.priority + -> scope:Logseq_sync_domain.Core.graph_scope -> consumer:string + -> priority:Logseq_sync_asset_transfer.Core.priority -> assets:Logseq_db_types.Asset_descriptor.t list -> (unit,string) result val asset_demand_creator_finished : t -> registration:int64 -> consumer:string -> unit val finalize_coordinated_asset_demand : t -> registration:int64 - -> scope:Logseq_sync_pure_reducer.Core.graph_scope -> consumer:string -> (unit,string) result + -> scope:Logseq_sync_domain.Core.graph_scope -> consumer:string -> (unit,string) result val release_coordinated_asset_demand : t -> registration:int64 - -> scope:Logseq_sync_pure_reducer.Core.graph_scope -> consumer:string -> (unit,string) result + -> scope:Logseq_sync_domain.Core.graph_scope -> consumer:string -> (unit,string) result type import_receipt = { operation : Logseq_db_types.Graph_types.Uuid.t ; graph_generation : int - ; scope : Logseq_sync_pure_reducer.Core.graph_scope + ; scope : Logseq_sync_domain.Core.graph_scope ; target : Logseq_db_types.Graph_types.Uuid.t ; asset : Logseq_db_types.Asset_descriptor.t ; file_type : string @@ -95,7 +112,7 @@ type import_receipt = } val import_asset - : ?scope:Logseq_sync_pure_reducer.Core.graph_scope + : ?scope:Logseq_sync_domain.Core.graph_scope -> ?current:(unit -> bool) -> t -> graph_generation:int @@ -104,17 +121,17 @@ val import_asset val retain_imported_file : t - -> scope:Logseq_sync_pure_reducer.Core.graph_scope + -> scope:Logseq_sync_domain.Core.graph_scope -> operation:Logseq_db_types.Graph_types.Uuid.t -> (string * string) option -val read_for_scope : t -> scope:Logseq_sync_pure_reducer.Core.graph_scope +val read_for_scope : t -> scope:Logseq_sync_domain.Core.graph_scope -> Worker_pure_read.packed_query -> Worker_pure_read.answer -val durable_mutation_for_scope : t -> scope:Logseq_sync_pure_reducer.Core.graph_scope +val durable_mutation_for_scope : t -> scope:Logseq_sync_domain.Core.graph_scope -> intent:Logseq_db_worker_contract.Worker_pure_mutation.durable_intent -> current:(unit -> bool) -> Logseq_db_worker_contract.Worker_pure_mutation.outcome -val graph_scopes : t -> Logseq_sync_pure_reducer.Core.graph_scope list -val graph_state_for_scope : t -> scope:Logseq_sync_pure_reducer.Core.graph_scope -> graph_state option +val graph_scopes : t -> Logseq_sync_domain.Core.graph_scope list +val graph_state_for_scope : t -> scope:Logseq_sync_domain.Core.graph_scope -> graph_state option type coordination_snapshot = { version : int64 @@ -134,16 +151,16 @@ val coordination_current : t -> registration:int64 -> bool val revoke_coordination : t -> unit val coordination_presentation_current : t -> registration:int64 -> presentation:int64 - -> owner:Logseq_sync_pure_reducer.Core.timeline_scope -> bool -val confirm_mutation_for_scope : t -> scope:Logseq_sync_pure_reducer.Core.graph_scope + -> owner:Logseq_sync_domain.Core.timeline_scope -> bool +val confirm_mutation_for_scope : t -> scope:Logseq_sync_domain.Core.graph_scope -> read:Worker_pure_read.packed_query -> intent:Logseq_db_worker_contract.Worker_pure_mutation.durable_intent -> original:(string * string * string) -> current:(unit -> bool) -> Worker_pure_read.answer -val recover_mutations_for_scope : t -> scope:Logseq_sync_pure_reducer.Core.graph_scope +val recover_mutations_for_scope : t -> scope:Logseq_sync_domain.Core.graph_scope -> after:int64 option -> maximum_items:int -> maximum_bytes:int -> current:(unit -> bool) -> (Logseq_db_worker_contract.Worker_pure_mutation.recovery_page,string) result -val acknowledge_mutation_for_scope : t -> scope:Logseq_sync_pure_reducer.Core.graph_scope +val acknowledge_mutation_for_scope : t -> scope:Logseq_sync_domain.Core.graph_scope -> intent:Logseq_db_worker_contract.Worker_pure_mutation.durable_intent -> original:(string * string * string) option -> current:(unit -> bool) -> (unit,string) result @@ -155,7 +172,7 @@ val register_observation : t -> registration:int64 -> id:Logseq_db_types.Graph_types.Uuid.t - -> scope:Logseq_sync_pure_reducer.Core.graph_scope + -> scope:Logseq_sync_domain.Core.graph_scope -> read:Worker_pure_read.packed_query -> current:(unit -> bool) -> notify:(epoch:int64 -> unit) @@ -165,55 +182,55 @@ val refresh_observation : t -> registration:int64 -> id:Logseq_db_types.Graph_types.Uuid.t - -> scope:Logseq_sync_pure_reducer.Core.graph_scope + -> scope:Logseq_sync_domain.Core.graph_scope -> (int64 * Worker_pure_read.answer, string) result val release_observation : t -> registration:int64 -> id:Logseq_db_types.Graph_types.Uuid.t - -> scope:Logseq_sync_pure_reducer.Core.graph_scope + -> scope:Logseq_sync_domain.Core.graph_scope -> (unit, string) result (** Retire only an unknown logical observation ID; no implicit-current graph or invented account lifecycle. A known physical allocation must use its owner. *) val observation_creator_finished : t -> registration:int64 -> id:Logseq_db_types.Graph_types.Uuid.t -> unit val finalize_observation : t -> registration:int64 -> id:Logseq_db_types.Graph_types.Uuid.t - -> owner:Logseq_sync_pure_reducer.Core.timeline_scope -> (unit,string) result + -> owner:Logseq_sync_domain.Core.timeline_scope -> (unit,string) result val release_unallocated_observation : t -> registration:int64 -> id:Logseq_db_types.Graph_types.Uuid.t - -> owner:Logseq_sync_pure_reducer.Core.timeline_scope -> (unit,string) result + -> owner:Logseq_sync_domain.Core.timeline_scope -> (unit,string) result val observation_scope : t -> registration:int64 -> id:Logseq_db_types.Graph_types.Uuid.t - -> Logseq_sync_pure_reducer.Core.graph_scope option + -> Logseq_sync_domain.Core.graph_scope option (** Match the original delivered batch and release the actual physical windows. *) -val acknowledge_change_batch : t -> registration:int64 -> scope:Logseq_sync_pure_reducer.Core.graph_scope +val acknowledge_change_batch : t -> registration:int64 -> scope:Logseq_sync_domain.Core.graph_scope -> batch:Worker_pure_read.change_batch -> current:(unit -> bool) -> (Worker_pure_read.change_ack,Worker_pure_read.error) result (** Writer-only actual I/O/callback barrier. No resource settlement or host permit is implied; the scope remains fenced until its execution incarnation retires. *) -val quiesce_writers : t -> registration:int64 -> scope:Logseq_sync_pure_reducer.Core.graph_scope -> (unit,string) result +val quiesce_writers : t -> registration:int64 -> scope:Logseq_sync_domain.Core.graph_scope -> (unit,string) result (** Host plans pin the authoritative baseline and client-known execution owners. Physical resource scopes are captured by the backend, not by this request. *) type host_plan={version:int64;id:Logseq_db_types.Graph_types.Uuid.t;registration:int64; - owner:Logseq_sync_pure_reducer.Core.account_scope;change:Pure_reducer.Host_handoff.change; - expected:Logseq_sync_pure_reducer.Core.graph_scope Rrbvec.t} + owner:Logseq_sync_domain.Core.account_scope;change:Pure_reducer.Host_handoff.change; + expected:Logseq_sync_domain.Core.graph_scope Rrbvec.t} val prepare_host_change:t -> host_plan -> (Pure_reducer.Host_handoff.receipt,string)result val apply_host_change:t -> Pure_reducer.Host_handoff.receipt -> (unit,string)result val confirm_host_change:t -> Pure_reducer.Host_handoff.receipt -> (bool,string)result -val recover_import_inputs_for_scope:t -> scope:Logseq_sync_pure_reducer.Core.graph_scope +val recover_import_inputs_for_scope:t -> scope:Logseq_sync_domain.Core.graph_scope -> after:int64 option -> maximum_items:int -> maximum_bytes:int -> current:(unit -> bool) -> (Logseq_db_types.Asset_import.Recovery.page,string)result -val confirm_import_input_for_scope:t -> scope:Logseq_sync_pure_reducer.Core.graph_scope +val confirm_import_input_for_scope:t -> scope:Logseq_sync_domain.Core.graph_scope -> input:string -> current:(unit -> bool) -> (Logseq_db_types.Asset_import.Recovery.item option,string)result -val acknowledge_import_input_for_scope:t -> scope:Logseq_sync_pure_reducer.Core.graph_scope +val acknowledge_import_input_for_scope:t -> scope:Logseq_sync_domain.Core.graph_scope -> input:string -> prepared:string -> current:(unit -> bool) -> (unit,string)result diff --git a/logseq_db_worker/lib/pure_reducer/asset_upload.ml b/logseq_db_worker/lib/pure_reducer/asset_upload.ml index 50cd99bd..851df650 100644 --- a/logseq_db_worker/lib/pure_reducer/asset_upload.ml +++ b/logseq_db_worker/lib/pure_reducer/asset_upload.ml @@ -1,7 +1,7 @@ (** One durable explicit import, owned by the worker and independent of download demand. *) module Intent = Logseq_db_types.Asset_upload_intent -module Scope = Logseq_sync_pure_reducer.Core +module Scope = Logseq_sync_domain.Core type failure = | Network diff --git a/logseq_db_worker/lib/pure_reducer/core.ml b/logseq_db_worker/lib/pure_reducer/core.ml index a0a65ad9..be54caf8 100644 --- a/logseq_db_worker/lib/pure_reducer/core.ml +++ b/logseq_db_worker/lib/pure_reducer/core.ml @@ -48,7 +48,7 @@ type lifecycle_result = | Lifecycle_failed of int * Logseq_db_worker_contract.Error.t type sync_worker_result = - { event : Logseq_sync_pure_reducer.Core.event option + { event : Logseq_sync_domain.Core.event option ; lifecycle : lifecycle_result } @@ -60,7 +60,7 @@ type _ runner_request = -> Worker_pure_read.answer runner_request | Close_database : database_handle -> unit runner_request | Handle_sync_worker_effect : - Logseq_sync_pure_reducer.Core.worker_effect + Logseq_sync_domain.Core.worker_effect -> sync_worker_result runner_request type runner_effect = Request : ticket * 'a runner_request -> runner_effect @@ -77,7 +77,7 @@ type asset_notice = { consumer : string ; asset : Logseq_db_types.Graph_types.Uuid.t ; descriptor : Logseq_db_types.Asset_descriptor.t - ; availability : Logseq_sync_pure_reducer.Asset_transfer.availability + ; availability : Logseq_sync_asset_transfer.Core.availability } | Asset_demand_accepted of string | Asset_backpressure of string @@ -90,44 +90,43 @@ type asset_notice = ; status : Asset_upload.status } +type lifecycle_intent = Sync_intent of Logseq_sync_domain.Core.event | Foreground_intent of bool | Stop_intent + type output = - | Asset_notice of Logseq_sync_pure_reducer.Core.graph_scope * asset_notice + | Lifecycle_intent of lifecycle_intent + | Asset_notice of Logseq_sync_domain.Core.graph_scope * asset_notice | Reply of request_id * Worker_pure_read.answer | Graph_push of Worker_pure_data.changefeed_signal - | Sync_output of Logseq_sync_pure_reducer.Core.output + | Sync_output of Logseq_sync_domain.Core.output | Graph_state_changed of graph_state - | Scoped_graph_state_changed of Logseq_sync_pure_reducer.Core.graph_scope * graph_state - | Scoped_graph_push of Logseq_sync_pure_reducer.Core.graph_scope * Worker_pure_data.changefeed_signal + | Scoped_graph_state_changed of Logseq_sync_domain.Core.graph_scope * graph_state + | Scoped_graph_push of Logseq_sync_domain.Core.graph_scope * Worker_pure_data.changefeed_signal | Diagnostic of string type upload_recovery_ticket = - { scope : Logseq_sync_pure_reducer.Core.graph_scope + { scope : Logseq_sync_domain.Core.graph_scope ; after : Logseq_db_types.Graph_types.Uuid.t option ; limit : int ; serial : int } type instruction = - | Sync_io_fence_reached of int64 * Logseq_sync_pure_reducer.Core.effect_scope * (unit,string) result | Asset_cleanup_barrier_reached of int64 - | Deliver_protected_result of Logseq_sync_pure_reducer.Core.protected_output - | Deliver_asset_result of Logseq_sync_pure_reducer.Core.asset_output + | Deliver_asset_result of Logseq_sync_domain.Core.asset_output | Read_uploads of upload_recovery_ticket - | Run_upload of Logseq_sync_pure_reducer.Core.asset_context * Asset_upload.instruction + | Run_upload of Logseq_sync_domain.Core.asset_context * Asset_upload.instruction | Run_asset of - Logseq_sync_pure_reducer.Core.asset_context - * Logseq_sync_pure_reducer.Asset_transfer.instruction - | Close_asset_scope of Logseq_sync_pure_reducer.Core.graph_scope + Logseq_sync_domain.Core.asset_context + * Logseq_sync_asset_transfer.Core.instruction + | Close_asset_scope of Logseq_sync_domain.Core.graph_scope | Run_worker of runner_effect - | Run_sync of Logseq_sync_pure_reducer.Core.runner_effect | Publish of output -module Sync = Logseq_sync_pure_reducer.Core +module Sync = Logseq_sync_domain.Core let instruction_diagnostic = function - | Sync_io_fence_reached _->"sync-io-fence" + | Publish (Lifecycle_intent _) -> "forward-lifecycle" | Asset_cleanup_barrier_reached _ -> "asset-cleanup-barrier" - | Deliver_protected_result _ -> "deliver-protected-result" | Deliver_asset_result _ -> "deliver-asset-result" | Read_uploads _ -> "read-uploads" | Run_upload _ -> "run-upload" @@ -138,7 +137,6 @@ let instruction_diagnostic = function | Run_worker (Request (_, Close_database _)) -> "run-worker:Close_database" | Run_worker (Request (_, Handle_sync_worker_effect _)) -> "run-worker:Handle_sync_worker_effect" - | Run_sync sync_effect -> "run-sync:" ^ Sync.runner_effect_diagnostic sync_effect | Publish (Reply _) -> "publish:reply" | Publish (Graph_push _ | Scoped_graph_push _) -> "publish:graph-push" | Publish (Sync_output _) -> "publish:sync-output" @@ -148,25 +146,19 @@ let instruction_diagnostic = function let equal_instruction left right = match left, right with - | Sync_io_fence_reached(li,ls,lr),Sync_io_fence_reached(ri,rs,rr)->li=ri && ls=rs && lr=rr | Asset_cleanup_barrier_reached l,Asset_cleanup_barrier_reached r->l=r - | Deliver_protected_result l, Deliver_protected_result r -> l = r | Deliver_asset_result l, Deliver_asset_result r -> l = r | Read_uploads l, Read_uploads r -> l = r | Run_upload (lc, li), Run_upload (rc, ri) -> lc = rc && li = ri | Run_asset (lc, li), Run_asset (rc, ri) -> lc = rc && li = ri | Close_asset_scope left, Close_asset_scope right -> left = right - | Run_sync left, Run_sync right -> Sync.equal_runner_effect left right | Run_worker left, Run_worker right -> left = right | Publish left, Publish right -> left = right - | ( ( Sync_io_fence_reached _ - | Asset_cleanup_barrier_reached _ + | ( ( Asset_cleanup_barrier_reached _ | Run_worker _ - | Run_sync _ | Run_asset _ | Run_upload _ | Read_uploads _ - | Deliver_protected_result _ | Deliver_asset_result _ | Close_asset_scope _ | Publish _ ) @@ -191,29 +183,27 @@ type view = type pending = | Pending_execute of request_id * Read.packed_query * ticket - | Pending_sync_worker of ticket * Sync.worker_effect type graph_session = { owner : Sync.graph_scope ; saved_graph : graph_state - ; saved_assets : (Sync.asset_context * Logseq_sync_pure_reducer.Asset_transfer.t) option + ; saved_assets : (Sync.asset_context * Logseq_sync_asset_transfer.Core.t) option ; saved_recovery : (Sync.graph_scope * Logseq_db_types.Graph_types.Uuid.t option * bool) option ; saved_recovery_pending : upload_recovery_ticket option ; saved_recovery_serial : int ; saved_uploads : (Sync.asset_context * Logseq_db_types.Graph_types.Uuid.t * Asset_upload.t) list ; saved_database : database_handle option - ; saved_deferred_detach : Sync.graph_scope option ; saved_pending : pending list } type state = - { account_reset : Sync.account_scope option - ; resident_mode : bool + { resident_mode : bool ; active_scope : Sync.graph_scope option ; graph_sessions : graph_session list ; graph : graph_state - ; sync_core : Sync.t - ; assets : (Sync.asset_context * Logseq_sync_pure_reducer.Asset_transfer.t) option + ; projection : Sync.state * Sync.coordination_state + ; external_fence : Sync.host_fence option + ; assets : (Sync.asset_context * Logseq_sync_asset_transfer.Core.t) option ; recovery : (Sync.graph_scope * Logseq_db_types.Graph_types.Uuid.t option * bool) option ; recovery_pending : upload_recovery_ticket option @@ -222,50 +212,42 @@ type state = (Sync.asset_context * Logseq_db_types.Graph_types.Uuid.t * Asset_upload.t) list ; foreground : bool ; database : database_handle option - ; deferred_detach : Sync.graph_scope option ; pending : pending list ; next_effect_id : int64 - ; lifecycle_generation : int64 ; shutdown : bool } type create_error = Invalid_create of string -let initial config = - Sync.initial config - |> Result.map_error (fun (Sync.Invalid_create message) -> Invalid_create message) - |> Result.map (fun sync_core -> - { account_reset = None - ; resident_mode = false - ; active_scope = None - ; graph_sessions = [] - ; graph = { generation = 0; graph_id = None; phase = Graph_closed; error = None } - ; sync_core - ; assets = None - ; recovery = None - ; recovery_pending = None - ; recovery_serial = 0 - ; uploads = [] - ; foreground = true - ; database = None - ; deferred_detach = None - ; pending = [] - ; next_effect_id = 0L - ; lifecycle_generation = 0L - ; shutdown = false - }) +let initial _config = + let startup : Sync.startup_facts = + { authenticated=false; local_access=Sync.Local_pending; network_phase=Sync.Local_only; + catalog_loading=false; awaiting_selection=false; restoring_local=false; bootstrapping=false; + awaiting_e2ee_password=false; failure=None; account_generation=0; graph_generation=0; presentation_generation=0 } in + let sync : Sync.state = + { snapshot={local_deletion=None;sync_phase=Sync.Offline;catalog=[];selected_graph=None;resident_graphs=[]; + applied_server_t=None;timeline_presentation_pending=false;unlock=Sync.Unlock_inactive;startup; + last_error=None;last_error_attention=None};diagnostics={groups=[]} } in + let coordination : Sync.coordination_state = + {account=None;desired_residents=[];residents=[];selected=None;deletion=None} in + Ok {resident_mode=true;active_scope=None;graph_sessions=[]; + graph={generation=0;graph_id=None;phase=Graph_closed;error=None};projection=(sync,coordination); + external_fence=None;assets=None;recovery=None;recovery_pending=None;recovery_serial=0;uploads=[]; + foreground=true;database=None;pending=[];next_effect_id=0L;shutdown=false} ;; +let sync_state state = fst state.projection +let sync_coordination state = snd state.projection +let scope_for_graph state ~graph_id = List.find_map (fun ((scope:Sync.graph_scope), _) -> + if scope.graph_id=graph_id then Some scope else None) (sync_coordination state).residents +let admitted_scope state = Option.bind (sync_coordination state).selected + (fun selected -> scope_for_graph state ~graph_id:selected.graph_id) + let view state = { graph = state.graph - ; sync = Sync.state state.sync_core + ; sync = sync_state state ; pending_requests = - List.fold_left - (fun count -> function - | Pending_execute _ -> count + 1 - | Pending_sync_worker _ -> count) - 0 - state.pending + List.length state.pending ; pending_effects = List.length state.pending ; shutdown = state.shutdown } @@ -274,6 +256,7 @@ let view state = let equal_view left right = left = right type event = + | Lifecycle_snapshot of Logseq_sync_domain.Core.state * Logseq_sync_domain.Core.coordination_state * (Logseq_sync_domain.Core.graph_scope * database_opened) list | Asset_cleanup_barrier of int64 | Upload_requested of { graph_generation : int @@ -285,19 +268,19 @@ type event = | Upload_completed of Asset_upload.ticket * Asset_upload.completion | Asset_requested of { graph_generation : int - ; event : Logseq_sync_pure_reducer.Asset_transfer.event + ; event : Logseq_sync_asset_transfer.Core.event } | Asset_completed of - Logseq_sync_pure_reducer.Core.graph_scope - * Logseq_sync_pure_reducer.Asset_transfer.event + Logseq_sync_domain.Core.graph_scope + * Logseq_sync_asset_transfer.Core.event | Graph_request_for_scope of - { scope : Logseq_sync_pure_reducer.Core.graph_scope + { scope : Logseq_sync_domain.Core.graph_scope ; id : request_id ; read : Worker_pure_read.packed_query } - | Projection_push_for_scope of Logseq_sync_pure_reducer.Core.graph_scope * Worker_pure_data.changefeed_signal - | Asset_requested_for_scope of Logseq_sync_pure_reducer.Core.graph_scope * Logseq_sync_pure_reducer.Asset_transfer.event - | Upload_requested_for_scope of Logseq_sync_pure_reducer.Core.graph_scope * Logseq_db_types.Graph_types.Uuid.t * Asset_upload.event + | Projection_push_for_scope of Logseq_sync_domain.Core.graph_scope * Worker_pure_data.changefeed_signal + | Asset_requested_for_scope of Logseq_sync_domain.Core.graph_scope * Logseq_sync_asset_transfer.Core.event + | Upload_requested_for_scope of Logseq_sync_domain.Core.graph_scope * Logseq_db_types.Graph_types.Uuid.t * Asset_upload.event | Start | Graph_request of { id : request_id @@ -322,7 +305,7 @@ let fresh_ticket state = ;; let pending_id = function - | Pending_execute (_, _, ticket) | Pending_sync_worker (ticket, _) -> ticket.id + | Pending_execute (_, _, ticket) -> ticket.id ;; let remove_pending id values = @@ -340,9 +323,6 @@ let unavailable_response read message = failed_read read (Read.Service_failed me ;; -let graph_pending = function - | Pending_sync_worker (_,(Sync.Reset_managed_account _ | Sync.Delete_mirror _)) -> false - | _ -> true let freeze_graph owner state = {owner ; saved_graph = state.graph @@ -352,13 +332,12 @@ let freeze_graph owner state = ; saved_recovery_serial = state.recovery_serial ; saved_uploads = state.uploads ; saved_database = state.database - ; saved_deferred_detach = state.deferred_detach - ; saved_pending = List.filter graph_pending state.pending + ; saved_pending = state.pending } let remember_graph state = let owner=match state.active_scope,state.graph.graph_id with | Some scope,_ -> Some scope - | None,Some id -> (match Sync.scope_for_graph state.sync_core ~graph_id:id with + | None,Some id -> (match scope_for_graph state ~graph_id:id with | Some scope when scope.graph_generation=state.graph.generation -> Some scope | _ -> None) | None,None -> None in match owner with @@ -376,100 +355,19 @@ let load_graph state (owner:Sync.graph_scope) = ; recovery_serial = saved.saved_recovery_serial ; uploads = saved.saved_uploads ; database = saved.saved_database - ; deferred_detach = saved.saved_deferred_detach - ; pending = List.filter (fun p->not(graph_pending p)) state.pending @ saved.saved_pending + ; pending = saved.saved_pending } | None -> {state with active_scope=Some owner; graph={generation=owner.graph_generation;graph_id=Some owner.graph_id;phase=Graph_opening;error=None}; assets=None;recovery=None;recovery_pending=None;recovery_serial=0;uploads=[]; - database=None;deferred_detach=None;pending=[]} + database=None;pending=[]} let restore_visible_graph state = - match Sync.admitted_graph_scope state.sync_core with + match admitted_scope state with | Some scope -> load_graph state scope | None -> state -let graph_publication state graph = - match state.active_scope with - | Some scope when state.resident_mode -> - Publish (Scoped_graph_state_changed (scope,graph)) - | _ -> Publish (Graph_state_changed graph) -let worker_effect_scope = function - | Sync.Inspect_mirror {scope;_} | Attach_graph {scope;_} | Detach_graph scope - | Inspect_sync scope | Apply_outbox_transition {scope;_} -> Some scope - | Activate_snapshot activation -> Some (Sync.snapshot_activation_scope activation) - | Apply_authoritative_batch {scope;_} -> Some scope.graph - | Reset_managed_account _ | Delete_mirror _ -> None -;; - -let translate_sync transition state = - let resident_mode=state.resident_mode || (Sync.state transition.Sync.next).snapshot.resident_graphs<>[] in - let state = { (remember_graph state) with sync_core = transition.Sync.next;resident_mode } in - let rec loop state reversed = function - | [] -> { next = remember_graph state; effects = List.rev reversed } - | Sync.Run sync_effect :: rest -> loop state (Run_sync sync_effect :: reversed) rest - | Sync.Publish (Sync.Protected_finished output) :: rest -> - loop state (Deliver_protected_result output :: reversed) rest - | Sync.Publish (Sync.Asset_finished output) :: rest -> - loop state (Deliver_asset_result output :: reversed) rest - | Sync.Publish output :: rest -> - loop state (Publish (Sync_output output) :: reversed) rest - | Sync.Delegate (Sync.Detach_graph scope) :: rest -> - let state = if not state.resident_mode then state else load_graph state scope in - let graph = { state.graph with phase = Graph_closing } in - loop - { state with graph; deferred_detach = Some scope } - (graph_publication state graph :: reversed) - rest - | Sync.Delegate (Sync.Reset_managed_account account) :: rest -> - let state=remember_graph state in - let sessions=List.map (fun session -> - if session.saved_database=None then session else - {session with saved_graph={session.saved_graph with phase=Graph_closing}; - saved_deferred_detach=Some session.owner}) state.graph_sessions in - let state={state with account_reset=Some account;graph_sessions=sessions} in - let state=match state.active_scope with - | None->state | Some scope -> - (match List.find_opt (fun s->s.owner=scope) sessions with - | None->state | Some session->{state with graph=session.saved_graph;deferred_detach=session.saved_deferred_detach}) in - loop state reversed rest - | Sync.Delegate worker_effect :: rest -> - let state = if not state.resident_mode then state - else Option.fold ~none:state ~some:(load_graph state) (worker_effect_scope worker_effect) in - let ticket, state = fresh_ticket state in - let state = - { state with - pending = Pending_sync_worker (ticket, worker_effect) :: state.pending - } - in - loop - state - (Run_worker (Request (ticket, Handle_sync_worker_effect worker_effect)) - :: reversed) - rest - in - loop state [] transition.effects -;; - -let apply_lifecycle state = function - | Lifecycle_unchanged -> state, [] - | Lifecycle_opened (opened, generation) -> - let graph = - { generation; graph_id = Some opened.graph_id; phase = Graph_open; error = None } - in - ( { state with graph; database = Some opened.database } - , [ graph_publication state graph ] ) - | Lifecycle_closed generation -> - let graph = { generation; graph_id = None; phase = Graph_closed; error = None } in - { state with graph; database = None }, [ graph_publication state graph ] - | Lifecycle_failed (generation, error) -> - let graph = - { state.graph with generation; phase = Graph_failed; error = Some error } - in - { state with graph; database = None }, [ graph_publication state graph ] -;; - let execute_completion state ticket result = match List.find_map (function - | Pending_execute(id,read,candidate) when candidate.id=ticket.id -> Some(id,read) + | Pending_execute(id,read,candidate) when candidate=ticket -> Some(id,read) | _ -> None) state.pending with | None -> no_effects state | Some(id,read) -> @@ -477,230 +375,39 @@ let execute_completion state ticket result = {next={state with pending=remove_pending ticket.id state.pending};effects=[Publish(Reply(id,response))]} ;; -let failure_diagnostic error = - let module D = Logseq_db_types.Failure_diagnostic in - List.find_map - (fun (detail : Logseq_db_worker_contract.Error.detail) -> - match detail.name, detail.value with - | name, Detail_string source when name = D.detail_name -> D.decode source - | _ -> None) - (Logseq_db_worker_contract.Error.details error) -;; - -let failure_signal fallback error = - let module D = Logseq_db_types.Failure_diagnostic in - Option.fold - ~none:fallback - ~some:(fun value -> D.signal_prefix ^ D.encode value) - (failure_diagnostic error) -;; - -let report_sync_failure state transition signal = - (* Sync owns scope acceptance. An ignored completion must not be re-labelled - as an error belonging to whichever graph is now on screen. *) - if Sync.state transition.next.sync_core = Sync.state state.sync_core - then transition - else { transition with effects = Publish (Diagnostic signal) :: transition.effects } -;; - -let step state event = - if state.shutdown - then ( - match event with - | Sync_event - (( Sync.Asset_requested _ | Sync.Protected_requested _ | Sync.Shutdown - | Sync.Runner_completed (Sync.Asset_completion _ | Sync.Protected_completion _) - ) as event) -> translate_sync (Sync.step state.sync_core event) state - | _ -> no_effects state) - else ( - match event with - | Asset_cleanup_barrier _ - | Uploads_loaded _ - | Upload_requested _ - | Upload_completed _ - | Asset_requested _ - | Asset_completed _ -> no_effects state - | Graph_request_for_scope _ | Projection_push_for_scope _ - | Asset_requested_for_scope _ | Upload_requested_for_scope _ -> no_effects state - | Start -> no_effects state - | Projection_push push -> - if - state.graph.phase = Graph_open - && Sync.admitted_graph_scope state.sync_core <> None - then { next = state; effects = [ Publish (Graph_push push) ] } - else no_effects state - | Graph_request { id; read } -> - let Read.Read_query(budget,query)=read in - let owner=Read.For_worker.read_budget_owner budget in - (match Read.validate_query owner budget query with - | Error error -> {next=state;effects=[Publish(Reply(id,failed_read read error))]} - | Ok () -> match state.graph.phase,state.database with - | Graph_open,Some database -> - let ticket,state=fresh_ticket state in - {next={state with pending=Pending_execute(id,read,ticket)::state.pending}; - effects=[Run_worker(Request(ticket,Execute_request{database;read}))]} - | _ -> {next=state;effects=[Publish(Reply(id,unavailable_response read "The graph is not open."))]}) - | Sync_event event -> translate_sync (Sync.step state.sync_core event) state - | Set_foreground foreground -> - let lifecycle_generation = Int64.succ state.lifecycle_generation in - let state = { state with lifecycle_generation; foreground } in - translate_sync - (Sync.step - state.sync_core - (Sync.Foreground_changed { foreground; lifecycle_generation })) - state - | Runner_completed (Execute_request_completed (ticket, result)) -> - if - ticket.generation <> state.graph.generation - || not (has_pending ticket.id state.pending) - then no_effects state - else execute_completion state ticket result - | Runner_completed (Close_database_completed _) -> no_effects state - | Runner_completed (Sync_worker_effect_completed (ticket, result)) -> - if - not (has_pending ticket.id state.pending) - || (ticket.generation <> state.graph.generation && not (List.exists (function - | Pending_sync_worker (candidate,(Sync.Reset_managed_account _ | Sync.Delete_mirror _))->candidate.id=ticket.id - | _->false) state.pending)) - then no_effects state - else ( - let worker_effect = - List.find_map - (function - | Pending_sync_worker (candidate, worker_effect) - when Int64.equal candidate.id ticket.id -> Some worker_effect - | Pending_sync_worker _ | Pending_execute _ -> None) - state.pending - in - let state = { state with pending = remove_pending ticket.id state.pending } in - match result with - | Error error -> - let message = Logseq_db_worker_contract.Error.message error in - (match worker_effect with - | Some (Sync.Inspect_mirror request) -> - let changed=translate_sync - (Sync.step state.sync_core (Sync.Graph_attachment_failed - {scope=Sync.effect_scope_of_graph request.scope;message})) state in - report_sync_failure state changed (failure_signal "worker.inspectMirror" error) - | Some (Sync.Attach_graph request) -> - let changed=translate_sync - (Sync.step state.sync_core (Sync.Graph_attachment_failed - {scope=Sync.effect_scope_of_graph request.scope;message})) state in - report_sync_failure state changed (failure_signal "worker.attachGraph" error) - | Some (Sync.Inspect_sync scope) -> - let changed=translate_sync - (Sync.step state.sync_core (Sync.Graph_attachment_failed - {scope=Sync.effect_scope_of_graph scope;message})) state in - report_sync_failure state changed (failure_signal "worker.inspectSync" error) - | Some (Sync.Activate_snapshot request) -> - let changed=translate_sync - (Sync.step state.sync_core - (Sync.Snapshot_activation_failed (Sync.snapshot_activation_failed request ~message))) state in - report_sync_failure state changed (failure_signal "worker.activateSnapshot" error) - | Some (Sync.Apply_authoritative_batch request) -> - let transitioned = - translate_sync - (Sync.step - state.sync_core - (Sync.Authoritative_batch_failed - { scope = Sync.effect_scope_of_graph request.scope.graph; message })) - state - in - let origin = (Logseq_db_worker_contract.Error.trace error).origin in - let signal = - match origin.component, origin.operation, origin.code with - | Managed_sync, "applyAuthoritative", Some ("authoritativeDecodeFailed" | "authoritativeIntegrityFailure" as category) -> - let module Diagnostic = Logseq_db_types.Authoritative_diagnostic in - let detail = List.find_map (fun (detail:Logseq_db_worker_contract.Error.detail) -> - match detail.name,detail.value with - | name,Detail_string source when name=Diagnostic.detail_name -> Diagnostic.decode source - | _ -> None) (Logseq_db_worker_contract.Error.details error) in - Option.fold ~none:("worker.applyAuthoritative." ^ category) - ~some:(fun value -> (if category="authoritativeIntegrityFailure" then Diagnostic.integrity_signal_prefix else Diagnostic.signal_prefix) ^ Diagnostic.encode value) detail - | Managed_sync, "applyAuthoritative", Some category -> - "worker.applyAuthoritative." ^ category - | _ -> "worker.applyAuthoritative" - in - report_sync_failure state transitioned (failure_signal signal error) - | Some (Sync.Apply_outbox_transition request) -> - let transitioned = - translate_sync - (Sync.step - state.sync_core - (Sync.Outbox_transition_failed { request; message })) - state - in - report_sync_failure state transitioned (failure_signal "worker.applyOutbox" error) - | Some (Sync.Detach_graph scope) -> - let changed=translate_sync - (Sync.step state.sync_core (Sync.Graph_detached (scope, Error message))) state in - report_sync_failure state changed (failure_signal "worker.closeDatabase" error) - | Some (Sync.Delete_mirror request) -> - let changed=translate_sync - (Sync.step state.sync_core (Sync.Mirror_deleted (request, Error message))) state in - report_sync_failure state changed (failure_signal "worker.deleteMirror" error) - | Some _ | None -> - let code = match worker_effect with - | Some (Sync.Inspect_mirror _) -> "worker.inspectMirror" - | Some (Sync.Attach_graph _) -> "worker.attachGraph" - | Some (Sync.Inspect_sync _) -> "worker.inspectSync" - | Some (Sync.Reset_managed_account _) -> "worker.resetAccount" - | Some _ | None -> "worker.operation" - in - { next = state; effects = [ Publish (Diagnostic (failure_signal code error)) ] }) - | Ok result -> - let state, lifecycle_effects = - match state.graph.phase, worker_effect with - | Graph_closing, Some (Sync.Detach_graph _) -> - apply_lifecycle state result.lifecycle - | Graph_closing, _ -> state, [] - | _ -> apply_lifecycle state result.lifecycle - in - let transitioned = - match result.event with - | None -> no_effects state - | Some event -> translate_sync (Sync.step state.sync_core event) state - in - { next = transitioned.next; effects = lifecycle_effects @ transitioned.effects }) - | Shutdown -> - let stopped_sync = translate_sync (Sync.step state.sync_core Sync.Shutdown) state in - let state = stopped_sync.next in - let close_effects, state = - match state.database with - | None -> [], state - | Some database -> - let ticket, state = fresh_ticket state in - [ Run_worker (Request (ticket, Close_database database)) ], state - in - { next = - { state with - shutdown = true - ; database = None - ; pending = [] - ; graph = { state.graph with phase = Graph_closed; graph_id = None } - } - ; effects = stopped_sync.effects @ close_effects - }) -;; - -(* Drain all already-issued database operations before executing the existing close. - Their replies still finish; Sync rejects their superseded graph events. *) let step state event = - let transition = step state event in - match transition.next.deferred_detach, List.filter graph_pending transition.next.pending with - | Some scope, [] when not transition.next.shutdown -> - let ticket, next = fresh_ticket transition.next in - let worker_effect = Sync.Detach_graph scope in - { next = - { next with - deferred_detach = None - ; pending = Pending_sync_worker (ticket, worker_effect) :: List.filter (fun p->not(graph_pending p)) next.pending - } - ; effects = - transition.effects - @ [ Run_worker (Request (ticket, Handle_sync_worker_effect worker_effect)) ] - } - | _ -> transition + match event with + | Sync_event event -> {next=state;effects=[Publish (Lifecycle_intent (Sync_intent event))]} + | Set_foreground foreground -> + {next={state with foreground};effects=[Publish (Lifecycle_intent (Foreground_intent foreground))]} + | Shutdown -> {next={state with shutdown=true};effects=[Publish (Lifecycle_intent Stop_intent)]} + | Runner_completed (Execute_request_completed (ticket,result)) -> + if ticket.generation=state.graph.generation then execute_completion state ticket result else no_effects state + | Runner_completed (Sync_worker_effect_completed _) -> no_effects state + | Runner_completed (Close_database_completed _) -> no_effects state + | Graph_request {id;read} when not state.shutdown -> + let Read.Read_query(budget,query)=read in + let owner=Read.For_worker.read_budget_owner budget in + let validation = match state.active_scope with + | Some scope when Read.matches_asset_scope owner scope -> Read.validate_query owner budget query + | Some _ -> Error (Read.Service_failed "The read belongs to another graph scope.") + | None -> Error (Read.Service_failed "The graph is not open.") in + (match validation with + | Error error -> {next=state;effects=[Publish(Reply(id,failed_read read error))]} + | Ok () -> match state.graph.phase,state.database with + | Graph_open,Some database -> + let ticket,state=fresh_ticket state in + {next={state with pending=Pending_execute(id,read,ticket)::state.pending}; + effects=[Run_worker(Request(ticket,Execute_request{database;read}))]} + | _ -> {next=state;effects=[Publish(Reply(id,unavailable_response read "The graph is not open."))]}) + | Projection_push push when not state.shutdown && state.graph.phase=Graph_open && admitted_scope state<>None -> + {next=state;effects=[Publish(Graph_push push)]} + | Graph_request {id;read} -> + {next=state;effects=[Publish(Reply(id,unavailable_response read "The worker is stopped."))]} + | Lifecycle_snapshot _ | Asset_cleanup_barrier _ | Uploads_loaded _ | Upload_requested _ + | Upload_completed _ | Asset_requested _ | Asset_completed _ | Graph_request_for_scope _ + | Projection_push_for_scope _ | Asset_requested_for_scope _ | Upload_requested_for_scope _ + | Start | Projection_push _ -> no_effects state ;; let complete_execute runner_effect result = @@ -710,26 +417,24 @@ let complete_execute runner_effect result = | Request (_, Close_database _) | Request (_, Handle_sync_worker_effect _) -> None ;; -module Transfer = Logseq_sync_pure_reducer.Asset_transfer +module Transfer = Logseq_sync_asset_transfer.Core let asset_scope state = - if state.shutdown || state.graph.phase <> Graph_open - then None - else match state.active_scope with - | Some scope when state.resident_mode -> Sync.asset_context_for_scope state.sync_core scope - | _ -> Sync.asset_context state.sync_core + if state.shutdown || state.graph.phase <> Graph_open then None else + Option.map (fun (scope:Sync.graph_scope) -> + let encrypted=List.exists (fun (graph:Sync.graph)->graph.graph_id=scope.Sync.graph_id && graph.encrypted) + (sync_state state).snapshot.catalog in + {Sync.scope;encrypted;key=(if encrypted then Some(Sync.graph_key_handle ~id:"direct-owner" ~scope) else None)}) state.active_scope ;; let asset_online state = - let sync=Option.value (Option.bind state.active_scope (Sync.state_for_scope state.sync_core)) - ~default:(Sync.state state.sync_core) in - state.foreground - && (let f=sync.snapshot.startup in - f.authenticated && f.network_phase<>Sync.Local_only) - && - match sync.snapshot.sync_phase with - | Offline | Paused | Failed -> false - | Connecting | Pulling | Submitting | Current -> true + let sync=Option.value (Option.bind state.active_scope + (fun scope->List.assoc_opt scope (sync_coordination state).residents)) ~default:(sync_state state) in + state.foreground && sync.snapshot.startup.authenticated + && sync.snapshot.startup.network_phase<>Sync.Local_only + && match sync.snapshot.sync_phase with + | Offline | Paused | Failed -> false + | Connecting | Pulling | Submitting | Current -> true ;; let asset_unlocked (context : Sync.asset_context) = @@ -1118,19 +823,46 @@ let import_context state ~graph_generation = let step_session = step -let graph_scopes state = - List.filter_map (fun id -> Sync.scope_for_graph state.sync_core ~graph_id:id) - (Sync.resident_graphs state.sync_core) +let graph_scopes state = List.map fst (sync_coordination state).residents let graph_state_for_scope state (scope:Sync.graph_scope) = - if Sync.scope_for_graph state.sync_core ~graph_id:scope.graph_id <> Some scope then None + if scope_for_graph state ~graph_id:scope.graph_id <> Some scope then None else Some (load_graph state scope).graph type coordination_state = { sync : Sync.coordination_state ; residents : (Sync.graph_scope * graph_state) list } let coordination_state state = - let sync=Sync.coordination_state state.sync_core in + let sync=sync_coordination state in {sync;residents=List.map (fun (scope,_) -> scope,(load_graph state scope).graph) sync.residents} +let external_fence_blocks state id = match state.external_fence with + | None -> false + | Some fence -> (match fence.target with Sync.All_graphs -> true | One_graph target -> target=id) +let lifecycle_projection state ~sync:(sync:Sync.state) ~coordination:(coordination:Sync.coordination_state) ~attachments = + let admitted id = not (external_fence_blocks state id) in + let residents=List.filter(fun((scope:Sync.graph_scope),_)->admitted scope.graph_id)coordination.residents in + let selected=Option.bind coordination.selected(fun(selected:Sync.timeline_scope)->if admitted selected.graph_id then Some selected else None) in + let coordination={coordination with residents;selected;desired_residents=List.filter admitted coordination.desired_residents} in + let sync={sync with snapshot={sync.snapshot with resident_graphs=List.map(fun((scope:Sync.graph_scope),_)->scope.graph_id)residents; + selected_graph=Option.map(fun(selected:Sync.timeline_scope)->selected.graph_id)selected}} in + sync,coordination,List.filter(fun((scope:Sync.graph_scope),_)->admitted scope.graph_id)attachments +let install_lifecycle state ~sync ~coordination ~attachments = + let state = remember_graph state in + let state = {state with projection=(sync,coordination);resident_mode=true;active_scope=None; + graph_sessions=List.filter_map (fun session -> + if List.mem_assoc session.owner attachments then Some session + else if session.saved_pending=[] then None + else Some {session with saved_database=None; + saved_graph={session.saved_graph with phase=Graph_closed}; + saved_assets=None;saved_uploads=[];saved_recovery=None;saved_recovery_pending=None}) state.graph_sessions} in + let state = List.fold_left (fun state (scope,(opened:database_opened)) -> + let state = load_graph state scope in + remember_graph {state with database=Some opened.database; + graph={generation=scope.graph_generation;graph_id=Some scope.graph_id;phase=Graph_open;error=None}}) state attachments in + match admitted_scope state with + | Some scope -> load_graph state scope + | None -> {state with active_scope=None;database=None;assets=None;uploads=[];pending=[]; + recovery=None;recovery_pending=None; + graph={generation=sync.snapshot.startup.graph_generation;graph_id=None;phase=Graph_closed;error=None}} let session_for_ticket state ticket = List.find_opt (fun s -> has_pending ticket.id s.saved_pending) (remember_graph state).graph_sessions let scope_of_event state = function @@ -1139,7 +871,7 @@ let scope_of_event state = function | Asset_completed (scope,_) -> Some scope | Upload_completed (ticket,_) -> Some ticket.Asset_upload.scope | Uploads_loaded (ticket,_) -> Some ticket.scope - | Sync_event event -> Sync.event_graph_scope state.sync_core event + | Sync_event _ -> None | Runner_completed (Execute_request_completed (ticket,_) | Close_database_completed (ticket,_) | Sync_worker_effect_completed (ticket,_)) -> Option.map (fun s->s.owner) (session_for_ticket state ticket) @@ -1147,7 +879,7 @@ let scope_of_event state = function let step state event = let state=match event with Sync_event (Sync.Resident_graphs_changed _) -> {state with resident_mode=true} | _->state in let known=state.resident_mode in - let admitted (scope:Sync.graph_scope)=Sync.scope_for_graph state.sync_core ~graph_id:scope.Sync.graph_id=Some scope in + let admitted (scope:Sync.graph_scope)=scope_for_graph state ~graph_id:scope.Sync.graph_id=Some scope in let owner=match event with | Graph_request_for_scope {scope;_} | Projection_push_for_scope (scope,_) | Asset_requested_for_scope (scope,_) | Upload_requested_for_scope (scope,_,_) @@ -1158,12 +890,12 @@ let step state event = let state=match owner with Some scope when known->load_graph state scope | _->state in let event=match event with | Graph_request_for_scope {scope;id;read} -> - if Sync.scope_for_graph state.sync_core ~graph_id:scope.graph_id = Some scope + if scope_for_graph state ~graph_id:scope.graph_id = Some scope then Graph_request {id;read} else Graph_request_for_scope {scope;id;read} | Asset_requested_for_scope (scope,event) when admitted scope -> Asset_requested {graph_generation=scope.graph_generation;event} | Upload_requested_for_scope (scope,operation,event) when admitted scope -> Upload_requested {graph_generation=scope.graph_generation;operation;event} - | Projection_push_for_scope (scope,push) when Sync.scope_for_graph state.sync_core ~graph_id:scope.graph_id=Some scope -> Projection_push push + | Projection_push_for_scope (scope,push) when scope_for_graph state ~graph_id:scope.graph_id=Some scope -> Projection_push push | event -> event in let transition=match event with | Graph_request_for_scope {id;read;_} -> @@ -1177,39 +909,78 @@ let step state event = let loaded=load_graph result.next session.owner in let loaded=if event=Shutdown then {loaded with shutdown=false} else loaded in let part=step_session loaded (if event=Shutdown then Shutdown else Start) in - {next=remember_graph part.next;effects=result.effects @ part.effects}) transition sessions + let effects=List.filter (function Publish(Lifecycle_intent _) -> false | _ -> true) part.effects in + {next=remember_graph part.next;effects=result.effects @ effects}) transition sessions | _->transition in let next=remember_graph transition.next in - let next,drains=List.fold_left (fun (next,effects) session -> - if session.saved_deferred_detach<>None && session.saved_pending=[] then - let drained=step_session (load_graph next session.owner) Start in - remember_graph drained.next,effects @ drained.effects - else next,effects) (next,[]) next.graph_sessions in - let next,account_effects=match next.account_reset with - | Some account when List.for_all (fun session->session.saved_database=None - && session.saved_deferred_detach=None && session.saved_pending=[]) next.graph_sessions -> - let ticket,next=fresh_ticket next in - let request=Sync.Reset_managed_account account in - {next with account_reset=None;pending=Pending_sync_worker(ticket,request)::next.pending}, - [Run_worker(Request(ticket,Handle_sync_worker_effect request))] - | _->next,[] in let effects=List.map (function | Publish (Graph_push push) -> (match owner with Some scope->Publish (Scoped_graph_push (scope,push)) | None->Publish (Graph_push push)) - | instruction -> instruction) (transition.effects @ drains @ account_effects) in + | instruction -> instruction) transition.effects in let next={next with graph_sessions=List.filter (fun session -> session.saved_database<>None || session.saved_graph.phase<>Graph_closed - || session.saved_pending<>[] || session.saved_deferred_detach<>None) next.graph_sessions} in - let next=if next.graph.phase=Graph_closed && next.database=None && List.filter graph_pending next.pending=[] + || session.saved_pending<>[]) next.graph_sessions} in + let next=if next.graph.phase=Graph_closed && next.database=None && next.pending=[] then {next with active_scope=None} else next in - {next=(if known || (Sync.state next.sync_core).snapshot.resident_graphs<>[] then restore_visible_graph next else next);effects} + {next=(if known || (sync_state next).snapshot.resident_graphs<>[] then restore_visible_graph next else next);effects} let step_without_asset_barrier=step let step state event=match event with - | Sync_event(Sync.IO_fence_completed(id,scope,result))->{next=state;effects=[Sync_io_fence_reached(id,scope,result)]} - | Sync_event(Sync.IO_fence_requested _ as event)->translate_sync(Sync.step state.sync_core event)state | Asset_cleanup_barrier token->{next=state;effects=[Asset_cleanup_barrier_reached token]} | _->step_without_asset_barrier state event +let step_with_sessions = step +let publish_lifecycle state sync coordination attachments = + let sync,coordination,attachments=lifecycle_projection state ~sync ~coordination ~attachments in + let cleanup = + (remember_graph state).graph_sessions + |> List.filter (fun session -> not (List.mem_assoc session.owner attachments)) + |> List.concat_map (fun session -> + let assets = match session.saved_assets with + | None -> [] + | Some (context, transfer) -> + let _, instructions = Transfer.step transfer Shutdown in + asset_instructions context instructions in + let uploads = List.concat_map (fun (context, _, upload) -> + let _, instructions = Asset_upload.step upload Shutdown in + List.map (fun instruction -> Run_upload (context, instruction)) instructions) + session.saved_uploads in + let closes = match session.saved_assets, session.saved_uploads with + | None, [] -> [] + | _ when external_fence_blocks state session.owner.graph_id -> [] + | _ -> [Close_asset_scope session.owner] in + assets @ uploads @ closes) in + let next=install_lifecycle state ~sync ~coordination ~attachments in + let next, recovery = List.fold_left (fun (state,effects) ((scope:Sync.graph_scope),_) -> + if external_fence_blocks state scope.Sync.graph_id then state,effects else + let resumed=step_session (load_graph state scope) Start in + remember_graph resumed.next,effects @ resumed.effects) (next,[]) attachments in + let next=restore_visible_graph next in + {next;effects=cleanup @ recovery @ (Publish(Sync_output(Sync.Resident_scopes_changed(List.map fst coordination.residents))) :: + Publish(Sync_output(Sync.State_changed sync)) :: + List.concat_map(fun(scope,state)-> + [Publish(Sync_output(Sync.Session_state_changed(scope,state))); + Publish(Scoped_graph_state_changed(scope,(load_graph next scope).graph))])coordination.residents)} +let step state event = match event,Some state.projection with + | Lifecycle_snapshot(sync,coordination,attachments),_ -> publish_lifecycle state sync coordination attachments + | Sync_event(Sync.Host_fence_started fence),Some(sync,coordination) -> + let owner_matches=Option.exists(fun(owner:Sync.account_scope)-> + Uri.equal owner.managed_sync_origin fence.owner.managed_sync_origin + && owner.user_id=fence.owner.user_id && owner.account_generation=fence.owner.account_generation + && owner.lifecycle_generation=fence.owner.lifecycle_generation)coordination.account in + let replacement=match state.external_fence,fence.previous with + | None,None -> true + | Some old,Some previous -> old.id=previous && old.target=fence.target + | Some old,None -> old.id=fence.id && old.target=fence.target + | _ -> false in + if not owner_matches || not replacement || Rrbvec.length fence.captured_scopes>8192 then no_effects state + else + let attachments=List.filter_map(fun session->Option.map(fun database->session.owner, + {database;graph_id=session.owner.graph_id})session.saved_database)(remember_graph state).graph_sessions in + publish_lifecycle {state with external_fence=Some fence} sync coordination attachments + | Sync_event(Sync.Host_fence_released id),Some _ -> + (match state.external_fence with Some fence when fence.id=id->no_effects {state with external_fence=None}|_->no_effects state) + | _,_ -> step_with_sessions state event + let asset_ticket_current_single = asset_ticket_current let asset_ticket_current state ticket = let scope=match ticket with Transfer.Lookup t->t.scope | Download t->t.scope in @@ -1226,13 +997,13 @@ let import_context state ~graph_generation = | Some scope -> import_context_single (load_graph state scope) ~graph_generation | None -> import_context_single state ~graph_generation let import_context_for_scope state (scope:Sync.graph_scope) = - if Sync.scope_for_graph state.sync_core ~graph_id:scope.graph_id<>Some scope then None + if scope_for_graph state ~graph_id:scope.graph_id<>Some scope then None else import_context_single (load_graph state scope) ~graph_generation:scope.graph_generation ;; module Observation_registry = struct module G = Logseq_db_types.Graph_types - module S = Logseq_sync_pure_reducer.Core + module S = Logseq_sync_domain.Core module Entries = Map.Make (struct type t = int64 * G.Uuid.t @@ -1557,7 +1328,7 @@ end module Orphan_cleanup = struct - type resource = { scope : Logseq_sync_pure_reducer.Core.graph_scope; handle : string } + type resource = { scope : Logseq_sync_domain.Core.graph_scope; handle : string } module Resources=Map.Make(struct type t=resource let compare=Stdlib.compare end) module Queue=Map.Make(Int64) type work={resource:resource;attempt:int64;failures:int} @@ -1589,16 +1360,16 @@ module Orphan_cleanup = struct end module Demand_registry = struct - type owner={registration:int64;scope:Logseq_sync_pure_reducer.Core.graph_scope;consumer:string} + type owner={registration:int64;scope:Logseq_sync_domain.Core.graph_scope;consumer:string} module Keys=Map.Make(struct type t=int64*string let compare=Stdlib.compare end) - type payload=Logseq_sync_pure_reducer.Asset_transfer.priority * Logseq_db_types.Asset_descriptor.t list + type payload=Logseq_sync_asset_transfer.Core.priority * Logseq_db_types.Asset_descriptor.t list type entry={owner:owner;payload:payload option;retired:bool;settled:bool} type t={entries:entry Keys.t;finished:unit Keys.t;closed:int64} type reservation=Start|Existing|Retired|Mismatch|Full|Invalid type release=Release|Already_settled|Unknown|Mismatch|Full let empty={entries=Keys.empty;finished=Keys.empty;closed=0L} let key owner=owner.registration,owner.consumer - let same_scope (a:Logseq_sync_pure_reducer.Core.graph_scope) (b:Logseq_sync_pure_reducer.Core.graph_scope)= + let same_scope (a:Logseq_sync_domain.Core.graph_scope) (b:Logseq_sync_domain.Core.graph_scope)= a.graph_id=b.graph_id && a.graph_generation=b.graph_generation && a.account.user_id=b.account.user_id && Uri.equal a.account.managed_sync_origin b.account.managed_sync_origin && a.account.account_generation=b.account.account_generation @@ -1658,7 +1429,7 @@ end module Writer_fence = struct module Scope = struct - type t=Logseq_sync_pure_reducer.Core.graph_scope + type t=Logseq_sync_domain.Core.graph_scope let key (scope:t)= (Uri.to_string scope.account.managed_sync_origin,scope.account.user_id, scope.account.account_generation,scope.graph_id,scope.graph_generation) @@ -1742,7 +1513,7 @@ module Registration_retirement = struct end module Host_handoff = struct - module S=Logseq_sync_pure_reducer.Core + module S=Logseq_sync_domain.Core module G=Logseq_db_types.Graph_types module Ids=Set.Make(G.Uuid) module Receipts=Map.Make(G.Uuid) @@ -1874,5 +1645,89 @@ module Host_handoff = struct |_->state,Error(if Ids.mem receipt.request.id state.retired then Retired else Not_settled) end -let host_fence_blocks state id=Logseq_sync_pure_reducer.Core.host_fence_blocks state.sync_core id -let host_fence_active state=Logseq_sync_pure_reducer.Core.host_fence_active state.sync_core +let host_fence_blocks state id = external_fence_blocks state id +let host_fence_active state = state.external_fence + +module Lifecycle = struct + type scope = Sync.graph_scope + type snapshot = { revision : int64; scopes : scope list; selected : scope option } + type attachment = { scope : scope; database : database_handle; retiring : bool } + type read = { scope : scope; id : request_id; query : Read.packed_query; ticket : ticket } + type t = + { snapshot : snapshot + ; attachments : attachment list + ; pending : read list + ; retired : scope list + ; serial : int64 + } + type event = + | Projection of snapshot + | Attach of scope * database_opened + | Retire of scope + | Read of { scope : scope; id : request_id; read : Read.packed_query } + | Push of scope * Worker_pure_data.changefeed_signal + | Completed of runner_completion + type instruction = + | Worker of runner_effect + | Publish of output + | Attached of scope + | Retired of scope + type transition = { next : t; effects : instruction list } + let empty = + { snapshot = { revision = -1L; scopes = []; selected = None } + ; attachments = []; pending = []; retired = []; serial = 0L } + let snapshot state = state.snapshot + let unchanged next = { next; effects = [] } + let admitted state scope = List.mem scope state.snapshot.scopes + let drain state = + let ready, attachments = List.partition (fun (attachment : attachment) -> + attachment.retiring && not (List.exists (fun (read : read) -> read.scope = attachment.scope) state.pending)) state.attachments in + { next = { state with attachments; retired = List.map (fun (attachment : attachment) -> attachment.scope) ready @ state.retired } + ; effects = List.map (fun (attachment : attachment) -> Retired attachment.scope) ready } + let retire state scope = + drain { state with attachments = List.map (fun (attachment : attachment) -> + if attachment.scope = scope then { attachment with retiring = true } else attachment) state.attachments } + let step state = function + | Projection snapshot when snapshot.revision <= state.snapshot.revision -> unchanged state + | Projection snapshot -> + if (match snapshot.selected with Some scope -> not (List.mem scope snapshot.scopes) | None -> false) + then unchanged state + else drain { state with snapshot; attachments = List.map (fun (attachment : attachment) -> + { attachment with retiring = attachment.retiring || not (List.mem attachment.scope snapshot.scopes) }) state.attachments } + | Attach (scope, opened) -> + if not (admitted state scope) || opened.graph_id <> scope.graph_id + || List.mem scope state.retired + || List.exists (fun (attachment : attachment) -> attachment.scope.graph_id = scope.graph_id) state.attachments + then unchanged state + else { next = { state with attachments = { scope; database = opened.database; retiring = false } :: state.attachments } + ; effects = [Attached scope] } + | Retire scope -> retire state scope + | Read {scope; id; read} -> + let Read.Read_query (budget, query) = read in + let owner = Read.For_worker.read_budget_owner budget in + let error = + if Read.matches_asset_scope owner scope then Read.validate_query owner budget query + else Error (Read.Service_failed "The read belongs to another graph scope.") + in + (match error with + | Error error -> { next = state; effects = [Publish (Reply (id, failed_read read error))] } + | Ok () -> + match List.find_opt (fun (attachment : attachment) -> attachment.scope = scope && not attachment.retiring) state.attachments with + | Some attachment when admitted state scope && state.serial < Int64.max_int -> + let ticket = { id = state.serial; generation = scope.graph_generation } in + { next = { state with serial = Int64.succ state.serial; pending = {scope; id; query = read; ticket} :: state.pending } + ; effects = [Worker (Request (ticket, Execute_request {database = attachment.database; read}))] } + | _ -> { next = state; effects = [Publish (Reply (id, unavailable_response read "The graph scope is no longer open."))] }) + | Push (scope, push) -> + if admitted state scope && List.exists (fun (attachment : attachment) -> attachment.scope = scope && not attachment.retiring) state.attachments + then { next = state; effects = [Publish (Scoped_graph_push (scope, push))] } + else unchanged state + | Completed (Execute_request_completed (ticket, result)) -> + (match List.find_opt (fun (read : read) -> read.ticket = ticket) state.pending with + | None -> unchanged state + | Some read -> + let answer = match result with Ok answer -> answer | Error error -> failed_read read.query (Read.Db_error error) in + let drained = drain { state with pending = List.filter (fun (read : read) -> read.ticket <> ticket) state.pending } in + { drained with effects = Publish (Reply (read.id, answer)) :: drained.effects }) + | Completed (Close_database_completed _ | Sync_worker_effect_completed _) -> unchanged state +end diff --git a/logseq_db_worker/lib/pure_reducer/dune b/logseq_db_worker/lib/pure_reducer/dune index 4b123027..88e5c426 100644 --- a/logseq_db_worker/lib/pure_reducer/dune +++ b/logseq_db_worker/lib/pure_reducer/dune @@ -11,7 +11,7 @@ logseq_db_worker.contract logseq_db_types logseq_overlay_db - logseq_sync.pure_reducer + logseq_sync.domain logseq_sync.asset_transfer uri yojson logseq_db_worker.typed_data)) diff --git a/logseq_db_worker/lui/dune b/logseq_db_worker/lui/dune index d7a99acc..f0628db5 100644 --- a/logseq_db_worker/lui/dune +++ b/logseq_db_worker/lui/dune @@ -6,21 +6,34 @@ journal_worker_eio_backend journal_worker_runtime journal_worker + journal_sync_registration logseq_db_worker_lui_service) (libraries logseq_db_worker.identity eio eio.core eio.unix eio_posix + digestif logseq_db_worker logseq_db_worker.typed_read logseq_db_worker.contract logseq_db_worker.pure_reducer logseq_db_types logseq_overlay_db - logseq_sync.effect_runner - logseq_sync.pure_reducer + logseq_journal.sync_lifecycle + journal_sync_lifecycle_impl + sync_dsl + execute + logseq_sync.worker_owner + logseq_sync.assets + logseq_sync.native_crypto + logseq_sync.transport + logseq_sync.native_transport + x509 + ca-certs-nss + logseq_sync.domain logseq_sync.asset_transfer melange-transit-native + mirage-crypto-rng mtime.clock mtime.clock.os rrbvec diff --git a/logseq_db_worker/lui/journal_sync_registration.ml b/logseq_db_worker/lui/journal_sync_registration.ml new file mode 100644 index 00000000..0e2a96b0 --- /dev/null +++ b/logseq_db_worker/lui/journal_sync_registration.ml @@ -0,0 +1,1148 @@ +module Dsl = Sync_dsl.Core +module Lifecycle = Journal_sync_lifecycle.Core +module Account = Journal_sync_lifecycle.Account +module Sync = Logseq_sync_domain.Core +module Worker = Logseq_db_worker +module Runner = Logseq_db_worker_effect_runner.Effect_runner +module Pure = Logseq_db_worker_pure_reducer.Core +module Uuid = Logseq_db_types.Graph_types.Uuid +module Database = Logseq_overlay_db.Database + +let ( let* ) = Result.bind + +let error_message = function + | Dsl.Authentication_failure message -> message + | error -> Dsl.error_message error +;; + +let storage_error message = Dsl.Storage_failure message + +type entry = + { scope : Sync.graph_scope + ; core : Lifecycle.t + ; mutable opened : Pure.database_opened option + ; mutable attachment_error : Dsl.error option + ; mutable status : Lifecycle.Service.status option + ; mutable local_key_attempted : bool + ; mutable retiring : bool + } + +type network_control = + | Sign_in of Sync.startup_scope + | Cancel_startup of Sync.startup_scope + | Refresh_catalog of Sync.startup_scope + | Recover_online of Sync.timeline_scope + +type t = + { services : Runner.services + ; validate_control : network_control -> (unit, Dsl.error) result + ; bind : Worker.t -> unit + ; post : Pure.event -> unit + ; lifecycle : Pure.lifecycle_intent -> unit + ; unlock : Logseq_sync_domain.Core.graph_scope -> string -> (unit, string) result + ; shutdown : unit -> unit + } + +let create ~sw ~clock ~network ~authenticator ~config ~overlay ~token () = + let (Worker.Config.Managed_sync { base_url }) = config.Worker.Config.target in + let root = config.application_support_directory in + let deadlines : Execute.Core.network_deadlines = + { setup_seconds = 30. + ; tls_handshake_seconds = 30. + ; http_response_seconds = 60. + ; download_seconds = 600. + } + in + let* limits = + Dsl.limits + ~maximum_items:256 + ~maximum_bytes:config.response_budget_bytes + ~maximum_pending:64 + ~maximum_resources:128 + ~steps_per_turn:256 + ~maximum_tasks:32 + ~maximum_messages:64 + ~maximum_message_bytes:config.response_budget_bytes + ~maximum_demands:256 + ~maximum_active_assets:8 + ~foreground_reserve:2 + ~foreground_demand_reserve:16 + ~foreground_descriptor_reserve_bytes:(min 16384 config.response_budget_bytes) + ~asset_cache_budget_bytes:536870912L + ~asset_maximum_file_bytes:104857600 + ~asset_pending_budget_bytes:536870912L + ~snapshot_maximum_artifact_bytes:1073741824L + ~snapshot_staging_budget_bytes:2147483648L + ~snapshot_scratch_budget_bytes:2147483648L + in + let* retry = + Dsl.retry_policy + ~initial_seconds:1. + ~maximum_seconds:30. + ~multiplier:2. + ~maximum_attempts:5 + in + let* policy = + Dsl.policy_config ~sync_retry:retry ~asset_retry:retry ~no_progress_seconds:30. + in + let worker = ref None in + let controller = ref None in + let entries = ref [] in + let user = ref None in + let authenticated = ref false in + let account_generation = ref 0 + and presentation = ref 0 in + let gate = ref Dsl.Closed in + let network_epoch = ref 0L in + let network_authorized = ref false in + let timeline_permission = ref true in + let foreground = ref true in + let network_phase = ref Sync.Local_only in + let failure = ref None in + let restore_diagnostic = ref false in + let presented = ref [] in + let resident_mode = ref false in + let baseline = ref None in + let stopped = ref false in + let account_lock = Eio.Mutex.create () in + let with_account action = Eio.Mutex.use_rw ~protect:true account_lock action in + let updates = Eio.Condition.create () in + let dirty = ref false in + let notify () = + dirty := true; + Eio.Condition.broadcast updates + in + let find scope = + List.find_opt (fun entry -> entry.scope = scope && not entry.retiring) !entries + in + let owner ~scope = + match find scope with + | None -> Error "The Sync graph owner is retired." + | Some entry -> Ok (Lifecycle.identity entry.core, Lifecycle.worker_owner entry.core) + in + let install_key entry wrapped = + let* input = Dsl.unlock_input ~wrapped_graph_key:wrapped in + let* ingress = Lifecycle.ingress entry.core in + ingress.submit_unlock ~identity:(Lifecycle.identity entry.core) ~valid_for:30. input + |> Result.map (fun _ -> ()) + in + let restore_key entry = + if + (not entry.local_key_attempted) + && Option.fold + ~none:false + ~some:(fun status -> + status.Lifecycle.Service.observation.state.keys = Dsl.Key_locked) + entry.status + then ( + entry.local_key_attempted <- true; + match + Logseq_sync_native_crypto.Platform_crypto.load_wrapped_graph_key + ~managed_sync_origin:entry.scope.account.managed_sync_origin + ~user_id:entry.scope.account.user_id + ~graph_id:entry.scope.graph_id + with + | Error _ -> () + | Ok wrapped -> ignore (install_key entry wrapped)) + in + let publish () = + match !worker, !baseline, !controller with + | Some worker, Some (initial : Sync.state), Some account -> + let entries = ref (List.filter (fun entry -> not entry.retiring) !entries) in + List.iter restore_key !entries; + List.iter + (fun entry -> + match entry.opened with + | Some _ -> () + | None -> + let owner = Lifecycle.database_owner entry.core in + let attached = + let* database = Execute.Core.database_if_open owner in + match database with + | None -> Ok None + | Some _ when entry.retiring -> Ok None + | Some _ -> + Worker.attach_database_owner worker ~scope:entry.scope owner + |> Result.map Option.some + |> Result.map_error storage_error + in + (match attached with + | Ok (Some _) when entry.retiring || !stopped -> + (match Worker.detach_database_owner worker ~scope:entry.scope with + | Ok () -> () + | Error message -> + failure := Some (storage_error message); + notify ()) + | Ok opened -> + entry.opened <- opened; + entry.attachment_error <- None + | Error error -> entry.attachment_error <- Some error)) + !entries; + entries := List.filter (fun entry -> (not entry.retiring) && not !stopped) !entries; + let selected_id = Account.selected account in + let selected = + List.find_opt (fun entry -> Some entry.scope.graph_id = selected_id) !entries + in + let state_for entry = + let observed = + Option.map (fun status -> status.Lifecycle.Service.observation) entry.status + in + let last_error = + Option.bind observed (fun observed -> + match observed.Execute.Core.status with + | Dsl.Service_blocked error -> Some error + | _ -> observed.state.last_failure) + in + let last_error = + match entry.attachment_error, last_error, !failure with + | Some error, _, _ -> Some error + | None, Some _, _ -> last_error + | None, None, failure -> failure + in + let phase = + match observed with + | None -> Sync.Connecting + | Some observed -> + (match observed.status with + | Dsl.Service_waiting_network -> Sync.Offline + | Service_waiting_unlock -> Paused + | Service_blocked _ -> Failed + | Service_backoff _ -> Connecting + | Service_connecting -> Connecting + | Service_idle -> Paused + | Service_pulling -> Pulling + | Service_submitting -> Submitting + | Service_online -> + if observed.state.pending_submissions > 0 then Submitting else Current) + in + let phase = + if Option.is_some entry.attachment_error || Option.is_some !failure + then Sync.Failed + else phase + in + let startup = + { Sync.authenticated = !authenticated + ; account_generation = entry.scope.account.account_generation + ; graph_generation = entry.scope.graph_generation + ; presentation_generation = !presentation + ; local_access = + (if Option.is_some entry.opened then Sync.Local_available else Local_pending) + ; network_phase = !network_phase + ; catalog_loading = false + ; awaiting_selection = false + ; restoring_local = false + ; bootstrapping = + Option.is_none entry.opened && Option.is_none entry.attachment_error + ; failure = + (if Option.is_some entry.attachment_error + then Some Sync.During_bootstrap + else initial.snapshot.startup.failure) + ; awaiting_e2ee_password = + Option.fold + ~none:false + ~some:(fun observed -> observed.Execute.Core.state.keys = Dsl.Key_locked) + observed + } + in + { Sync.diagnostics = + (match entry.attachment_error with + | None -> initial.diagnostics + | Some error -> + { groups = + initial.diagnostics.groups + @ [ { Sync.title = "Worker errors" + ; entries = [ "attachDatabaseOwner", error_message error ] + } + ] + }) + ; snapshot = + { initial.snapshot with + sync_phase = phase + ; catalog = Account.catalog account + ; timeline_presentation_pending = + Option.is_some entry.opened && not (List.mem entry.scope !presented) + ; selected_graph = Some entry.scope.graph_id + ; resident_graphs = List.map (fun entry -> entry.scope.graph_id) !entries + ; applied_server_t = + Option.bind observed (fun observed -> + observed.Execute.Core.state.checkpoint) + ; startup + ; last_error = Option.map error_message last_error + ; last_error_attention = + Option.map (fun _ -> Sync.Requires_attention) last_error + } + } + in + let residents = List.map (fun entry -> entry.scope, state_for entry) !entries in + let sync = + match selected with + | Some entry -> state_for entry + | None -> + { initial with + snapshot = + { initial.snapshot with + catalog = Account.catalog account + ; selected_graph = None + ; resident_graphs = List.map (fun entry -> entry.scope.graph_id) !entries + ; startup = + { initial.snapshot.startup with + authenticated = !authenticated + ; account_generation = !account_generation + ; network_phase = !network_phase + ; catalog_loading = false + ; restoring_local = false + ; bootstrapping = false + ; presentation_generation = !presentation + ; awaiting_selection = !user <> None + } + ; last_error = Option.map error_message !failure + ; last_error_attention = + Option.map (fun _ -> Sync.Requires_attention) !failure + } + } + in + let sync = + if not !restore_diagnostic + then sync + else + { sync with + diagnostics = + { groups = + [ { Sync.title = "Worker errors" + ; entries = + [ ( "inspectMirror" + , "Local graph restoration failed. Reopen Journal or retry \ + local recovery." ) + ] + } + ] + } + } + in + let account_scope = + Option.map + (fun user_id -> + { Sync.managed_sync_origin = Uri.of_string base_url + ; user_id + ; account_generation = !account_generation + ; presentation_generation = !presentation + ; lifecycle_generation = 0L + }) + !user + in + let coordination : Sync.coordination_state = + { account = account_scope + ; desired_residents = List.map (fun entry -> entry.scope.graph_id) !entries + ; residents + ; selected = + Option.map + (fun entry -> + { Sync.startup = + { account_generation = entry.scope.account.account_generation + ; presentation_generation = !presentation + } + ; graph_id = entry.scope.graph_id + ; graph_generation = entry.scope.graph_generation + }) + selected + ; deletion = None + } + in + Worker.post + worker + (Pure.Lifecycle_snapshot + ( sync + , coordination + , List.filter_map + (fun entry -> Option.map (fun opened -> entry.scope, opened) entry.opened) + !entries )) + | _ -> () + in + let catalog = + Account.native_catalog + ~sw + ~clock + ~network + ~authenticator + ~deadlines + ~token:(fun ~account ~epoch ~refresh -> + token + ~user:account.Account.account + ~account_generation:!account_generation + ~epoch + ~refresh) + in + let* account = + Account.create + ~maximum_services:16 + ~catalog + ~start: + (fun + ~account + ~account_generation:generation + ~graph_generation + ~graph + ~upload_store + ~network:gate + ~on_change -> + let* identity = + Dsl.identity + ~origin:account.Account.origin + ~account:account.account + ~graph:(Uuid.to_string graph.Logseq_db_types.Managed_graph.graph_id) + ~account_generation:generation + ~graph_generation + in + if generation > Int64.of_int max_int || graph_generation > Int64.of_int max_int + then Error (Dsl.Capacity_exceeded "Graph generation") + else ( + let scope : Sync.graph_scope = + { account = + { managed_sync_origin = Uri.of_string account.origin + ; user_id = account.account + ; account_generation = Int64.to_int generation + ; presentation_generation = !presentation + ; lifecycle_generation = 0L + } + ; graph_id = graph.graph_id + ; graph_generation = Int64.to_int graph_generation + } + in + account_generation := Int64.to_int generation; + let graph_root = root in + let pending = ref None + and admitted = ref None in + let* core = + Lifecycle.start_owned_native + ~sw + ~clock + ~network + ~authenticator + ~config:{ identity; limits; network = gate; policy } + ~deadlines + ~liveness: + (Execute.Core.Ping_pong { interval_seconds = 30.; timeout_seconds = 10. }) + ~cleanup_seconds:30. + ~root:graph_root + ~dependencies:overlay + ~graph_name:graph.name + ~upload_store + ~namespace: + (Logseq_db_worker_contract.Worker_pure_mutation.namespace_for_owner + ~origin:scope.account.managed_sync_origin + ~user:scope.account.user_id + ~graph:scope.graph_id) + ~encrypted:graph.encrypted + ~token: + (token + ~user:account.account + ~account_generation:scope.account.account_generation) + ~on_change:(fun status -> + pending := Some status; + Option.iter + (fun entry -> if not entry.retiring then entry.status <- Some status) + !admitted; + on_change status; + notify ()) + ~on_error:(fun error -> + failure := Some error; + notify ()) + in + let entry = + { scope + ; core + ; opened = None + ; attachment_error = None + ; status = !pending + ; local_key_attempted = false + ; retiring = false + } + in + admitted := Some entry; + entries := !entries @ [ entry ]; + notify (); + Ok + (Account.service ~set_network:(Lifecycle.set_network core) ~close:(fun () -> + entry.retiring <- true; + notify (); + let diagnose stage = Result.map_error (fun error -> + Printf.eprintf "journal native-close stage=%s\n%!" stage; + error) in + let* () = Lifecycle.set_network core Dsl.Closed + |> diagnose "network-close" in + let* () = + Logseq_sync_worker_owner.Core.fence + (Lifecycle.worker_owner core) + ~identity:(Lifecycle.identity core) + |> diagnose "worker-fence" + in + let* () = + match !worker, entry.opened with + | Some worker, Some _ -> + Worker.detach_database_owner worker ~scope + |> Result.map_error storage_error + |> diagnose "database-detach" + | _ -> Ok () + in + let result = + try + Lifecycle.close core; + Ok () + with + | Lifecycle.Service.Close_failed { error = Some error; _ } -> Error error + | error -> Error (Dsl.Unexpected_failure (Printexc.to_string error)) + in + let* () = result in + entries := List.filter (fun current -> current != entry) !entries; + notify (); + Ok ())))) + ~delete_local:(fun ~account:_ ~graph -> + let bytes = Mirage_crypto_rng.generate 16 in + let hex = + String.concat + "" + (List.init 16 (fun index -> Printf.sprintf "%02x" (Char.code bytes.[index]))) + in + let retirement = Database.retirement_id_of_string hex in + let* retirement = retirement |> Result.map_error storage_error in + let* prepared, _ = + Database.prepare_mirror_deletion + ~application_support_directory:root + ~graph_id:graph.graph_id + ~retirement + |> Result.map_error (fun _ -> + storage_error "Unable to prepare local mirror deletion") + in + let committed = + Database.commit_mirror_deletion prepared + |> Result.map (fun _ -> ()) + |> Result.map_error (fun _ -> storage_error "Unable to delete local mirror") + in + let released = + Database.cancel_mirror_deletion_checked prepared + |> Result.map_error storage_error + in + let* () = released in + committed) + ~on_change:(fun _ _ -> notify ()) + in + controller := Some account; + let cache_path user_id = + let identity = user_id ^ "\000" ^ base_url in + let name = Digestif.SHA256.digest_string identity |> Digestif.SHA256.to_hex in + Filename.concat + (Filename.concat root "logseq-db-worker/sync-catalogs") + (name ^ ".json") + in + let restore_local user_id = + let path = cache_path user_id in + if not (Sys.file_exists path) + then Ok () + else ( + try + let channel = open_in_bin path in + let source = + Fun.protect + ~finally:(fun () -> close_in_noerr channel) + (fun () -> + let length = in_channel_length channel in + if length > 16777216 then raise Exit; + really_input_string channel length) + in + let* cache = + Sync.decode_catalog_cache source + |> Result.map_error (fun _ -> storage_error "Invalid local graph catalog") + in + if Sync.catalog_cache_user_id cache <> user_id + then Error Dsl.Stale_context + else + let* () = Account.restore_catalog account (Sync.catalog_cache_graphs cache) in + let selected = Sync.catalog_cache_selected_graph cache in + let residents = Sync.catalog_cache_resident_graphs cache in + let* () = + List.fold_left + (fun result id -> + let* () = result in + Account.select account id) + (Ok ()) + residents + in + match selected with + | Some id -> Account.select account id + | None -> Account.deselect account + with + | Sys_error _ | Exit -> + Error (storage_error "Unable to restore local graph catalog")) + in + let persist_local () = + match !user with + | None -> Ok () + | Some user_id -> + let cache = + Sync.catalog_cache + ~user_id + ~graphs:(Account.catalog account) + ~selected_graph:(Account.selected account) + in + let cache = + { cache with + Logseq_db_types.Sync_account.cache_resident_graphs = + Some + (List.filter_map + (fun entry -> if entry.retiring then None else Some entry.scope.graph_id) + !entries) + } + in + let path = cache_path user_id in + let temporary = path ^ ".pending" in + (try + List.iter + (fun directory -> + if not (Sys.file_exists directory) then Unix.mkdir directory 0o700) + [ Filename.concat root "logseq-db-worker"; Filename.dirname path ]; + Fun.protect + ~finally:(fun () -> if Sys.file_exists temporary then Sys.remove temporary) + (fun () -> + let channel = open_out_bin temporary in + Fun.protect + ~finally:(fun () -> close_out_noerr channel) + (fun () -> + output_string channel (Sync.encode_catalog_cache cache); + close_out channel); + Sys.rename temporary path); + Ok () + with + | Sys_error _ | Unix.Unix_error _ -> + Error (storage_error "Unable to save local graph catalog")) + in + let checked action = + with_account (fun () -> + let result = action () in + (match result with + | Error Dsl.Stale_context -> () + | Ok _ -> + failure := None; + notify () + | Error error -> + failure := Some error; + notify ()); + result) + in + let replace ?(restore = false) user_id = + if (not restore) && !user = Some user_id + then Ok () + else + let* () = Account.replace account { origin = base_url; account = user_id; root } in + user := Some user_id; + incr account_generation; + incr presentation; + gate := Dsl.Closed; + network_authorized := false; + timeline_permission := true; + network_phase := Sync.Local_only; + authenticated := false; + restore_diagnostic := false; + let result = restore_local user_id in + (match result with + | Ok () -> () + | Error _ -> restore_diagnostic := true); + result + in + let apply_network () = + if !user = None + then Ok () + else + let* next = + if (not !authenticated) || (not !network_authorized) || not !foreground + then Ok Dsl.Closed + else ( + match !gate with + | Dsl.Open _ -> Ok !gate + | Dsl.Closed -> + if !network_epoch = Int64.max_int + then Error (Dsl.Capacity_exceeded "Network authorization epoch") + else ( + network_epoch := Int64.succ !network_epoch; + Ok (Dsl.Open { authorization_epoch = !network_epoch }))) + in + if next = !gate + then Ok () + else ( + gate := next; + match Account.set_network account next with + | Ok () -> Ok () + | Error error -> + gate := Dsl.Closed; + network_authorized := false; + (match Account.set_network account Dsl.Closed with + | Ok () -> Error error + | Error retirement -> Error retirement)) + in + let current_presentation (scope : Sync.startup_scope) = + if + (not !stopped) + && scope.account_generation = !account_generation + && scope.presentation_generation = !presentation + then Ok () + else Error Dsl.Stale_context + in + let current_startup scope = + let* () = current_presentation scope in + if !user <> None then Ok () else Error Dsl.Stale_context + in + let current_timeline (scope : Sync.timeline_scope) = + let* () = current_startup scope.startup in + if + Account.selected account = Some scope.graph_id + && List.exists + (fun entry -> + (not entry.retiring) + && entry.scope.graph_id = scope.graph_id + && entry.scope.graph_generation = scope.graph_generation + && entry.scope.account.account_generation = scope.startup.account_generation + && Some entry.scope.account.user_id = !user) + !entries + then Ok () + else Error Dsl.Stale_context + in + let refresh () = + let* () = apply_network () in + let* _ = Account.refresh account in + persist_local () + in + let select id = + let* () = Account.select account id in + incr presentation; + persist_local () + in + let event = function + | Pure.Sync_event (Sync.Restore_local_account { user_id }) -> + checked (fun () -> replace ~restore:true user_id) + | Sync_event (Account_authenticated { user_id = None }) -> + checked (fun () -> + authenticated := false; + user := None; + incr account_generation; + incr presentation; + network_authorized := false; + gate := Dsl.Closed; + Account.logout account) + | Sync_event (Account_authenticated { user_id = Some user_id }) -> + checked (fun () -> + let authorized = !network_authorized in + let* () = replace user_id in + authenticated := true; + network_authorized := authorized; + let* () = apply_network () in + let generation = !account_generation + and epoch = !network_epoch in + if !foreground + then + Eio.Fiber.fork ~sw (fun () -> + if + generation = !account_generation + && !user = Some user_id + && epoch = !network_epoch + && !gate <> Dsl.Closed + && not !stopped + then ( + let result = refresh () in + if + generation = !account_generation + && !user = Some user_id + && epoch = !network_epoch + && !gate <> Dsl.Closed + && not !stopped + then ignore (checked (fun () -> result)))); + Ok ()) + | Sync_event Sync.Shutdown -> checked (fun () -> Account.logout account) + | Sync_event (Local_cache_deletion_requested id) -> + checked (fun () -> + let* () = Account.delete_local account id in + persist_local ()) + | Sync_event (Graph_selected id | Visible_graph_changed id) -> + checked (fun () -> select id) + | Sync_event (Resident_graphs_changed ids) -> + resident_mode := true; + checked (fun () -> + List.fold_left + (fun result id -> + let* () = result in + Account.select account id) + (Ok ()) + ids) + | Sync_event Graph_picker_requested -> + incr presentation; + checked (fun () -> + let* () = + match !resident_mode, Account.selected account with + | false, Some id -> Account.release_graph account id + | true, _ | false, None -> Ok () + in + let* () = Account.deselect account in + persist_local ()) + | Sync_event (Sign_in_requested scope) -> + checked (fun () -> + let* () = current_presentation scope in + network_phase := Sync.Online_by_user; + timeline_permission := true; + network_authorized := true; + apply_network ()) + | Sync_event (Catalog_refresh_requested scope) -> + checked (fun () -> + let* () = current_startup scope in + refresh ()) + | Sync_event (Online_recovery_requested scope) -> + checked (fun () -> + let* () = current_timeline scope in + let selected_failure = + Option.bind + (List.find_opt + (fun entry -> + (not entry.retiring) + && entry.scope.graph_id = scope.graph_id + && entry.scope.graph_generation = scope.graph_generation) + !entries) + (fun entry -> + match entry.attachment_error, entry.status with + | Some error, _ -> Some error + | None, Some status -> + let observed = status.Lifecycle.Service.observation in + (match observed.status with + | Dsl.Service_blocked error -> Some error + | _ -> observed.state.last_failure) + | None, None -> None) + in + let* retry = + match selected_failure, !failure with + | _, Some error + when match error with + | Dsl.Network_closed | Network_failure _ -> false + | _ -> true -> Error error + | Some (Dsl.Network_closed | Network_failure _), _ + | None, (None | Some (Dsl.Network_closed | Network_failure _)) -> Ok true + | Some _, _ -> Ok false + | None, Some error -> Error error + in + if not retry + then Ok () + else + let* () = + if !authenticated && !network_authorized && !foreground + then ( + gate := Dsl.Closed; + Account.set_network account Dsl.Closed) + else Ok () + in + refresh ()) + | Sync_event (Online_cancelled scope) -> + checked (fun () -> + let* () = current_presentation scope in + timeline_permission := false; + network_phase := Sync.Local_only; + network_authorized := false; + apply_network ()) + | Pure.Set_foreground value -> + checked (fun () -> + foreground := value; + if !user = None then Ok () else apply_network ()) + | Sync_event (Local_graph_outbox_changed scope) -> + checked (fun () -> + match find scope with + | Some entry -> Lifecycle.projection_changed entry.core + | None -> Error Dsl.Stale_context) + | Sync_event Local_outbox_changed -> + checked (fun () -> + List.fold_left + (fun result entry -> + let* () = result in + Lifecycle.projection_changed entry.core) + (Ok ()) + !entries) + | Sync_event (Timeline_presented scope) -> + checked (fun () -> + let* () = current_timeline scope in + let* () = if !timeline_permission then Ok () else Error Dsl.Stale_context in + match + List.find_opt + (fun entry -> + (not entry.retiring) + && entry.scope.graph_id = scope.graph_id + && entry.scope.graph_generation = scope.graph_generation + && Option.is_some entry.opened) + !entries + with + | Some entry when not (List.mem entry.scope !presented) -> + presented := entry.scope :: !presented; + network_phase := Sync.Online_after_timeline; + network_authorized := true; + apply_network () + | Some _ -> Ok () + | None -> Error Dsl.Stale_context) + | Sync_event (Local_feed_acknowledged | Local_account_unavailable _) -> Ok () + | event -> + Option.iter (fun worker -> Worker.post worker event) !worker; + Ok () + in + let unlock scope password = + checked (fun () -> + match find scope with + | None -> Error Dsl.Stale_context + | Some entry -> + let check_current () = + if + !stopped + || (not + (Option.fold + ~none:false + ~some:(fun current -> current == entry) + (find scope))) + || !user <> Some scope.account.user_id + || not !authenticated + then Error Dsl.Stale_context + else Ok () + in + let admit_network () = + let* () = check_current () in + match !gate with + | Dsl.Closed -> Error Dsl.Network_closed + | Open _ -> Ok () + in + let* () = admit_network () in + let epoch = + match !gate with + | Dsl.Open { authorization_epoch } -> authorization_epoch + | Closed -> 0L + in + let client = + Logseq_sync_transport.Http_client.create + ~backend: + (Logseq_sync_transport.Http_client.native_backend + ~sw + ~clock + ~network + ~authenticator + ~deadlines) + ~token: + (token + ~user:scope.account.user_id + ~account_generation:scope.account.account_generation) + in + let perform make = + Logseq_sync_transport.Http_client.perform + client + ~scope:{ check_current; admit_network } + ~epoch + ~authenticated:true + (fun token -> make (Option.value token ~default:"")) + in + let* keys = + perform (fun token -> + Logseq_sync_native_transport.Http.e2ee_user_keys + ~base_url:scope.account.managed_sync_origin + ~token) + in + let* package = + Logseq_sync_native_crypto.E2ee.user_keys_response keys.body + |> Result.map_error (fun message -> Dsl.Protocol_failure message) + in + let* () = check_current () in + let* () = + Logseq_sync_native_crypto.Platform_crypto.unlock_private_key + ~managed_sync_origin:scope.account.managed_sync_origin + ~user_id:scope.account.user_id + ~password + ~private_key_package:package + |> Result.map_error (fun message -> Dsl.Crypto_failure message) + in + let* graph = + perform (fun token -> + Logseq_sync_native_transport.Http.e2ee_graph_key + ~base_url:scope.account.managed_sync_origin + ~graph_id:scope.graph_id + ~token) + in + let* wrapped = + Logseq_sync_native_crypto.E2ee.graph_key_response graph.body + |> Result.map_error (fun message -> Dsl.Protocol_failure message) + in + let* () = check_current () in + let* () = + Logseq_sync_native_crypto.Platform_crypto.verify_and_save_wrapped_graph_key + ~managed_sync_origin:scope.account.managed_sync_origin + ~user_id:scope.account.user_id + ~graph_id:scope.graph_id + ~encrypted_graph_key:wrapped + |> Result.map_error (fun message -> Dsl.Crypto_failure message) + in + install_key entry wrapped) + |> Result.map_error error_message + in + let await_io ~(scope : Sync.effect_scope) = + List.fold_left + (fun result entry -> + let* () = result in + if + Option.fold + ~none:true + ~some:(( = ) entry.scope.account.account_generation) + scope.Sync.account_generation + && Option.fold + ~none:true + ~some:(( = ) entry.scope.graph_generation) + scope.graph_generation + then + let* () = + Lifecycle.set_network entry.core Dsl.Closed |> Result.map_error error_message + in + let* () = + Logseq_sync_worker_owner.Core.fence + (Lifecycle.worker_owner entry.core) + ~identity:(Lifecycle.identity entry.core) + |> Result.map_error error_message + in + Logseq_sync_worker_owner.Core.await_io + (Lifecycle.worker_owner entry.core) + ~identity:(Lifecycle.identity entry.core) + |> Result.map_error error_message + else Ok ()) + (Ok ()) + !entries + in + let shutdown () = + with_account (fun () -> + if not !stopped + then ( + stopped := true; + Eio.Condition.broadcast updates; + try + match Account.logout account with + | Error error -> raise (Failure (error_message error)) + | Ok () -> () + with + | error -> + stopped := false; + raise error)) + in + let asset_scope (scope : Sync.graph_scope) : Logseq_sync_assets.Namespace.graph_scope = + { account = + { managed_sync_origin = scope.account.managed_sync_origin + ; user_id = scope.account.user_id + } + ; graph_id = scope.graph_id + } + in + let asset_error = function + | Logseq_sync_assets.Cache.Invalid message | Io message -> + Sync.Asset_invalid_content message + | Full -> Sync.Asset_invalid_content "Asset retirement storage is full." + | Stale -> Sync.Asset_cancelled + | Checksum_mismatch -> + Sync.Asset_invalid_content "Asset retirement checksum mismatch." + in + let asset_root = Filename.concat root "logseq-db-worker/assets" in + let finish deletion commit = + let scope = asset_scope (Sync.asset_deletion_scope deletion) in + let retirement = Sync.asset_deletion_retirement deletion in + let deletion = + Logseq_sync_assets.Namespace.asset_deletion_reference ~scope ~retirement + in + Logseq_sync_assets.Cache.finish_graph_deletion ~root:asset_root ~deletion ~commit + |> Result.map (fun () -> Sync.Asset_unit) + |> Result.map_error asset_error + in + let asset (request : Sync.asset_request) = + match request.action with + | Sync.Prepare_graph_asset_deletion retirement -> + Logseq_sync_assets.Cache.prepare_graph_deletion + ~root:asset_root + ~scope:(asset_scope request.scope) + ~retirement + |> Result.map (fun () -> + Sync.Asset_deletion_prepared + (Sync.asset_deletion_reference ~scope:request.scope ~retirement)) + |> Result.map_error asset_error + | Commit_graph_asset_deletion deletion -> finish deletion true + | Cancel_graph_asset_deletion deletion -> finish deletion false + | Sync.Cancel_asset_operation _ | Await_asset_operations -> Ok Sync.Asset_unit + | Close_asset_scope -> + with_account (fun () -> + if not (List.exists (fun entry -> entry.scope = request.scope) !entries) + then Ok Sync.Asset_unit + else + (let* () = Account.release_graph account request.scope.graph_id in + let* () = + match Account.selected account, !entries with + | None, entry :: _ -> Account.select account entry.scope.graph_id + | Some _, _ | None, [] -> Ok () + in + notify (); + Ok Sync.Asset_unit) + |> Result.map_error (fun _ -> Sync.Asset_cancelled)) + | _ -> Error (Sync.Asset_invalid_content "Asset request requires a graph owner") + in + Eio.Fiber.fork ~sw (fun () -> + let rec loop () = + Eio.Condition.loop_no_mutex updates (fun () -> + if !dirty || !stopped then Some () else None); + if not !stopped + then ( + dirty := false; + publish (); + loop ()) + in + loop ()); + let intents = Eio.Stream.create 256 in + Eio.Fiber.fork ~sw (fun () -> + let rec loop () = + let intent = Eio.Stream.take intents in + (match intent with + | Pure.Sync_intent (Sync.E2ee_password_submitted (challenge, password)) -> + ignore (unlock (Sync.unlock_challenge_scope challenge) password) + | Sync_intent value -> ignore (event (Pure.Sync_event value)) + | Foreground_intent foreground -> ignore (event (Pure.Set_foreground foreground)) + | Stop_intent -> shutdown ()); + notify (); + if not !stopped then loop () + in + loop ()); + let lifecycle intent = + if not !stopped + then + if Eio.Stream.length intents >= 256 + then ( + failure := Some (Dsl.Capacity_exceeded "The account command queue is full."); + notify ()) + else Eio.Stream.add intents intent + in + let services : Runner.services = + { post = lifecycle + ; owner + ; asset + ; await_io + ; shutdown + ; retire_account = + (fun ~account:_ -> Account.logout account |> Result.map_error error_message) + } + in + let validate_control control = + with_account (fun () -> + match control with + | Sign_in scope | Cancel_startup scope -> current_presentation scope + | Refresh_catalog scope -> current_startup scope + | Recover_online scope -> current_timeline scope) + in + Ok + { services + ; validate_control + ; bind = + (fun value -> + worker := Some value; + baseline := Some (Worker.view value).sync; + notify ()) + ; post = (fun value -> Option.iter (fun worker -> Worker.post worker value) !worker) + ; lifecycle + ; unlock + ; shutdown + } +;; diff --git a/logseq_db_worker/lui/journal_sync_registration.mli b/logseq_db_worker/lui/journal_sync_registration.mli new file mode 100644 index 00000000..47e2c551 --- /dev/null +++ b/logseq_db_worker/lui/journal_sync_registration.mli @@ -0,0 +1,25 @@ +type network_control = + | Sign_in of Logseq_sync_domain.Core.startup_scope + | Cancel_startup of Logseq_sync_domain.Core.startup_scope + | Refresh_catalog of Logseq_sync_domain.Core.startup_scope + | Recover_online of Logseq_sync_domain.Core.timeline_scope + +type t = + { services : Logseq_db_worker_effect_runner.Effect_runner.services + ; validate_control : network_control -> (unit, Sync_dsl.Core.error) result + ; bind : Logseq_db_worker.t -> unit + ; post : Logseq_db_worker_pure_reducer.Core.event -> unit + ; lifecycle : Logseq_db_worker_pure_reducer.Core.lifecycle_intent -> unit + ; unlock : Logseq_sync_domain.Core.graph_scope -> string -> (unit, string) result + ; shutdown : unit -> unit + } + +val create + : sw:Eio.Switch.t + -> clock:_ Eio.Time.Mono.t + -> network:_ Eio.Net.t + -> authenticator:X509.Authenticator.t + -> config:Logseq_db_worker.Config.t + -> overlay:Logseq_overlay_db.Database.dependencies + -> token:(user:string -> account_generation:int -> epoch:int64 -> refresh:bool -> (string, Sync_dsl.Core.error) result) + -> unit -> (t, Sync_dsl.Core.error) result diff --git a/logseq_db_worker/lui/logseq_db_worker_lui_service.ml b/logseq_db_worker/lui/logseq_db_worker_lui_service.ml index bfc678ba..c97980be 100644 --- a/logseq_db_worker/lui/logseq_db_worker_lui_service.ml +++ b/logseq_db_worker/lui/logseq_db_worker_lui_service.ml @@ -2,9 +2,8 @@ module Read = Worker_pure_read module Db = Logseq_db_worker module Pure = Logseq_db_worker_pure_reducer.Core module Worker_runner = Logseq_db_worker_effect_runner.Effect_runner -module Sync = Logseq_sync_pure_reducer.Core -module Unlock = Sync -module Sync_runner = Logseq_sync_effect_runner.Effect_runner +module Sync = Logseq_sync_domain.Core +module Unlock = Logseq_db_types.Sync_account module Overlay = Logseq_overlay_db.Database module ID = Journal_worker_ids module Authorized_graphs = Set.Make (String) @@ -42,25 +41,25 @@ type startup_failure_stage = | During_bootstrap | During_e2ee -type local_unavailability = Logseq_sync_pure_reducer.Core.local_unavailability = +type local_unavailability = Logseq_db_types.Sync_account.local_unavailability = | No_local_account | Local_catalog_unavailable | Local_mirror_unavailable | Local_key_unavailable | Local_restore_failed -type local_access = Logseq_sync_pure_reducer.Core.local_access = +type local_access = Logseq_db_types.Sync_account.local_access = | Local_pending | Local_available | Local_unavailable of local_unavailability -type startup_network_phase = Logseq_sync_pure_reducer.Core.startup_network_phase = +type startup_network_phase = Logseq_db_types.Sync_account.startup_network_phase = | Local_only | Online_after_timeline | Online_by_user -type startup_scope = Logseq_sync_pure_reducer.Core.startup_scope -type timeline_scope = Logseq_sync_pure_reducer.Core.timeline_scope +type startup_scope = Logseq_db_types.Sync_account.startup_scope +type timeline_scope = Logseq_db_types.Sync_account.timeline_scope type startup_facts = { authenticated : bool @@ -77,16 +76,16 @@ type startup_facts = ; presentation_generation : int } -type local_deletion_stage = Logseq_sync_pure_reducer.Core.local_deletion_stage = +type local_deletion_stage = Logseq_db_types.Sync_account.local_deletion_stage = | Closing_graph | Deleting_mirror | Clearing_selection -type local_deletion = Logseq_sync_pure_reducer.Core.local_deletion = +type local_deletion = Logseq_db_types.Sync_account.local_deletion = | Deletion_in_progress of local_deletion_stage | Deletion_failed of local_deletion_stage -type error_attention = Logseq_sync_pure_reducer.Core.error_attention = +type error_attention = Logseq_db_types.Sync_account.error_attention = | Transient_connection | Requires_attention @@ -220,15 +219,16 @@ type client_command = | Refresh_catalog of startup_scope | Begin_online_recovery of timeline_scope | Submit_e2ee_password of Unlock.unlock_challenge * string + | Submit_native_password of Logseq_db_types.Sync_account.graph_scope * string | Delete_local_cache of Sync.graph_id | Set_foreground of bool module Asset = struct - type priority = Logseq_sync_pure_reducer.Asset_transfer.priority = + type priority = Logseq_sync_asset_transfer.Core.priority = | Foreground | Background - type failure = Logseq_sync_pure_reducer.Asset_transfer.failure = + type failure = Logseq_sync_asset_transfer.Core.failure = | Network | Not_found | Checksum_mismatch @@ -237,7 +237,7 @@ module Asset = struct | Storage_full | Invalid_content of string - type availability = Logseq_sync_pure_reducer.Asset_transfer.availability = + type availability = Logseq_sync_asset_transfer.Core.availability = | Queued | Downloading | Ready of string @@ -251,15 +251,15 @@ module Asset = struct } end -type account_scope = Logseq_sync_pure_reducer.Core.account_scope -type asset_scope = Logseq_sync_pure_reducer.Core.graph_scope +type account_scope = Logseq_db_types.Sync_account.account_scope +type asset_scope = Logseq_db_types.Sync_account.graph_scope type asset_notice = Logseq_db_worker_pure_reducer.Core.asset_notice = | Asset_availability of { consumer : string ; asset : Logseq_db_types.Graph_types.Uuid.t ; descriptor : Logseq_db_types.Asset_descriptor.t - ; availability : Logseq_sync_pure_reducer.Asset_transfer.availability + ; availability : Logseq_sync_asset_transfer.Core.availability } | Asset_demand_accepted of string | Asset_backpressure of string @@ -275,7 +275,7 @@ type asset_notice = Logseq_db_worker_pure_reducer.Core.asset_notice = type asset_command = | Replace_asset_demand of { consumer : string - ; priority : Logseq_sync_pure_reducer.Asset_transfer.priority + ; priority : Logseq_sync_asset_transfer.Core.priority ; assets : Logseq_db_types.Asset_descriptor.t list } | Release_asset_demand of string @@ -339,7 +339,7 @@ type request = {registration:int64;owner:timeline_scope;observation_id:Logseq_db_types.Graph_types.Uuid.t} | Coordination_asset_demand of {registration:int64;owner:timeline_scope;request_id:Logseq_db_types.Graph_types.Uuid.t; - consumer:string;priority:Logseq_sync_pure_reducer.Asset_transfer.priority; + consumer:string;priority:Logseq_sync_asset_transfer.Core.priority; assets:Logseq_db_types.Asset_descriptor.t list} | Coordination_release_asset_demand of {registration:int64;owner:asset_scope;request_id:Logseq_db_types.Graph_types.Uuid.t;consumer:string} @@ -397,15 +397,15 @@ type request = ; command : asset_command } | Acquire_imported_file of - { scope : Logseq_sync_pure_reducer.Core.graph_scope + { scope : Logseq_db_types.Sync_account.graph_scope ; operation : Logseq_db_types.Graph_types.Uuid.t } | Acquire_asset_file of - { scope : Logseq_sync_pure_reducer.Core.graph_scope + { scope : Logseq_db_types.Sync_account.graph_scope ; handle : string } | Release_asset_file of - { scope : Logseq_sync_pure_reducer.Core.graph_scope + { scope : Logseq_db_types.Sync_account.graph_scope ; handle : string } @@ -504,7 +504,7 @@ type push = | Bootstrap_progress of bootstrap_progress | Graph_state_changed of Db.graph_state | Asset_notice of - Logseq_sync_pure_reducer.Core.graph_scope + Logseq_db_types.Sync_account.graph_scope * Logseq_db_worker_pure_reducer.Core.asset_notice let invalidation_topic = ID.Worker.Push_topic.of_int 0 @@ -627,15 +627,9 @@ let coalesce_push ~topic previous next = | _ -> next) ;; -type dependencies = - { overlay : Overlay.dependencies - ; tls_authenticator : Sync_runner.tls_authenticator - ; secrets : Sync_runner.secrets - ; crypto : Sync_runner.crypto - } +type dependencies = { overlay : Overlay.dependencies } -let dependencies ~overlay ~tls_authenticator ~secrets ~crypto = - { overlay; tls_authenticator; secrets; crypto } +let dependencies ~overlay = { overlay } ;; let production_dependencies () = @@ -657,10 +651,7 @@ let production_dependencies () = ~limits |> Result.get_ok in - let secrets = Sync_runner.apple_secrets () |> Result.get_ok in - let crypto = Sync_runner.apple_crypto () |> Result.get_ok in - let tls_authenticator = Sync_runner.system_tls_authenticator () |> Result.get_ok in - { overlay; tls_authenticator; secrets; crypto } + { overlay } ;; (* Diagnostic strings are internal signals, never display text. Unknown values @@ -799,7 +790,7 @@ let diagnostic_asset_notice notice = match notice with | Asset_availability{availability=Failed{failure;retry_scheduled=false;_};_}-> let reason,source=match failure with - | Logseq_sync_pure_reducer.Asset_transfer.Network -> "networkFailed",None + | Logseq_sync_asset_transfer.Core.Network -> "networkFailed",None | Not_found -> "notFound",None | Checksum_mismatch -> "checksumMismatch",None | Authentication -> "authenticationFailed",None | Locked -> "locked",None | Storage_full -> "storageFull",None | Invalid_content s -> "invalidContent",Some s in @@ -814,7 +805,7 @@ let diagnostic_asset_notice notice = | _ -> None ;; -let publish context = +let publish ~lifecycle context = let latest : state option ref = ref None in let resident_scopes = ref [] in let owner = ref None in @@ -827,8 +818,11 @@ let publish context = in { state with diagnostics = { groups = - (if !reports = [] then [] else [{title="Worker errors";entries}]) - @ List.filter (fun (g:diagnostic_group)->g.title<>"Worker errors") state.diagnostics.groups + (if !reports = [] then state.diagnostics.groups + else {title="Worker errors";entries=entries @ List.concat_map + (fun (group:diagnostic_group)->if group.title="Worker errors" then group.entries else []) + state.diagnostics.groups} + :: List.filter (fun (group:diagnostic_group)->group.title<>"Worker errors") state.diagnostics.groups) } } in @@ -842,6 +836,7 @@ let publish context = Option.iter emit !latest in function + | Pure.Lifecycle_intent intent -> lifecycle intent | Pure.Asset_notice (scope, notice) -> (match !latest with | Some current when current.snapshot.startup.account_generation=scope.account.account_generation @@ -883,7 +878,9 @@ let publish context = let state={state with snapshot={state.snapshot with resident_scopes=scopes}} in latest:=Some state;emit state) | Session_state_changed (scope,state) -> - Journal_worker.Session_context.emit context ~topic:session_topic (Session_states [scope,client_state state]) + let state=client_state state in + let state={state with snapshot={state.snapshot with resident_scopes= !resident_scopes}} in + Journal_worker.Session_context.emit context ~topic:session_topic (Session_states [scope,state]) | Asset_finished _ | Protected_finished _ -> ()) | Graph_state_changed state -> Journal_worker.Session_context.emit @@ -900,45 +897,6 @@ let sync_limits config = ~submission_batch_size:32 ;; -let sync_dependencies dependencies context config id_token_provider = - let environment = Journal_worker.Session_context.environment context in - let clock = Eio.Stdenv.clock environment in - Result.bind - (Sync_runner.runtime - ~fork:(fun ~sw task -> Eio.Fiber.fork ~sw task) - ~sleep:(Eio.Time.sleep clock)) - (fun runtime -> - Result.bind - (Sync_runner.transport - ~websocket_liveness: - (Sync_runner.Ping_pong { interval_seconds = 30.; timeout_seconds = 10. }) - ~tls_authenticator:dependencies.tls_authenticator - ~network:(Eio.Stdenv.net environment) - ~clock) - (fun transport -> - Result.bind - (Sync_runner.local_store - ~application_support_directory: - config.Db.Config.application_support_directory - ()) - (fun local_store -> - Result.bind - (Sync_runner.artifact_store - ~staging_directory: - (Filename.concat - config.application_support_directory - "sync-staging")) - (fun artifact_store -> - Sync_runner.dependencies - ~runtime - ~transport - ~local_store - ~artifact_store - ~secrets:dependencies.secrets - ~crypto:dependencies.crypto - ~id_token_provider)))) -;; - let client_event = function | Restore_local_account { user_id } -> Pure.Sync_event (Sync.Restore_local_account { user_id }) @@ -949,6 +907,7 @@ let client_event = function | Request_sign_in scope -> Pure.Sync_event (Sync.Sign_in_requested scope) | Cancel_online scope -> Pure.Sync_event (Sync.Online_cancelled scope) | Local_account_unavailable (scope,reason) -> Pure.Sync_event (Sync.Local_account_unavailable (scope,reason)) + | Submit_native_password _ -> invalid_arg "Native unlock requires the lifecycle registration" | Provide_token _ | Reject_token _ -> invalid_arg "token commands are handled by the service" | Configure_resident_graphs ids -> Pure.Sync_event (Sync.Resident_graphs_changed ids) @@ -968,14 +927,6 @@ let error_message = function | Sync.Invalid_config message -> message ;; -let sync_create_error_message = function - | Sync_runner.Invalid_create message -> message -;; - -let sync_dependency_error_message = function - | Sync_runner.Invalid_dependency message -> message -;; - let worker_dependency_error_message = function | Worker_runner.Invalid_dependency message -> message ;; @@ -1006,10 +957,13 @@ let discarded_asset_retain request response = | _->None ;; -let create ~(dependencies : dependencies) = +let create_with_network ~network ~(dependencies : dependencies) = let module Session = struct type t = { worker : Db.t + ; post_client : client_command -> unit + ; validate_network : Journal_sync_registration.network_control -> (unit, Sync_dsl.Core.error) result + ; delete_local : account_generation:int -> graph_id:graph_id -> (unit,string) result ; application_support_directory : string ; mutable catalog_source : Sync.graph list ; mutable authorized_graphs : Authorized_graphs.t @@ -1094,6 +1048,8 @@ let create ~(dependencies : dependencies) = | _ -> invalid_arg "Only token replies and authentication state are short controls") ~init:(fun context config -> let sw = Journal_worker.Session_context.switch context in + let clock = Journal_worker.Session_context.clock context in + let network = network ~sw ~clock (Journal_worker.Session_context.net context) in let event_sink = ref (fun (_ : Pure.event) -> ()) in let asset_diagnostic_sink = ref (fun ~scope:_ ~operation:_ ~request_operation:_ @@ -1113,16 +1069,8 @@ let create ~(dependencies : dependencies) = ~topic:auth_topic (Need_id_token { request; control })) in - let id_token_provider = - Sync_runner.id_token_provider - ~acquire:(fun (account : Sync.account_scope) -> - if !authenticated_user <> Some account.user_id - then Error "ID token request is no longer current." - else Worker_runner.acquire_id_token token_cache ~account) - ~invalidate:(fun account ~token -> - Worker_runner.invalidate_id_token token_cache ~account ~token) - in let (Managed_sync { base_url }) = config.Db.Config.target in + let registration = ref None in let selected = match sync_limits config with | Error error -> Error (error_message error) @@ -1130,29 +1078,25 @@ let create ~(dependencies : dependencies) = (match Sync.config ~managed_sync_origin:(Uri.of_string base_url) ~limits with | Error error -> Error (error_message error) | Ok sync_config -> - (match sync_dependencies dependencies context config id_token_provider with - | Error error -> Error (sync_dependency_error_message error) - | Ok runner_dependencies -> - (match - Sync_runner.create ~sw runner_dependencies - ~on_asset_diagnostic:(fun ~scope ~operation ~request_operation diagnostic -> - !asset_diagnostic_sink ~scope ~operation ~request_operation diagnostic) - ~post:(fun event -> - !event_sink (Pure.Sync_event event)) - with - | Error error -> Error (sync_create_error_message error) - | Ok runner -> - Ok - ( sync_config - , Worker_runner.sync_runner - ~retire_account:(Sync_runner.retire_account runner) - ~submit:(Sync_runner.submit runner) - ~shutdown:(fun () -> Sync_runner.shutdown runner) - () )))) + let tokens=Hashtbl.create 4 in + (match Journal_sync_registration.create ~sw ~clock + ~network ~authenticator:(Ca_certs_nss.authenticator () |> Result.get_ok) + ~config ~overlay:dependencies.overlay + ~token:(fun ~user ~account_generation ~epoch:_ ~refresh -> + let account:Sync.account_scope={managed_sync_origin=Uri.of_string base_url;user_id=user; + account_generation;presentation_generation=0;lifecycle_generation=0L} in + if !authenticated_user<>Some user then Error(Sync_dsl.Core.Authentication_failure "Authentication authority changed") else ( + if refresh then Option.iter(fun token->Worker_runner.invalidate_id_token token_cache ~account ~token)(Hashtbl.find_opt tokens user); + Worker_runner.acquire_id_token token_cache ~account + |> Result.map(fun token->Hashtbl.replace tokens user token;token) + |> Result.map_error(fun message->Sync_dsl.Core.Authentication_failure message))) () with + | Error error -> Error(Sync_dsl.Core.error_message error) + | Ok bridge -> registration:=Some bridge;Ok(sync_config,bridge.services)) +) in match selected with | Error message -> Error message - | Ok (sync_config, selected_sync_runner) -> + | Ok (sync_config, selected_services) -> (match Worker_runner.runtime ~sleep: @@ -1168,14 +1112,23 @@ let create ~(dependencies : dependencies) = ~runtime ~config ~overlay:dependencies.overlay - ~sync_runner:selected_sync_runner - ~publish:(publish context) + ~services:selected_services + ~publish:(publish ~lifecycle:selected_services.post context) with | Error error -> Error (worker_dependency_error_message error) | Ok runner_dependencies -> - (match Db.create ~sw ~config:pure_config ~runner_dependencies with + (match Db.create_lifecycle ~sw ~config:pure_config ~runner_dependencies with | Error (Db.Invalid_create message) -> Error message | Ok worker -> + Option.iter(fun bridge->bridge.Journal_sync_registration.bind worker)!registration; + let post_client command=match !registration,command with + | Some bridge,Submit_native_password(scope,password)-> + bridge.Journal_sync_registration.unlock scope password + |> Result.fold ~ok:Fun.id ~error:failwith + | Some bridge,_->bridge.Journal_sync_registration.post(client_event command) + | None,_->Db.post worker(client_event command) in + let delete_local ~account_generation ~graph_id= + Db.delete_local_copy worker ~account_generation ~graph_id in event_sink := Db.post worker; asset_diagnostic_sink := Db.report_asset_diagnostic worker; let auth_seen = ref false in @@ -1200,9 +1153,7 @@ let create ~(dependencies : dependencies) = Eio.Condition.loop_no_mutex auth_wake (fun () -> !auth_pending) in auth_pending := None; - Db.post - worker - (client_event (Reconcile_authenticated_user { user_id })); + post_client (Reconcile_authenticated_user { user_id }); auth_enqueued_sequence := max !auth_enqueued_sequence sequence; Eio.Condition.broadcast auth_wake; deliver () @@ -1211,6 +1162,11 @@ let create ~(dependencies : dependencies) = Ok Session. { worker + ; post_client + ; validate_network=(fun control -> match !registration with + | Some bridge -> bridge.Journal_sync_registration.validate_control control + | None -> Error Sync_dsl.Core.Stale_context) + ; delete_local ; application_support_directory = config.Db.Config.application_support_directory ; catalog_source = [] ; authorized_graphs = Authorized_graphs.empty @@ -1492,7 +1448,7 @@ let create ~(dependencies : dependencies) = | Some scope -> let event=match command with | Retry_upload operation -> Pure.Upload_requested_for_scope (scope,operation,Logseq_db_worker_pure_reducer.Asset_upload.Retry) - | Replace_asset_demand {consumer;priority;assets} -> Pure.Asset_requested_for_scope (scope,Logseq_sync_pure_reducer.Asset_transfer.Replace {consumer;priority;assets}) + | Replace_asset_demand {consumer;priority;assets} -> Pure.Asset_requested_for_scope (scope,Logseq_sync_asset_transfer.Core.Replace {consumer;priority;assets}) | Release_asset_demand consumer -> Pure.Asset_requested_for_scope (scope,Release consumer) | Retry_asset asset -> Pure.Asset_requested_for_scope (scope,Retry asset) in Db.post session.worker event;Ok Client_command_completed) @@ -1500,7 +1456,7 @@ let create ~(dependencies : dependencies) = Eio.Condition.loop_no_mutex session.auth_wake (fun () -> if !(session.auth_enqueued_sequence) >= !(session.auth_sequence) then Some () else None); Ok (Local_copy_deleted (request, - Db.delete_local_copy session.worker ~account_generation:request.account_generation + session.Session.delete_local ~account_generation:request.account_generation ~graph_id:request.graph_id)) | Inspect_local_copy request -> let snapshot = (Db.view session.worker).sync.snapshot in @@ -1541,7 +1497,7 @@ let create ~(dependencies : dependencies) = Pure.Asset_requested { graph_generation ; event = - Logseq_sync_pure_reducer.Asset_transfer.Replace + Logseq_sync_asset_transfer.Core.Replace { consumer; priority; assets } } | Release_asset_demand consumer -> @@ -1568,27 +1524,40 @@ let create ~(dependencies : dependencies) = "host rejected request"; Ok Client_command_completed | Client_command (Reconcile_authenticated_user _) -> - (* Latest controls may coalesce an identical user even after local startup - closed the network gate. An admitted fresh verification must still - reach Core. Reassert the latest observed SDK identity, never the - possibly superseded value carried by an old admitted request. *) - if !(session.auth_seen) then session.queue_auth !(session.authenticated_user); Eio.Condition.loop_no_mutex session.auth_wake (fun () -> if !(session.auth_enqueued_sequence) >= !(session.auth_sequence) then Some () else None); Ok Client_command_completed | Client_command (Restore_local_account { user_id }) -> - Db.post session.worker (client_event (Restore_local_account { user_id })); + session.Session.post_client (Restore_local_account { user_id }); (* Local restoration may arrive after the SDK authentication event. Reassert the observed SDK state after restoration's ordered post. *) if !(session.auth_seen) then session.queue_auth !(session.authenticated_user); Ok Client_command_completed | Client_command (Delete_local_cache _) -> Error "Local deletion requires an account-scoped request." + | Client_command ((Request_sign_in _ | Cancel_online _ | Refresh_catalog _ | Begin_online_recovery _) as command) -> + let control = match command with + | Request_sign_in scope -> Journal_sync_registration.Sign_in scope + | Cancel_online scope -> Cancel_startup scope + | Refresh_catalog scope -> Refresh_catalog scope + | Begin_online_recovery scope -> Recover_online scope + | _ -> assert false in + (match session.Session.validate_network control with + | Error error -> Error (Sync_dsl.Core.error_message error) + | Ok () -> + session.post_client command; + (match command with + | Request_sign_in _ when !(session.auth_seen) -> session.queue_auth !(session.authenticated_user) + | _ -> ()); + Ok Client_command_completed) | Client_command command -> (match command with Submit_e2ee_password _ -> unlock_queue_log context | _ -> ()); - Db.post session.worker (client_event command); + session.Session.post_client command; + (match command with + | Request_sign_in _ when !(session.auth_seen) -> session.queue_auth !(session.authenticated_user) + | _ -> ()); Ok Client_command_completed ) ~shutdown:(fun session -> @@ -1597,8 +1566,16 @@ let create ~(dependencies : dependencies) = () ;; +let create ~dependencies = + create_with_network ~network:(fun ~sw:_ ~clock:_ net -> net) ~dependencies + let service = create ~dependencies:(production_dependencies ()) +module For_testing = struct + let service_with_network network = + create_with_network ~network ~dependencies:(production_dependencies ()) +end + let answer_token client request answer = let command = match answer with diff --git a/logseq_db_worker/lui/logseq_db_worker_lui_service.mli b/logseq_db_worker/lui/logseq_db_worker_lui_service.mli index 9bb892b8..411dee03 100644 --- a/logseq_db_worker/lui/logseq_db_worker_lui_service.mli +++ b/logseq_db_worker/lui/logseq_db_worker_lui_service.mli @@ -4,14 +4,14 @@ type graph = Logseq_db_types.Managed_graph.t (** Narrow projection of the unlock operation; construction evidence stays with the sync owner. UI clients need not depend on the sync library directly. *) module Unlock : sig - type graph_scope = Logseq_sync_pure_reducer.Core.graph_scope - type unlock_challenge = Logseq_sync_pure_reducer.Core.unlock_challenge - type unlock_submission = Logseq_sync_pure_reducer.Core.unlock_submission - type unlock_attached = Logseq_sync_pure_reducer.Core.unlock_attached - type unlock_recovery = Logseq_sync_pure_reducer.Core.unlock_recovery = + type graph_scope = Logseq_db_types.Sync_account.graph_scope + type unlock_challenge = Logseq_db_types.Sync_account.unlock_challenge + type unlock_submission = Logseq_db_types.Sync_account.unlock_submission + type unlock_attached = Logseq_db_types.Sync_account.unlock_attached + type unlock_recovery = Logseq_db_types.Sync_account.unlock_recovery = | Retry_password of unlock_challenge * string | Retry_preparation of string - type unlock_phase = Logseq_sync_pure_reducer.Core.unlock_phase = + type unlock_phase = Logseq_db_types.Sync_account.unlock_phase = | Unlock_inactive | Unlock_challenge of unlock_challenge | Unlock_running of unlock_submission @@ -44,25 +44,25 @@ type startup_failure_stage = | During_bootstrap | During_e2ee -type local_unavailability = Logseq_sync_pure_reducer.Core.local_unavailability = +type local_unavailability = Logseq_db_types.Sync_account.local_unavailability = | No_local_account | Local_catalog_unavailable | Local_mirror_unavailable | Local_key_unavailable | Local_restore_failed -type local_access = Logseq_sync_pure_reducer.Core.local_access = +type local_access = Logseq_db_types.Sync_account.local_access = | Local_pending | Local_available | Local_unavailable of local_unavailability -type startup_network_phase = Logseq_sync_pure_reducer.Core.startup_network_phase = +type startup_network_phase = Logseq_db_types.Sync_account.startup_network_phase = | Local_only | Online_after_timeline | Online_by_user -type startup_scope = Logseq_sync_pure_reducer.Core.startup_scope -type timeline_scope = Logseq_sync_pure_reducer.Core.timeline_scope +type startup_scope = Logseq_db_types.Sync_account.startup_scope +type timeline_scope = Logseq_db_types.Sync_account.timeline_scope type startup_facts = { authenticated : bool @@ -79,16 +79,16 @@ type startup_facts = ; presentation_generation : int } -type local_deletion_stage = Logseq_sync_pure_reducer.Core.local_deletion_stage = +type local_deletion_stage = Logseq_db_types.Sync_account.local_deletion_stage = | Closing_graph | Deleting_mirror | Clearing_selection -type local_deletion = Logseq_sync_pure_reducer.Core.local_deletion = +type local_deletion = Logseq_db_types.Sync_account.local_deletion = | Deletion_in_progress of local_deletion_stage | Deletion_failed of local_deletion_stage -type error_attention = Logseq_sync_pure_reducer.Core.error_attention = +type error_attention = Logseq_db_types.Sync_account.error_attention = | Transient_connection | Requires_attention @@ -149,15 +149,16 @@ type client_command = | Refresh_catalog of startup_scope | Begin_online_recovery of timeline_scope | Submit_e2ee_password of Unlock.unlock_challenge * string + | Submit_native_password of Logseq_db_types.Sync_account.graph_scope * string | Delete_local_cache of graph_id | Set_foreground of bool module Asset : sig - type priority = Logseq_sync_pure_reducer.Asset_transfer.priority = + type priority = Logseq_sync_asset_transfer.Core.priority = | Foreground | Background - type failure = Logseq_sync_pure_reducer.Asset_transfer.failure = + type failure = Logseq_sync_asset_transfer.Core.failure = | Network | Not_found | Checksum_mismatch @@ -166,7 +167,7 @@ module Asset : sig | Storage_full | Invalid_content of string - type availability = Logseq_sync_pure_reducer.Asset_transfer.availability = + type availability = Logseq_sync_asset_transfer.Core.availability = | Queued | Downloading | Ready of string @@ -180,15 +181,15 @@ module Asset : sig } end -type account_scope = Logseq_sync_pure_reducer.Core.account_scope -type asset_scope = Logseq_sync_pure_reducer.Core.graph_scope +type account_scope = Logseq_db_types.Sync_account.account_scope +type asset_scope = Logseq_db_types.Sync_account.graph_scope type asset_notice = Logseq_db_worker_pure_reducer.Core.asset_notice = | Asset_availability of { consumer : string ; asset : Logseq_db_types.Graph_types.Uuid.t ; descriptor : Logseq_db_types.Asset_descriptor.t - ; availability : Logseq_sync_pure_reducer.Asset_transfer.availability + ; availability : Logseq_sync_asset_transfer.Core.availability } | Asset_demand_accepted of string | Asset_backpressure of string @@ -204,7 +205,7 @@ type asset_notice = Logseq_db_worker_pure_reducer.Core.asset_notice = type asset_command = | Replace_asset_demand of { consumer : string - ; priority : Logseq_sync_pure_reducer.Asset_transfer.priority + ; priority : Logseq_sync_asset_transfer.Core.priority ; assets : Logseq_db_types.Asset_descriptor.t list } | Release_asset_demand of string @@ -263,7 +264,7 @@ type request = {registration:int64;owner:timeline_scope;observation_id:Logseq_db_types.Graph_types.Uuid.t} | Coordination_asset_demand of {registration:int64;owner:timeline_scope;request_id:Logseq_db_types.Graph_types.Uuid.t; - consumer:string;priority:Logseq_sync_pure_reducer.Asset_transfer.priority; + consumer:string;priority:Logseq_sync_asset_transfer.Core.priority; assets:Logseq_db_types.Asset_descriptor.t list} | Coordination_release_asset_demand of {registration:int64;owner:asset_scope;request_id:Logseq_db_types.Graph_types.Uuid.t;consumer:string} @@ -321,15 +322,15 @@ type request = ; command : asset_command } | Acquire_imported_file of - { scope : Logseq_sync_pure_reducer.Core.graph_scope + { scope : Logseq_db_types.Sync_account.graph_scope ; operation : Logseq_db_types.Graph_types.Uuid.t } | Acquire_asset_file of - { scope : Logseq_sync_pure_reducer.Core.graph_scope + { scope : Logseq_db_types.Sync_account.graph_scope ; handle : string } | Release_asset_file of - { scope : Logseq_sync_pure_reducer.Core.graph_scope + { scope : Logseq_db_types.Sync_account.graph_scope ; handle : string } @@ -412,7 +413,7 @@ type push = | Bootstrap_progress of bootstrap_progress | Graph_state_changed of Logseq_db_worker.graph_state | Asset_notice of - Logseq_sync_pure_reducer.Core.graph_scope + Logseq_db_types.Sync_account.graph_scope * Logseq_db_worker_pure_reducer.Core.asset_notice val invalidation_topic : Journal_worker_ids.Worker.Push_topic.t @@ -435,9 +436,6 @@ type dependencies val dependencies : overlay:Logseq_overlay_db.Database.dependencies - -> tls_authenticator:Logseq_sync_effect_runner.Effect_runner.tls_authenticator - -> secrets:Logseq_sync_effect_runner.Effect_runner.secrets - -> crypto:Logseq_sync_effect_runner.Effect_runner.crypto -> dependencies (** Pure cleanup routing for the Mailbox's exactly-once unpublished success. @@ -451,6 +449,12 @@ val create val service : (Logseq_db_worker.Config.t, request, response, push) Journal_worker.Service.t +module For_testing : sig + val service_with_network + : (sw:Eio.Switch.t -> clock:Journal_worker.mono_clock -> Journal_worker.net -> Journal_worker.net) + -> (Logseq_db_worker.Config.t, request, response, push) Journal_worker.Service.t +end + (** Fixed operation, phase and display text for an internal diagnostic signal. Unknown signals use a safe fallback without displaying the input. *) val diagnostic_report : string -> string * string * string diff --git a/logseq_db_worker/spec/effect_runner/dune b/logseq_db_worker/spec/effect_runner/dune index 99a2ba5a..8054f3f5 100644 --- a/logseq_db_worker/spec/effect_runner/dune +++ b/logseq_db_worker/spec/effect_runner/dune @@ -12,4 +12,7 @@ logseq_db_worker.contract logseq_db_worker.pure_reducer logseq_overlay_db - logseq_sync.pure_reducer)) + execute + sync_dsl + logseq_sync.worker_owner + logseq_sync.domain logseq_sync.asset_transfer)) diff --git a/logseq_db_worker/spec/effect_runner/effect_runner.mli b/logseq_db_worker/spec/effect_runner/effect_runner.mli index 1d88bd17..72898d44 100644 --- a/logseq_db_worker/spec/effect_runner/effect_runner.mli +++ b/logseq_db_worker/spec/effect_runner/effect_runner.mli @@ -13,7 +13,7 @@ -> post:(Logseq_db_worker_pure_reducer.Core.event -> unit) -> recovery_current:(Logseq_db_worker_pure_reducer.Core.upload_recovery_ticket -> bool) -> upload_current:(Logseq_db_worker_pure_reducer.Asset_upload.ticket -> bool) - -> asset_current:(Logseq_sync_pure_reducer.Asset_transfer.ticket -> bool) + -> asset_current:(Logseq_sync_asset_transfer.Core.ticket -> bool) -> (t, create_error) result val submit : t -> Logseq_db_worker_pure_reducer.Core.instruction -> unit val shutdown : t -> unit @@ -21,7 +21,17 @@ type t type runtime -type sync_runner + +(** [await_io] joins physical work; cleanup remains legal after fencing. *) +type services = + { post : Logseq_db_worker_pure_reducer.Core.lifecycle_intent -> unit + ; owner : scope:Logseq_sync_domain.Core.graph_scope + -> (Sync_dsl.Core.identity * Logseq_sync_worker_owner.Core.t, string) result + ; asset : Logseq_sync_domain.Core.asset_request -> Logseq_sync_domain.Core.asset_result + ; await_io : scope:Logseq_sync_domain.Core.effect_scope -> (unit, string) result + ; shutdown : unit -> unit + ; retire_account : account:Logseq_sync_domain.Core.account_scope -> (unit, string) result + } type dependencies type dependency_error = Invalid_dependency of string type create_error = Invalid_create of string @@ -64,7 +74,7 @@ val id_token_cache val acquire_id_token : id_token_cache - -> account:Logseq_sync_pure_reducer.Core.account_scope + -> account:Logseq_sync_domain.Core.account_scope -> (string, string) result val provide_id_token : id_token_cache -> id_token_request -> string -> unit @@ -72,7 +82,7 @@ val reject_id_token : id_token_cache -> id_token_request -> string -> unit val invalidate_id_token : id_token_cache - -> account:Logseq_sync_pure_reducer.Core.account_scope + -> account:Logseq_sync_domain.Core.account_scope -> token:string -> unit @@ -86,18 +96,30 @@ val runtime -> sleep:(float -> unit) -> (runtime, dependency_error) result -val sync_runner - : submit:(Logseq_sync_pure_reducer.Core.runner_effect -> unit) - -> shutdown:(unit -> unit) - -> retire_account:(account:Logseq_sync_pure_reducer.Core.account_scope -> (unit,string)result) - -> unit - -> sync_runner +val attach_database_owner : t -> scope:Logseq_sync_domain.Core.graph_scope + -> Execute.Core.database_owner -> (Logseq_db_worker_pure_reducer.Core.database_opened, string) result + +val detach_database_owner : t -> scope:Logseq_sync_domain.Core.graph_scope -> (unit, string) result +val delete_local_copy : t -> ?prepared_handoff:Logseq_db_types.Graph_types.Uuid.t + -> account:Logseq_sync_domain.Core.account_scope + -> graph_id:Logseq_db_types.Graph_types.Uuid.t -> unit -> (unit, string) result + +val database_owner + : t -> scope:Logseq_sync_domain.Core.graph_scope + -> (Execute.Core.database_owner, string) result + +(** Borrowed until Worker shutdown; checked Sync retirement must precede database close. *) +val sync_upload_store : t -> (Logseq_sync_worker_owner.Core.store, string) result + +val worker_owner + : t -> scope:Logseq_sync_domain.Core.graph_scope + -> (Sync_dsl.Core.identity * Logseq_sync_worker_owner.Core.t, string) result val dependencies : runtime:runtime -> config:Logseq_db_worker_contract.Config.t -> overlay:Logseq_overlay_db.Database.dependencies - -> sync_runner:sync_runner + -> services:services -> publish:(Logseq_db_worker_pure_reducer.Core.output -> unit) -> (dependencies, dependency_error) result @@ -107,7 +129,7 @@ val create -> post:(Logseq_db_worker_pure_reducer.Core.event -> unit) -> recovery_current:(Logseq_db_worker_pure_reducer.Core.upload_recovery_ticket -> bool) -> upload_current:(Logseq_db_worker_pure_reducer.Asset_upload.ticket -> bool) - -> asset_current:(Logseq_sync_pure_reducer.Asset_transfer.ticket -> bool) + -> asset_current:(Logseq_sync_asset_transfer.Core.ticket -> bool) -> (t, create_error) result (** {1 Capability details} *) @@ -118,7 +140,7 @@ val submit : t -> Logseq_db_worker_pure_reducer.Core.instruction -> unit internal request identifier is used for correlation and is never displayed. *) val report_asset_diagnostic : t - -> scope:Logseq_sync_pure_reducer.Core.graph_scope + -> scope:Logseq_sync_domain.Core.graph_scope -> operation:string -> request_operation:string -> Logseq_db_types.Failure_diagnostic.t @@ -135,13 +157,13 @@ val await_reply val retain_asset_file : t - -> scope:Logseq_sync_pure_reducer.Core.graph_scope + -> scope:Logseq_sync_domain.Core.graph_scope -> handle:string -> (string * string) option val release_asset_file : t - -> scope:Logseq_sync_pure_reducer.Core.graph_scope + -> scope:Logseq_sync_domain.Core.graph_scope -> handle:string -> unit @@ -149,7 +171,7 @@ val release_asset_file retired scope and storage failure remain errors; callers retain cleanup ownership and retry the same handle rather than inventing an ACK. *) val release_asset_file_checked - : t -> scope:Logseq_sync_pure_reducer.Core.graph_scope -> handle:string + : t -> scope:Logseq_sync_domain.Core.graph_scope -> handle:string -> (unit, string) result (** Transfer a discarded successful acquisition to this actual resource owner. @@ -159,44 +181,44 @@ val release_asset_file_checked its exact attempt and cannot be settled by an earlier ACK. Shutdown does not turn pending cleanup into a success. These counts are required by handoff. *) val abandon_asset_file - : t -> scope:Logseq_sync_pure_reducer.Core.graph_scope -> handle:string -> unit + : t -> scope:Logseq_sync_domain.Core.graph_scope -> handle:string -> unit val unsettled_orphan_retains : t -> int (** Capture after the serial reducer has submitted the retiring effects. Await joins their IO and callback barrier, including stale-result release children; pending physical failures remain errors. New unrelated work is not a fence. *) type asset_cleanup_snapshot -val asset_cleanup_snapshot : t -> scope:Logseq_sync_pure_reducer.Core.graph_scope -> asset_cleanup_snapshot +val asset_cleanup_snapshot : t -> scope:Logseq_sync_domain.Core.graph_scope -> asset_cleanup_snapshot val await_asset_cleanup : t -> asset_cleanup_snapshot -> (unit,string) result val prepare_import : t - -> context:Logseq_sync_pure_reducer.Core.asset_context + -> context:Logseq_sync_domain.Core.asset_context -> Logseq_db_types.Asset_import.t -> current:(unit -> bool) -> (Logseq_db_types.Asset_upload_intent.t, string) result val retain_imported_file : t - -> scope:Logseq_sync_pure_reducer.Core.graph_scope + -> scope:Logseq_sync_domain.Core.graph_scope -> operation:Logseq_db_types.Graph_types.Uuid.t -> (string * string) option (** Original typed durable intents use the existing scoped database attachment. Rechecks authority before committing; accepted interruption remains unknown. *) -val durable_mutation : t -> scope:Logseq_sync_pure_reducer.Core.graph_scope +val durable_mutation : t -> scope:Logseq_sync_domain.Core.graph_scope -> intent:Logseq_db_worker_contract.Worker_pure_mutation.durable_intent -> current:(unit -> bool) -> Logseq_db_worker_contract.Worker_pure_mutation.outcome (** Read-only original-proof lookup and one bounded canonical snapshot read. *) -val confirm_mutation : t -> scope:Logseq_sync_pure_reducer.Core.graph_scope +val confirm_mutation : t -> scope:Logseq_sync_domain.Core.graph_scope -> read:Worker_pure_read.packed_query -> intent:Logseq_db_worker_contract.Worker_pure_mutation.durable_intent -> original:(string * string * string) -> current:(unit -> bool) -> Worker_pure_read.answer (** Read-only item/byte bounded recovery. Complete immutable intent bytes and original proofs are returned; pending work remains Unknown and is not run. *) -val recover_mutations : t -> scope:Logseq_sync_pure_reducer.Core.graph_scope +val recover_mutations : t -> scope:Logseq_sync_domain.Core.graph_scope -> after:int64 option -> maximum_items:int -> maximum_bytes:int -> current:(unit -> bool) -> (Logseq_db_worker_contract.Worker_pure_mutation.recovery_page,string) result @@ -204,7 +226,7 @@ val recover_mutations : t -> scope:Logseq_sync_pure_reducer.Core.graph_scope (** Acknowledges only a stored rejection or the exact immutable commit proof. The caller admits committed ACK only after applying a confirmation in the current presentation. Proof and outbox contents are retained. *) -val acknowledge_mutation : t -> scope:Logseq_sync_pure_reducer.Core.graph_scope +val acknowledge_mutation : t -> scope:Logseq_sync_domain.Core.graph_scope -> intent:Logseq_db_worker_contract.Worker_pure_mutation.durable_intent -> original:(string * string * string) option -> current:(unit -> bool) -> (unit,string) result @@ -221,7 +243,7 @@ val register_observation : t -> registration:int64 -> id:Logseq_db_types.Graph_types.Uuid.t - -> scope:Logseq_sync_pure_reducer.Core.graph_scope + -> scope:Logseq_sync_domain.Core.graph_scope -> read:Worker_pure_read.packed_query -> current:(unit -> bool) -> notify:(epoch:int64 -> unit) @@ -231,14 +253,14 @@ val refresh_observation : t -> registration:int64 -> id:Logseq_db_types.Graph_types.Uuid.t - -> scope:Logseq_sync_pure_reducer.Core.graph_scope + -> scope:Logseq_sync_domain.Core.graph_scope -> (int64 * Worker_pure_read.answer, string) result val release_observation : t -> registration:int64 -> id:Logseq_db_types.Graph_types.Uuid.t - -> scope:Logseq_sync_pure_reducer.Core.graph_scope + -> scope:Logseq_sync_domain.Core.graph_scope -> (unit, string) result val release_observations : t -> registration:int64 -> unit @@ -250,22 +272,22 @@ val retire_observation_registration : t -> registration:int64 -> unit val observation_creator_finished : t -> registration:int64 -> id:Logseq_db_types.Graph_types.Uuid.t -> unit val finalize_observation : t -> registration:int64 -> id:Logseq_db_types.Graph_types.Uuid.t - -> owner:Logseq_sync_pure_reducer.Core.timeline_scope -> (unit,string) result + -> owner:Logseq_sync_domain.Core.timeline_scope -> (unit,string) result (** Retire only an unknown logical observation ID; no implicit-current graph or invented account lifecycle. A known physical allocation must use its owner. *) val release_unallocated_observation : t -> registration:int64 -> id:Logseq_db_types.Graph_types.Uuid.t - -> owner:Logseq_sync_pure_reducer.Core.timeline_scope -> (unit,string) result + -> owner:Logseq_sync_domain.Core.timeline_scope -> (unit,string) result val observation_scope : t -> registration:int64 -> id:Logseq_db_types.Graph_types.Uuid.t - -> Logseq_sync_pure_reducer.Core.graph_scope option + -> Logseq_sync_domain.Core.graph_scope option (** Match the original delivered batch and release the actual physical windows. *) -val acknowledge_change_batch : t -> registration:int64 -> scope:Logseq_sync_pure_reducer.Core.graph_scope +val acknowledge_change_batch : t -> registration:int64 -> scope:Logseq_sync_domain.Core.graph_scope -> batch:Worker_pure_read.change_batch -> current:(unit -> bool) -> (Worker_pure_read.change_ack,Worker_pure_read.error) result @@ -277,38 +299,38 @@ val retire_changefeed : t -> registration:int64 -> unit join already reserved protected tasks even if Service waiters canceled, then join their actual serial reducer callback barrier. This is only writer evidence; resources, exclusive local-copy ownership and host permits require further ACKs. *) -val quiesce_writers : t -> scope:Logseq_sync_pure_reducer.Core.graph_scope -> (unit,string) result +val quiesce_writers : t -> scope:Logseq_sync_domain.Core.graph_scope -> (unit,string) result (** Join the Sync owner's captured ordinary/protected/timer/WebSocket/asset I/O and actual callback posting via its existing serial event channel. The caller first retires target producers; this function itself grants no admission or host permit and does not stop the shared client. *) -val await_sync_io : t -> scope:Logseq_sync_pure_reducer.Core.effect_scope -> (unit,string) result +val await_sync_io : t -> scope:Logseq_sync_domain.Core.effect_scope -> (unit,string) result (** Actual retained physical scope roster, including old failed cleanup owners. This is captured by the backend, never supplied by the host as a substitute for ownership. No admission or acknowledgement is implied. *) -val physical_host_scopes : t -> Logseq_sync_pure_reducer.Core.graph_scope Rrbvec.t +val physical_host_scopes : t -> Logseq_sync_domain.Core.graph_scope Rrbvec.t (** The caller has withdrawn producers and gated reopen. Joins real graph tasks, Sync callbacks, database/listener close and resource children; success also requires the actual scoped cache's checked close. A failed DB close remains a physical failure, even if its logical session has already retired. *) -val settle_host_scope : t -> scope:Logseq_sync_pure_reducer.Core.graph_scope -> (unit,string) result +val settle_host_scope : t -> scope:Logseq_sync_domain.Core.graph_scope -> (unit,string) result (** Complete the real Sync account owner's checked secret/cache cleanup after the pure owner has fenced its lifecycle and all admitted IO has joined. *) -val retire_account:t -> account:Logseq_sync_pure_reducer.Core.account_scope -> (unit,string)result +val retire_account:t -> account:Logseq_sync_domain.Core.account_scope -> (unit,string)result (** Hold the existing exclusive prepared_mirror_deletion owner before issuing a removal receipt, including copies whose DB cannot be opened. Retry may rebind an unconsumed failed round's same held owner; it cannot borrow a deletion in progress. No namespace mutation or new identity-probe API is involved. *) val prepare_host_copy:t -> registration:int64 -> handoff_id:Logseq_db_types.Graph_types.Uuid.t - -> owner:Logseq_sync_pure_reducer.Core.account_scope -> graph_id:Logseq_db_types.Graph_types.Uuid.t + -> owner:Logseq_sync_domain.Core.account_scope -> graph_id:Logseq_db_types.Graph_types.Uuid.t -> (Logseq_overlay_db.Types.mirror_generation,string)result val bind_host_copy:t -> handoff_id:Logseq_db_types.Graph_types.Uuid.t - -> Logseq_sync_pure_reducer.Core.mirror_deletion -> (unit,Logseq_db_worker_pure_reducer.Core.effect_error)result + -> Logseq_sync_domain.Core.mirror_deletion -> (unit,Logseq_db_worker_pure_reducer.Core.effect_error)result val release_host_copy:t -> registration:int64 -> (unit,string)result (** {2 Import source recovery} *) @@ -316,17 +338,17 @@ val release_host_copy:t -> registration:int64 -> (unit,string)result (** Read-only complete source recovery: missing store is known empty; no schema, staging, acquisition or execution is performed. Cursor pages obey both item and canonical encoded byte budgets. Every proof rechecks its logical owner. *) -val recover_import_inputs:t -> scope:Logseq_sync_pure_reducer.Core.graph_scope +val recover_import_inputs:t -> scope:Logseq_sync_domain.Core.graph_scope -> after:int64 option -> maximum_items:int -> maximum_bytes:int -> current:(unit -> bool) -> (Logseq_db_types.Asset_import.Recovery.page,string)result -val confirm_import_input:t -> scope:Logseq_sync_pure_reducer.Core.graph_scope +val confirm_import_input:t -> scope:Logseq_sync_domain.Core.graph_scope -> input:string -> current:(unit -> bool) -> (Logseq_db_types.Asset_import.Recovery.item option,string)result (** Writes only the pending-membership ACK for exact stored original Prepared evidence, under the existing writer fence. Complete bytes and proof remain. *) -val acknowledge_import_input:t -> scope:Logseq_sync_pure_reducer.Core.graph_scope +val acknowledge_import_input:t -> scope:Logseq_sync_domain.Core.graph_scope -> input:string -> prepared:string -> current:(unit -> bool) -> (unit,string)result (** Stop this runner and its delegated Sync runner. Actual scoped cleanup and diff --git a/logseq_db_worker/spec/pure_reducer/asset_upload.mli b/logseq_db_worker/spec/pure_reducer/asset_upload.mli index 746a2cf7..21ed33b0 100644 --- a/logseq_db_worker/spec/pure_reducer/asset_upload.mli +++ b/logseq_db_worker/spec/pure_reducer/asset_upload.mli @@ -16,7 +16,7 @@ module Intent = Logseq_db_types.Asset_upload_intent -module Scope = Logseq_sync_pure_reducer.Core +module Scope = Logseq_sync_domain.Core type t diff --git a/logseq_db_worker/spec/pure_reducer/core.mli b/logseq_db_worker/spec/pure_reducer/core.mli index 6cf7f1ec..fb9e1dfb 100644 --- a/logseq_db_worker/spec/pure_reducer/core.mli +++ b/logseq_db_worker/spec/pure_reducer/core.mli @@ -1,47 +1,13 @@ -(** {1 TLDR} - - Own pure DB Worker lifecycle and request routing, including graph state, - pending requests, Sync delegation and resource cleanup ledgers. The host owns - {!state}; {!initial} validates construction, and {!step} returns a {!transition} - with [next] and ordered [effects]. Install [next] before interpreting effects; - callbacks queue exact completion events for a later serialized turn. No DB, - network, clock or callback I/O occurs in this reducer. - - The runner GADT and dependent instruction/event families remain below. - These are accurate entry-point excerpts, not additional declarations: - {[ - val initial : config -> (state, create_error) result - val step : state -> event -> transition - val view : state -> view - ]} - Cleanup admission is not a physical ACK. Scope/attempt identity must still be - checked at runtime; process-local tickets do not replace durable intent. *) - -(** {1 Minimal usage example} - - The caller supplies validated [config] and the worker runner's [interpret]: - {[ - match initial config with - | Error error -> Error error - | Ok first -> - let state = ref first in - let transition = step !state Start in - state := transition.next; - List.iter interpret transition.effects; - Ok state - ]} - [interpret] queues real completions, preserving the original effect ticket. - The host applies later events with the same install-before-interpret order. - On retirement feed [Shutdown], then continue processing admitted cleanup and - callback barriers; the view's shutdown flag alone is not a physical join. *) - -(** {1 Construction and validation} *) +(** Worker read routing and resource cleanup. App owns account and graph lifecycle. + Install [next] before interpreting effects; enqueue exact completions later. + Projection withdrawal rejects new reads but retains admitted terminal replies. + Cleanup admission and [shutdown] are not physical settlement. *) type config val config : worker:Logseq_db_worker_contract.Config.t - -> sync:Logseq_sync_pure_reducer.Core.config + -> sync:Logseq_sync_domain.Core.config -> config (** {1 Capability details} *) @@ -92,7 +58,7 @@ type lifecycle_result = | Lifecycle_failed of int * Logseq_db_worker_contract.Error.t type sync_worker_result = - { event : Logseq_sync_pure_reducer.Core.event option + { event : Logseq_sync_domain.Core.event option ; lifecycle : lifecycle_result } @@ -104,7 +70,7 @@ type _ runner_request = -> Worker_pure_read.answer runner_request | Close_database : database_handle -> unit runner_request | Handle_sync_worker_effect : - Logseq_sync_pure_reducer.Core.worker_effect + Logseq_sync_domain.Core.worker_effect -> sync_worker_result runner_request type runner_effect = Request : ticket * 'a runner_request -> runner_effect @@ -121,7 +87,7 @@ type asset_notice = { consumer : string ; asset : Logseq_db_types.Graph_types.Uuid.t ; descriptor : Logseq_db_types.Asset_descriptor.t - ; availability : Logseq_sync_pure_reducer.Asset_transfer.availability + ; availability : Logseq_sync_asset_transfer.Core.availability } | Asset_demand_accepted of string | Asset_backpressure of string @@ -134,38 +100,38 @@ type asset_notice = ; status : Asset_upload.status } +type lifecycle_intent = Sync_intent of Logseq_sync_domain.Core.event | Foreground_intent of bool | Stop_intent + type output = - | Asset_notice of Logseq_sync_pure_reducer.Core.graph_scope * asset_notice + | Lifecycle_intent of lifecycle_intent + | Asset_notice of Logseq_sync_domain.Core.graph_scope * asset_notice | Reply of request_id * Worker_pure_read.answer | Graph_push of Worker_pure_data.changefeed_signal - | Sync_output of Logseq_sync_pure_reducer.Core.output + | Sync_output of Logseq_sync_domain.Core.output | Graph_state_changed of graph_state - | Scoped_graph_state_changed of Logseq_sync_pure_reducer.Core.graph_scope * graph_state - | Scoped_graph_push of Logseq_sync_pure_reducer.Core.graph_scope * Worker_pure_data.changefeed_signal + | Scoped_graph_state_changed of Logseq_sync_domain.Core.graph_scope * graph_state + | Scoped_graph_push of Logseq_sync_domain.Core.graph_scope * Worker_pure_data.changefeed_signal | Diagnostic of string (** {2 Upload recovery and instructions} *) type upload_recovery_ticket = private - { scope : Logseq_sync_pure_reducer.Core.graph_scope + { scope : Logseq_sync_domain.Core.graph_scope ; after : Logseq_db_types.Graph_types.Uuid.t option ; limit : int ; serial : int } type instruction = - | Sync_io_fence_reached of int64 * Logseq_sync_pure_reducer.Core.effect_scope * (unit,string) result | Asset_cleanup_barrier_reached of int64 - | Deliver_protected_result of Logseq_sync_pure_reducer.Core.protected_output - | Deliver_asset_result of Logseq_sync_pure_reducer.Core.asset_output + | Deliver_asset_result of Logseq_sync_domain.Core.asset_output | Read_uploads of upload_recovery_ticket - | Run_upload of Logseq_sync_pure_reducer.Core.asset_context * Asset_upload.instruction + | Run_upload of Logseq_sync_domain.Core.asset_context * Asset_upload.instruction | Run_asset of - Logseq_sync_pure_reducer.Core.asset_context - * Logseq_sync_pure_reducer.Asset_transfer.instruction - | Close_asset_scope of Logseq_sync_pure_reducer.Core.graph_scope + Logseq_sync_domain.Core.asset_context + * Logseq_sync_asset_transfer.Core.instruction + | Close_asset_scope of Logseq_sync_domain.Core.graph_scope | Run_worker of runner_effect - | Run_sync of Logseq_sync_pure_reducer.Core.runner_effect | Publish of output val instruction_diagnostic : instruction -> string @@ -174,7 +140,7 @@ val equal_instructions : instruction list -> instruction list -> bool type view = { graph : graph_state - ; sync : Logseq_sync_pure_reducer.Core.state + ; sync : Logseq_sync_domain.Core.state ; pending_requests : int ; pending_effects : int ; shutdown : bool @@ -189,6 +155,7 @@ val initial : config -> (state, create_error) result val view : state -> view type event = + | Lifecycle_snapshot of Logseq_sync_domain.Core.state * Logseq_sync_domain.Core.coordination_state * (Logseq_sync_domain.Core.graph_scope * database_opened) list | Asset_cleanup_barrier of int64 | Upload_requested of { graph_generation : int @@ -200,25 +167,25 @@ type event = | Upload_completed of Asset_upload.ticket * Asset_upload.completion | Asset_requested of { graph_generation : int - ; event : Logseq_sync_pure_reducer.Asset_transfer.event + ; event : Logseq_sync_asset_transfer.Core.event } | Asset_completed of - Logseq_sync_pure_reducer.Core.graph_scope - * Logseq_sync_pure_reducer.Asset_transfer.event + Logseq_sync_domain.Core.graph_scope + * Logseq_sync_asset_transfer.Core.event | Graph_request_for_scope of - { scope : Logseq_sync_pure_reducer.Core.graph_scope + { scope : Logseq_sync_domain.Core.graph_scope ; id : request_id ; read : Worker_pure_read.packed_query } - | Projection_push_for_scope of Logseq_sync_pure_reducer.Core.graph_scope * Worker_pure_data.changefeed_signal - | Asset_requested_for_scope of Logseq_sync_pure_reducer.Core.graph_scope * Logseq_sync_pure_reducer.Asset_transfer.event - | Upload_requested_for_scope of Logseq_sync_pure_reducer.Core.graph_scope * Logseq_db_types.Graph_types.Uuid.t * Asset_upload.event + | Projection_push_for_scope of Logseq_sync_domain.Core.graph_scope * Worker_pure_data.changefeed_signal + | Asset_requested_for_scope of Logseq_sync_domain.Core.graph_scope * Logseq_sync_asset_transfer.Core.event + | Upload_requested_for_scope of Logseq_sync_domain.Core.graph_scope * Logseq_db_types.Graph_types.Uuid.t * Asset_upload.event | Start | Graph_request of { id : request_id ; read : Worker_pure_read.packed_query } - | Sync_event of Logseq_sync_pure_reducer.Core.event + | Sync_event of Logseq_sync_domain.Core.event | Projection_push of Worker_pure_data.changefeed_signal | Set_foreground of bool | Runner_completed of runner_completion @@ -236,26 +203,29 @@ val complete_execute -> (Worker_pure_read.answer, effect_error) result -> event option -val asset_ticket_current : state -> Logseq_sync_pure_reducer.Asset_transfer.ticket -> bool -val asset_scope_current : state -> Logseq_sync_pure_reducer.Core.graph_scope -> bool +val asset_ticket_current : state -> Logseq_sync_asset_transfer.Core.ticket -> bool +val asset_scope_current : state -> Logseq_sync_domain.Core.graph_scope -> bool val upload_ticket_current : state -> Asset_upload.ticket -> bool val upload_recovery_current : state -> upload_recovery_ticket -> bool val import_context : state -> graph_generation:int - -> Logseq_sync_pure_reducer.Core.asset_context option + -> Logseq_sync_domain.Core.asset_context option -val graph_scopes : state -> Logseq_sync_pure_reducer.Core.graph_scope list +val graph_scopes : state -> Logseq_sync_domain.Core.graph_scope list -val graph_state_for_scope : state -> Logseq_sync_pure_reducer.Core.graph_scope -> graph_state option +val graph_state_for_scope : state -> Logseq_sync_domain.Core.graph_scope -> graph_state option type coordination_state = - { sync : Logseq_sync_pure_reducer.Core.coordination_state - ; residents : (Logseq_sync_pure_reducer.Core.graph_scope * graph_state) list + { sync : Logseq_sync_domain.Core.coordination_state + ; residents : (Logseq_sync_domain.Core.graph_scope * graph_state) list } val coordination_state : state -> coordination_state -val import_context_for_scope : state -> Logseq_sync_pure_reducer.Core.graph_scope -> Logseq_sync_pure_reducer.Core.asset_context option +val install_lifecycle : state -> sync:Logseq_sync_domain.Core.state + -> coordination:Logseq_sync_domain.Core.coordination_state + -> attachments:(Logseq_sync_domain.Core.graph_scope * database_opened) list -> state +val import_context_for_scope : state -> Logseq_sync_domain.Core.graph_scope -> Logseq_sync_domain.Core.asset_context option (** {1 Lifecycle and recovery} *) @@ -287,7 +257,7 @@ module Observation_registry : sig : t -> registration:int64 -> id:Logseq_db_types.Graph_types.Uuid.t - -> scope:Logseq_sync_pure_reducer.Core.graph_scope + -> scope:Logseq_sync_domain.Core.graph_scope -> read:Worker_pure_read.packed_query -> t * decision @@ -295,14 +265,14 @@ module Observation_registry : sig : t -> registration:int64 -> id:Logseq_db_types.Graph_types.Uuid.t - -> scope:Logseq_sync_pure_reducer.Core.graph_scope + -> scope:Logseq_sync_domain.Core.graph_scope -> t * release (** Fence an unallocated logical ID even after its graph owner has disappeared. This cannot release a known live/closing resource or grant a graph fence. *) val release_unallocated : t -> registration:int64 -> id:Logseq_db_types.Graph_types.Uuid.t - -> owner:Logseq_sync_pure_reducer.Core.timeline_scope -> t * release + -> owner:Logseq_sync_domain.Core.timeline_scope -> t * release (** External completion after physical listener/read cleanup; only then is its active-resource capacity reusable. *) @@ -310,7 +280,7 @@ module Observation_registry : sig : t -> registration:int64 -> id:Logseq_db_types.Graph_types.Uuid.t - -> scope:Logseq_sync_pure_reducer.Core.graph_scope + -> scope:Logseq_sync_domain.Core.graph_scope -> t val advance @@ -330,7 +300,7 @@ module Observation_registry : sig : t -> registration:int64 -> id:Logseq_db_types.Graph_types.Uuid.t - -> Logseq_sync_pure_reducer.Core.graph_scope option + -> Logseq_sync_domain.Core.graph_scope option val epoch : t @@ -348,7 +318,7 @@ module Observation_registry : sig and physical settlement. Already absent is a non-allocating no-op; it grants no release or host fence. Failures retain the same finalization obligation. *) val finalize : t -> registration:int64 -> id:Logseq_db_types.Graph_types.Uuid.t - -> owner:Logseq_sync_pure_reducer.Core.timeline_scope -> t * (unit,string) result + -> owner:Logseq_sync_domain.Core.timeline_scope -> t * (unit,string) result val active : t -> int val retained : t -> int @@ -416,7 +386,7 @@ end failure retains that obligation. Attempt tokens prevent an old ACK from settling a retry. No clock, callback, transport or filesystem access occurs here. *) module Orphan_cleanup : sig - type resource = { scope : Logseq_sync_pure_reducer.Core.graph_scope; handle : string } + type resource = { scope : Logseq_sync_domain.Core.graph_scope; handle : string } type t type work val empty : t @@ -435,11 +405,11 @@ end Physical settlement remains separate from command admission. *) module Demand_registry : sig type t - type owner={registration:int64;scope:Logseq_sync_pure_reducer.Core.graph_scope;consumer:string} + type owner={registration:int64;scope:Logseq_sync_domain.Core.graph_scope;consumer:string} type reservation=Start|Existing|Retired|Mismatch|Full|Invalid type release=Release|Already_settled|Unknown|Mismatch|Full val empty:t - val reserve:t -> owner -> priority:Logseq_sync_pure_reducer.Asset_transfer.priority + val reserve:t -> owner -> priority:Logseq_sync_asset_transfer.Core.priority -> assets:Logseq_db_types.Asset_descriptor.t list -> t * reservation val admitted:t -> owner -> bool val retire:t -> owner -> t * release @@ -466,15 +436,15 @@ module Writer_fence : sig type fence type error = Fenced | Full val empty : t - val admit : t -> Logseq_sync_pure_reducer.Core.graph_scope -> t * (operation,error) result - val admit_cleanup : t -> Logseq_sync_pure_reducer.Core.graph_scope -> t * (operation,error) result + val admit : t -> Logseq_sync_domain.Core.graph_scope -> t * (operation,error) result + val admit_cleanup : t -> Logseq_sync_domain.Core.graph_scope -> t * (operation,error) result val finish : t -> operation -> t - val freeze : t -> Logseq_sync_pure_reducer.Core.graph_scope -> t * (fence,error) result + val freeze : t -> Logseq_sync_domain.Core.graph_scope -> t * (fence,error) result val drained : t -> fence -> bool - val fenced : t -> Logseq_sync_pure_reducer.Core.graph_scope -> bool - val pending : t -> Logseq_sync_pure_reducer.Core.graph_scope -> int + val fenced : t -> Logseq_sync_domain.Core.graph_scope -> bool + val pending : t -> Logseq_sync_domain.Core.graph_scope -> int val operation_id : operation -> int64 - val same_incarnation : Logseq_sync_pure_reducer.Core.graph_scope -> Logseq_sync_pure_reducer.Core.graph_scope -> bool + val same_incarnation : Logseq_sync_domain.Core.graph_scope -> Logseq_sync_domain.Core.graph_scope -> bool end (** Pure unregister roster. The first authoritative account stays immutable @@ -482,14 +452,14 @@ end that actually settled the returned scopes and optional old account IO. *) module Registration_retirement : sig type t - type plan={owner:Logseq_sync_pure_reducer.Core.account_scope option; - scopes:Logseq_sync_pure_reducer.Core.graph_scope Rrbvec.t;join_account:bool;blocked:bool} + type plan={owner:Logseq_sync_domain.Core.account_scope option; + scopes:Logseq_sync_domain.Core.graph_scope Rrbvec.t;join_account:bool;blocked:bool} val empty:t - val bind:t -> registration:int64 -> account:Logseq_sync_pure_reducer.Core.account_scope option -> (t,string)result - val observe:t -> registration:int64 option -> account:Logseq_sync_pure_reducer.Core.account_scope option -> t - val begin_release:t -> registration:int64 -> active_account:Logseq_sync_pure_reducer.Core.account_scope option - -> active_scopes:Logseq_sync_pure_reducer.Core.graph_scope Rrbvec.t - -> physical_scopes:Logseq_sync_pure_reducer.Core.graph_scope Rrbvec.t -> t * plan + val bind:t -> registration:int64 -> account:Logseq_sync_domain.Core.account_scope option -> (t,string)result + val observe:t -> registration:int64 option -> account:Logseq_sync_domain.Core.account_scope option -> t + val begin_release:t -> registration:int64 -> active_account:Logseq_sync_domain.Core.account_scope option + -> active_scopes:Logseq_sync_domain.Core.graph_scope Rrbvec.t + -> physical_scopes:Logseq_sync_domain.Core.graph_scope Rrbvec.t -> t * plan val released:t -> registration:int64 -> t end @@ -503,12 +473,12 @@ module Host_handoff : sig type change=Close_graph of Logseq_db_types.Graph_types.Uuid.t |Remove_graph of Logseq_db_types.Graph_types.Uuid.t * mode |Replace_account|Stop_connection - type facts={authenticated:bool;version:int64;registration:int64 option;account:Logseq_sync_pure_reducer.Core.account_scope option; - scopes:Logseq_sync_pure_reducer.Core.graph_scope Rrbvec.t;catalog:Logseq_db_types.Graph_types.Uuid.t Rrbvec.t} + type facts={authenticated:bool;version:int64;registration:int64 option;account:Logseq_sync_domain.Core.account_scope option; + scopes:Logseq_sync_domain.Core.graph_scope Rrbvec.t;catalog:Logseq_db_types.Graph_types.Uuid.t Rrbvec.t} type request={version:int64;id:Logseq_db_types.Graph_types.Uuid.t;registration:int64; - owner:Logseq_sync_pure_reducer.Core.account_scope;change:change; - expected:Logseq_sync_pure_reducer.Core.graph_scope Rrbvec.t; - captured:Logseq_sync_pure_reducer.Core.graph_scope Rrbvec.t} + owner:Logseq_sync_domain.Core.account_scope;change:change; + expected:Logseq_sync_domain.Core.graph_scope Rrbvec.t; + captured:Logseq_sync_domain.Core.graph_scope Rrbvec.t} type t type attempt type receipt @@ -516,8 +486,8 @@ module Host_handoff : sig type phase=Preparing|Safe of receipt|Consumed of receipt|Failed of error type admission=Start of attempt|Existing of phase|Refused of error val empty:lifetime:Logseq_db_types.Graph_types.Uuid.t -> t - val capture:owner:Logseq_sync_pure_reducer.Core.account_scope -> change:change - -> Logseq_sync_pure_reducer.Core.graph_scope Rrbvec.t -> Logseq_sync_pure_reducer.Core.graph_scope Rrbvec.t + val capture:owner:Logseq_sync_domain.Core.account_scope -> change:change + -> Logseq_sync_domain.Core.graph_scope Rrbvec.t -> Logseq_sync_domain.Core.graph_scope Rrbvec.t val prepare:t -> facts -> request -> t * admission val observe:t -> facts -> t val settled:t -> attempt -> copy_generation:Logseq_overlay_db.Types.mirror_generation option -> t @@ -542,4 +512,30 @@ end (** {1 Observation and helpers} *) val host_fence_blocks : state -> Logseq_db_types.Graph_types.Uuid.t -> bool -val host_fence_active : state -> Logseq_sync_pure_reducer.Core.host_fence option +val host_fence_active : state -> Logseq_sync_domain.Core.host_fence option + +module Lifecycle : sig + type scope = Logseq_sync_domain.Core.graph_scope + type snapshot = { revision : int64; scopes : scope list; selected : scope option } + type t + type event = + | Projection of snapshot + | Attach of scope * database_opened + | Retire of scope + | Read of { scope : scope; id : request_id; read : Worker_pure_read.packed_query } + | Push of scope * Worker_pure_data.changefeed_signal + | Completed of runner_completion + type instruction = + | Worker of runner_effect + | Publish of output + | Attached of scope + | Retired of scope + type transition = { next : t; effects : instruction list } + val empty : t + val snapshot : t -> snapshot + + (** Attach borrows the service database. Retired acknowledges physical read drain; + only the external owner closes the database after this acknowledgement. + Snapshot revisions and scope generations must increase across replacement. *) + val step : t -> event -> transition +end diff --git a/logseq_db_worker/spec/pure_reducer/dune b/logseq_db_worker/spec/pure_reducer/dune index 52cd1f58..d19f478d 100644 --- a/logseq_db_worker/spec/pure_reducer/dune +++ b/logseq_db_worker/spec/pure_reducer/dune @@ -11,4 +11,4 @@ logseq_db_worker.contract logseq_db_types logseq_overlay_db - logseq_sync.pure_reducer)) + logseq_sync.domain logseq_sync.asset_transfer)) diff --git a/logseq_db_worker/test/dune b/logseq_db_worker/test/dune index 058cdd78..85f37d84 100644 --- a/logseq_db_worker/test/dune +++ b/logseq_db_worker/test/dune @@ -32,7 +32,8 @@ logseq_db_worker.contract logseq_db_worker.pure_reducer logseq_overlay_db - logseq_sync.pure_reducer + logseq_sync.protocol + logseq_sync.domain logseq_sync.asset_transfer uri yojson rrbvec @@ -49,13 +50,13 @@ logseq_db_worker logseq_db_worker.lui logseq_db_worker_test_support - logseq_sync.effect_runner + logseq_sync.native_crypto logseq_sync.assets digestif eio logseq_db_storage logseq_db_types logseq_db_worker.pure_reducer - logseq_sync.pure_reducer + logseq_sync.domain logseq_sync.asset_transfer rrbvec sqlite3 threads @@ -66,10 +67,9 @@ (test (name test_effect_runner) (modules test_effect_runner) - (libraries datascript-ocaml-native logseq_db_worker.typed_data logseq_db_worker.typed_read logseq_db_worker.identity + (libraries datascript_ocaml.types sync_dsl execute logseq_sync.worker_owner logseq_sync.crypto datascript-ocaml-native logseq_db_worker.typed_data logseq_db_worker.typed_read logseq_db_worker.identity alcotest datascript_ocaml - datascript_ocaml.types eio eio_main eio.unix @@ -82,8 +82,9 @@ logseq_db_worker.pure_reducer logseq_db_worker_test_support logseq_overlay_db_test_support - logseq_sync.effect_runner - logseq_sync.pure_reducer + logseq_sync.native_crypto logseq_sync.assets + logseq_sync.protocol + logseq_sync.domain logseq_sync.asset_transfer rrbvec unix uri @@ -127,7 +128,8 @@ logseq_db_types logseq_db_worker logseq_db_worker.lui - logseq_sync.pure_reducer + logseq_sync.protocol + logseq_sync.domain logseq_sync.asset_transfer managed_sync_e2e_support mirage-crypto-rng mirage-crypto-rng.unix @@ -154,10 +156,10 @@ alcotest logseq_db_worker.pure_reducer logseq_db_types - logseq_sync.pure_reducer + logseq_sync.domain logseq_sync.asset_transfer uri)) (test (name test_asset_protocol) (modules test_asset_protocol) - (libraries alcotest logseq_db_worker logseq_db_worker.contract logseq_db_worker.typed_read logseq_db_worker.typed_data logseq_db_worker.identity logseq_db_types logseq_sync.pure_reducer uri yojson)) + (libraries alcotest logseq_db_worker logseq_db_worker.contract logseq_db_worker.typed_read logseq_db_worker.typed_data logseq_db_worker.identity logseq_db_types logseq_sync.domain logseq_sync.asset_transfer uri yojson)) diff --git a/logseq_db_worker/test/managed_sync_protocol_probe.ml b/logseq_db_worker/test/managed_sync_protocol_probe.ml index 9fd037b2..1be5dbec 100644 --- a/logseq_db_worker/test/managed_sync_protocol_probe.ml +++ b/logseq_db_worker/test/managed_sync_protocol_probe.ml @@ -1,4 +1,4 @@ -module Protocol = Logseq_sync_pure_reducer.Sync_protocol +module Protocol = Logseq_sync_protocol.Sync_protocol type t = { descriptor : Httpun_ws.Wsd.t diff --git a/logseq_db_worker/test/managed_sync_protocol_probe.mli b/logseq_db_worker/test/managed_sync_protocol_probe.mli index 2896b8f2..654b7679 100644 --- a/logseq_db_worker/test/managed_sync_protocol_probe.mli +++ b/logseq_db_worker/test/managed_sync_protocol_probe.mli @@ -1,4 +1,4 @@ -module Protocol = Logseq_sync_pure_reducer.Sync_protocol +module Protocol = Logseq_sync_protocol.Sync_protocol type t diff --git a/logseq_db_worker/test/test_asset_protocol.ml b/logseq_db_worker/test/test_asset_protocol.ml index c00acf44..45f69d53 100644 --- a/logseq_db_worker/test/test_asset_protocol.ml +++ b/logseq_db_worker/test/test_asset_protocol.ml @@ -1,7 +1,7 @@ module A = Logseq_db_types.Asset_descriptor module M = Logseq_db_worker_contract.Worker_pure_mutation module Read = Worker_pure_read -module Sync = Logseq_sync_pure_reducer.Core +module Sync = Logseq_sync_domain.Core module Id = Journal_worker_ids let id="78000000-0000-4000-8000-000000000001" diff --git a/logseq_db_worker/test/test_asset_upload.ml b/logseq_db_worker/test/test_asset_upload.ml index e7d5e011..331f89b0 100644 --- a/logseq_db_worker/test/test_asset_upload.ml +++ b/logseq_db_worker/test/test_asset_upload.ml @@ -1,7 +1,7 @@ module U = Logseq_db_worker_pure_reducer.Asset_upload module I = Logseq_db_types.Asset_upload_intent module A = Logseq_db_types.Asset_descriptor -module C = Logseq_sync_pure_reducer.Core +module C = Logseq_sync_domain.Core let get = function | Ok value -> value diff --git a/logseq_db_worker/test/test_effect_runner.ml b/logseq_db_worker/test/test_effect_runner.ml index ec91a41a..56d5aeda 100644 --- a/logseq_db_worker/test/test_effect_runner.ml +++ b/logseq_db_worker/test/test_effect_runner.ml @@ -1,8 +1,18 @@ module Core = Logseq_db_worker_pure_reducer.Core module Runner = Logseq_db_worker_effect_runner.Effect_runner -module Sync = Logseq_sync_pure_reducer.Core +module Sync = Logseq_sync_domain.Core module T = Logseq_db_worker_test_support.Test_support +let services ?(post = fun _ -> ()) () : Runner.services = + { post + ; owner = (fun ~scope:_ -> Error "No graph owner installed") + ; asset = (fun _ -> Error Sync.Asset_cancelled) + ; await_io = (fun ~scope:_ -> Ok ()) + ; shutdown = (fun () -> ()) + ; retire_account = (fun ~account:_ -> Ok ()) + } +;; + let account ?(generation = 1) user_id : Sync.account_scope = { managed_sync_origin = Uri.of_string "https://api.logseq.io" ; user_id @@ -12,123 +22,6 @@ let account ?(generation = 1) user_id : Sync.account_scope = } ;; -module Sync_runner = Logseq_sync_effect_runner.Effect_runner - -let asset_io_runner ?(pending_budget_bytes=16L) ?(fork=(fun ~sw:_ task->task())) ?on_asset_diagnostic ?(token_failure = "test must not use network") ~env ~sw ~support ~post () = - let module R = Sync_runner in - let dependencies = - R.dependencies - ~runtime: - (R.runtime ~fork ~sleep:(fun _ -> ()) |> Result.get_ok) - ~transport: - (R.transport - ~tls_authenticator:(R.system_tls_authenticator () |> Result.get_ok) - ~network:(Eio.Stdenv.net env) - ~clock:(Eio.Stdenv.clock env) - ~websocket_liveness:R.Disabled - |> Result.get_ok) - ~local_store: - (R.local_store - ~asset_cache_budget_bytes:16L - ~asset_maximum_file_bytes:16 - ~asset_pending_budget_bytes:pending_budget_bytes - ~application_support_directory:support - () - |> Result.get_ok) - ~artifact_store: - (R.artifact_store ~staging_directory:(Filename.concat support "snapshot-staging") - |> Result.get_ok) - ~secrets: - (R.secrets - ~unlock_private_key: - (fun - ~managed_sync_origin:_ ~user_id:_ ~password:_ ~private_key_package:_ -> - Error "unused") - ~unlock_graph_key: - (fun - ~managed_sync_origin:_ ~user_id:_ ~encrypted_graph_key:_ -> Error "unused") - ~load_wrapped_graph_key:(fun ~managed_sync_origin:_ ~user_id:_ ~graph_id:_ -> - Error (R.Wrapped_graph_key_unavailable "unused")) - ~verify_and_save_wrapped_graph_key: - (fun - ~managed_sync_origin:_ ~user_id:_ ~graph_id:_ ~encrypted_graph_key:_ -> - Error "unused") - ~delete_account_secrets:(fun ~managed_sync_origin:_ ~user_id:_ -> Ok ()) - |> Result.get_ok) - ~crypto: - (R.crypto - ~encrypt_aes_gcm:(fun ~key:_ ~plaintext:_ -> Error "unused") - ~decrypt_aes_gcm:(fun ~key:_ ~iv:_ ~ciphertext:_ -> Error "unused") - |> Result.get_ok) - ~id_token_provider: - (R.id_token_provider - ~acquire:(fun _ -> Error token_failure) - ~invalidate:(fun _ ~token:_ -> ())) - |> Result.get_ok - in - R.create ?on_asset_diagnostic ~sw dependencies ~post |> Result.get_ok -;; - -let failure_details outputs = - let module D = Logseq_db_types.Failure_diagnostic in - List.filter_map - (function - | Core.Diagnostic signal when String.starts_with ~prefix:D.signal_prefix signal -> - D.decode (String.sub signal (String.length D.signal_prefix) - (String.length signal - String.length D.signal_prefix)) - | _ -> None) - outputs -;; - -let has_failure_reason outputs operation reason = - let module D = Logseq_db_types.Failure_diagnostic in - List.exists (fun detail -> D.operation detail = operation && D.reason detail = reason) - (failure_details outputs) -;; - -let test_unlock_runner_logging () = - let path = Filename.temp_file "journal-unlock-runner-log-" "" in - let saved = Unix.dup Unix.stderr in - let output = Unix.openfile path [Unix.O_WRONLY; Unix.O_TRUNC] 0o600 in - let previous = Sys.getenv_opt "JOURNAL_UNLOCK_DIAGNOSTICS" in - Unix.putenv "JOURNAL_UNLOCK_DIAGNOSTICS" "1"; - Unix.dup2 output Unix.stderr; - let has text value = - let n = String.length value in - let rec loop i = i + n <= String.length text - && (String.sub text i n = value || loop (i + 1)) in loop 0 - in - Fun.protect ~finally:(fun () -> - flush stderr; Unix.dup2 saved Unix.stderr; Unix.close saved; Unix.close output; - Unix.putenv "JOURNAL_UNLOCK_DIAGNOSTICS" (Option.value previous ~default:""); Sys.remove path) - (fun () -> - Eio_main.run (fun env -> Eio.Switch.run (fun sw -> T.with_managed (fun fixture -> - let completions = ref [] in - let runner = asset_io_runner ~env ~sw - ~support:fixture.config.application_support_directory - ~post:(fun event -> completions := event :: !completions) () in - let limits = Sync.limits ~maximum_response_bytes:1024 ~maximum_artifact_bytes:1024 - ~submission_batch_size:1 |> Result.get_ok in - let config = Sync.config ~managed_sync_origin:(Uri.of_string "https://example.invalid") - ~limits |> Result.get_ok in - let state = Sync.initial config |> Result.get_ok in - let state = (Sync.step state (Sync.Sign_in_requested {account_generation=0; presentation_generation=0})).next in - let transition = Sync.step state (Sync.Account_authenticated {user_id=Some "LOG_SECRET_USER"}) in - let request = List.find_map (function - | Sync.Run (Sync.Request (_, Sync.Fetch_catalog _) as request) -> Some request - | _ -> None) transition.effects |> Option.get in - Sync_runner.submit runner request; - Alcotest.(check bool) "runner still posts typed completion" true (!completions <> []); - Sync_runner.shutdown runner))); - flush stderr; - let logs = In_channel.with_open_bin path In_channel.input_all in - List.iter (fun value -> Alcotest.(check bool) value true (has logs value)) - [ "source=runner"; "request=fetch-catalog"; "phase=begin"; "phase=end" - ; "result=effect-failed"; "failure_category=token"; "elapsed_ms="; "op=" ]; - List.iter (fun value -> Alcotest.(check bool) "sensitive runner value excluded" false (has logs value)) - [ "LOG_SECRET_USER"; "test must not use network"; "example.invalid" ]) -;; - let far_future_token = "header.eyJleHAiOjIwMDAwMDAwMDB9.signature" let two_hour_token = "header.eyJleHAiOjE3MDAwMDcyMDB9.signature" let one_hour_token = "header.eyJleHAiOjE3MDAwMDM2MDB9.signature" @@ -454,18 +347,20 @@ let test_sync_and_publish_instructions_are_not_reduced_recursively () = ~fork:(fun ~sw:_ task -> task ()) |> Result.get_ok in - let sync_runner = - Runner.sync_runner ~retire_account:(fun ~account:_->Alcotest.fail "Unexpected account retirement in fixture") - ~submit:(fun runner_effect -> submitted := runner_effect :: !submitted) - ~shutdown:(fun () -> ()) - () + let services : Runner.services = + {post=(fun intent -> submitted := intent :: !submitted); + owner=(fun ~scope:_ -> Error "No graph owner installed"); + asset=(fun _ -> Alcotest.fail "Unexpected asset request"); + await_io=(fun ~scope:_ -> Alcotest.fail "Unexpected I/O join"); + retire_account=(fun ~account:_ -> Alcotest.fail "Unexpected account retirement in fixture"); + shutdown=(fun () -> ())} in let dependencies = Runner.dependencies ~runtime ~config:fixture.config ~overlay:fixture.overlay - ~sync_runner + ~services ~publish:(fun output -> published := output :: !published) |> Result.get_ok in @@ -480,41 +375,74 @@ let test_sync_and_publish_instructions_are_not_reduced_recursively () = ~post:(fun event -> posted := event :: !posted) |> Result.get_ok in - let limits = - Sync.limits - ~maximum_response_bytes:1024 - ~maximum_artifact_bytes:1024 - ~submission_batch_size:1 - |> Result.get_ok - in - let sync = - Sync.config ~managed_sync_origin:(Uri.of_string "https://api.logseq.io") ~limits - |> Result.get_ok - |> Sync.initial - |> Result.get_ok - in - let transition = Sync.step sync (Restore_local_account { user_id = "user-1" }) in - List.iter - (function - | Sync.Run runner_effect -> Runner.submit runner (Core.Run_sync runner_effect) - | Sync.Publish output -> - Runner.submit runner (Core.Publish (Core.Sync_output output)) - | Sync.Delegate _ -> ()) - transition.effects; - Alcotest.check Alcotest.int "sync runner receives run" 1 (List.length !submitted); + let intent = Core.Sync_intent (Sync.Restore_local_account {user_id="user-1"}) in + Runner.submit runner (Core.Publish (Core.Lifecycle_intent intent)); + Runner.submit runner (Core.Publish (Core.Diagnostic "fixture publication")); + Alcotest.check Alcotest.int "lifecycle receives post" 1 (List.length !submitted); + Alcotest.(check bool) "original intent retained" true (!submitted=[intent]); Alcotest.check Alcotest.int "host receives publish" 1 (List.length !published); Alcotest.check Alcotest.int "runner never calls reducer" 0 (List.length !posted); Runner.shutdown runner))) ;; +let test_direct_owner_io_failure_has_no_event_bridge () = + T.with_managed (fun fixture -> Eio_main.run (fun _ -> Eio.Switch.run (fun sw -> + let posted = ref [] in + let calls = ref 0 in + let services : Runner.services = + {post=(fun _ -> Alcotest.fail "Unexpected lifecycle intent"); + owner=(fun ~scope:_ -> Error "No graph owner installed"); + asset=(fun _ -> Error Sync.Asset_cancelled); + await_io=(fun ~scope:_ -> incr calls; Error "Original owner cleanup failed"); + retire_account=(fun ~account:_ -> Ok ()); shutdown=(fun () -> ())} in + let dependencies = Runner.dependencies + ~runtime:(Runner.runtime ~fork:(fun ~sw task -> Eio.Fiber.fork ~sw task) + ~sleep:(fun _ -> ()) |> Result.get_ok) + ~config:fixture.config ~overlay:fixture.overlay ~services + ~publish:(fun _ -> ()) |> Result.get_ok in + let runner = Runner.create ~sw dependencies ~post:(fun event -> posted := event :: !posted) + ~recovery_current:(fun _ -> false) ~upload_current:(fun _ -> false) + ~asset_current:(fun _ -> false) |> Result.get_ok in + let account : Sync.account_scope = + {managed_sync_origin=Uri.of_string "https://api.logseq.io";user_id="user-1";account_generation=0; + presentation_generation=0;lifecycle_generation=0L} in + Alcotest.(check (result unit string)) "physical error preserved" + (Error "Original owner cleanup failed") + (Runner.await_sync_io runner ~scope:(Sync.effect_scope_of_account account)); + Alcotest.(check int) "direct callback called once" 1 !calls; + Alcotest.(check int) "no reducer fence event" 0 (List.length !posted); + Runner.shutdown runner; + Alcotest.(check bool) "stopped worker cannot borrow upload store" true + (Result.is_error (Runner.sync_upload_store runner))))) +;; + let asset_sync_config () = let limits=Sync.limits ~maximum_response_bytes:1024 ~maximum_artifact_bytes:1024 ~submission_batch_size:1 |> Result.get_ok in Sync.config ~managed_sync_origin:(Uri.of_string "https://api.logseq.io") ~limits |> Result.get_ok ;; (* Scheduling failures belong to the effect runner, not the pure reducer. *) +let lifecycle_projection config (scope : Sync.graph_scope) = + let state = Core.initial (Core.config ~worker:config ~sync:(asset_sync_config ())) |> Result.get_ok in + let initial = (Core.view state).sync in + let graph : Sync.graph = {graph_id=scope.graph_id;name="Direct owner fixture"; + schema={major=65;minor=33;exact=true};encrypted=false} in + let sync : Sync.state = {initial with snapshot={initial.snapshot with + selected_graph=Some graph.graph_id;catalog=[graph];resident_graphs=[graph.graph_id]; + startup={initial.snapshot.startup with authenticated=true;local_access=Sync.Local_available; + account_generation=scope.account.account_generation;graph_generation=scope.graph_generation; + presentation_generation=scope.account.presentation_generation}}} in + let coordination : Sync.coordination_state = {account=Some scope.account; + selected=Some {startup={account_generation=scope.account.account_generation; + presentation_generation=scope.account.presentation_generation};graph_id=scope.graph_id; + graph_generation=scope.graph_generation}; + residents=[scope,sync];desired_residents=[scope.graph_id];deletion=None} in + sync, coordination +;; + let test_worker_schedule_failure_completes ?(failure=Unix.Unix_error (Unix.EACCES, "fork", "PRIVATE_SYNTHETIC_PAYLOAD")) ?(expected_source="io-permission") ?(expected_reason="resourceFailed") () = + let module Sync = Logseq_sync_domain.Core in T.with_managed (fun fixture -> Eio_main.run (fun _ -> Eio.Switch.run (fun sw -> let posted = ref [] in let dependencies = Runner.dependencies @@ -522,35 +450,53 @@ let test_worker_schedule_failure_completes raise failure) ~sleep:(fun _ -> ()) |> Result.get_ok) ~config:fixture.config ~overlay:fixture.overlay - ~sync_runner:(Runner.sync_runner ~retire_account:(fun ~account:_ -> Ok ()) - ~submit:(fun _ -> ()) ~shutdown:(fun () -> ()) ()) + ~services:(services ()) ~publish:(fun _ -> ()) |> Result.get_ok in let runner = Runner.create ~sw dependencies ~recovery_current:(fun _ -> false) ~upload_current:(fun _ -> false) ~asset_current:(fun _ -> false) ~post:(fun event -> posted := event :: !posted) |> Result.get_ok in let state = Core.initial (Core.config ~worker:fixture.config ~sync:(asset_sync_config ())) |> Result.get_ok in - let state = (Core.step state (Core.Sync_event (Sync.Sign_in_requested - {account_generation=0;presentation_generation=0}))).next in - let signed_in = Core.step state (Core.Sync_event (Sync.Account_authenticated {user_id=Some "user-1"})) in let graph : Sync.graph = {graph_id=(Logseq_db_types.Graph_types.Uuid.of_string "94000000-0000-4000-8000-000000000709" |> Result.get_ok);name="fixture"; schema={major=65;minor=33;exact=true};encrypted=false} in - let catalog = List.find_map (function - | Core.Run_sync (Sync.Request (ticket, Sync.Fetch_catalog _)) -> - Some (Core.step signed_in.next (Core.Sync_event (Sync.Runner_completed - (Sync.Completion (ticket, Ok [graph]))))) - | _ -> None) signed_in.effects |> Option.get in - let selected = Core.step catalog.next (Core.Sync_event (Sync.Graph_selected graph.graph_id)) in - let work = List.find (function Core.Run_worker _ -> true | _ -> false) selected.effects in + let scope : Sync.graph_scope = {account=account "user-1";graph_id=graph.graph_id;graph_generation=1} in + let initial = (Core.view state).sync in + let sync : Sync.state = {initial with snapshot={initial.snapshot with + selected_graph=Some graph.graph_id;catalog=[graph];resident_graphs=[graph.graph_id]; + startup={initial.snapshot.startup with authenticated=true;local_access=Sync.Local_available; + account_generation=1;graph_generation=1;presentation_generation=1}}} in + let coordination : Sync.coordination_state = {account=Some scope.account; + selected=Some {startup={account_generation=1;presentation_generation=1};graph_id=graph.graph_id;graph_generation=1}; + residents=[scope,sync];desired_residents=[graph.graph_id];deletion=None} in + let opened = Core.database_opened ~database_id:"schedule-fixture" ~graph_id:graph.graph_id in + let snapshot = Core.step state (Core.Lifecycle_snapshot (sync, coordination, [scope, opened])) in + let module Read = Worker_pure_read in + let module F = Read.For_worker in + let binding = F.binding ~serial:1L ~incarnation:(Read.worker_incarnation + ~runtime_epoch:(Journal_worker_ids.Runtime.Epoch.of_int64 1L) + ~worker_generation:(Journal_worker_ids.Worker.Generation.of_int64 1L)) + ~origin:(Uri.to_string scope.account.managed_sync_origin) ~user:scope.account.user_id in + let owner : Sync.timeline_scope = {startup={account_generation=scope.account.account_generation; + presentation_generation=scope.account.presentation_generation};graph_id=scope.graph_id; + graph_generation=scope.graph_generation} in + let limits : Worker_pure_data.capability_limits = {response_budget_bytes=fixture.config.response_budget_bytes; + outbox_max_records=4096;outbox_max_bytes=8388608;change_max_items=4096;change_max_bytes=1048576; + dispatcher_capacity=256;wire_batch_max_bytes=262144} in + let session = F.session ~binding ~graph:scope.graph_id ~account:scope.account.account_generation + ~lifecycle:scope.account.lifecycle_generation ~generation:scope.graph_generation ~scope:owner ~asset_scope:scope ~limits in + let budget = Read.read_budget session ~maximum_items:1 ~maximum_bytes:limits.response_budget_bytes |> Result.get_ok in + let requested = Core.step snapshot.next (Core.Graph_request_for_scope + {scope;id=Core.request_id_of_int64 1L;read=F.pack_query budget Read.Admission}) in + let work = List.find (function Core.Run_worker _ -> true | _ -> false) requested.effects in Runner.submit runner work; match !posted with - | [Core.Runner_completed (Core.Sync_worker_effect_completed (_, Error error))] -> + | [Core.Runner_completed (Core.Execute_request_completed (_, Error error))] -> let module E = Logseq_db_worker_contract.Error in let module D = Logseq_db_types.Failure_diagnostic in let detail = List.find_map (fun (item:E.detail) -> match item.value with | E.Detail_string source when item.name=D.detail_name -> D.decode source | _ -> None) (E.details error) |> Option.get in - Alcotest.(check string) "operation retained" "inspectMirror" (D.operation detail); + Alcotest.(check string) "operation retained" "workerOperation" (D.operation detail); Alcotest.(check string) "source cause retained" expected_source (D.source_class detail); Alcotest.(check string) "specific reason retained" expected_reason (D.reason detail); let encoded=Yojson.Safe.to_string(E.to_yojson error) in @@ -561,244 +507,6 @@ let test_worker_schedule_failure_completes | _ -> Alcotest.fail "Scheduling failure must post exactly one typed completion"))) ;; -let admitted_asset_state graph_id = - let initial=Sync.initial(asset_sync_config()) |> Result.get_ok in - let graph : Sync.graph = - { graph_id - ; name = "Upload checkpoint fixture" - ; schema = { major = 1; minor = 0; exact = true } - ; encrypted = false - } - in - let initial = (Sync.step initial (Sync.Sign_in_requested {account_generation=0; presentation_generation=0})).next in - let authenticated = - Sync.step initial (Sync.Account_authenticated { user_id = Some "user-1" }) - in - let catalog = - List.find_map - (function - | Sync.Run (Sync.Request (ticket, Sync.Fetch_catalog _)) -> - Some - (Sync.step - authenticated.next - (Sync.Runner_completed (Sync.Completion (ticket, Ok [ graph ])))) - | _ -> None) - authenticated.effects - |> Option.get - in - let selected = Sync.step catalog.next (Sync.Graph_selected graph.graph_id) in - let mirror = - List.find_map - (function - | Sync.Delegate (Sync.Inspect_mirror request) -> Some request - | _ -> None) - selected.effects - |> Option.get - in - let admitted = - Sync.step selected.next (Sync.Mirror_inspected (Sync.Mirror_available mirror)) - in - let context = Sync.asset_context admitted.next |> Option.get in - admitted.next, context -;; - -type asset_bridge_fixture = - { sw:Eio.Switch.t - ; runner : Runner.t - ; context : Sync.asset_context - ; support : string - ; gate : ?hold_callback:bool -> (Sync.asset_output -> bool) -> deliver_first:bool -> unit Eio.Promise.t - ; deliver_held : ?keep_gate:bool -> unit -> unit - ; hold_asset_barrier : unit -> unit Eio.Promise.t - ; deliver_asset_barriers : unit -> unit - ; set_asset_current : (Logseq_sync_pure_reducer.Asset_transfer.ticket -> bool) -> unit - ; on_sync_fence : (int64 * Sync.effect_scope -> unit) -> unit - ; on_asset_completed : (Logseq_sync_pure_reducer.Asset_transfer.event -> unit) -> unit - ; held_path : string option ref - ; actions : Sync.asset_action list ref - ; fixture_result : Sync.asset_action -> Sync.asset_result - ; fixture_asset : Sync.asset_action -> Sync.asset_value - } - -let with_asset_bridge ?(asynchronous=false) ?(asynchronous_io=false) ?(pending_budget_bytes=16L) run = - T.with_managed (fun fixture -> - Eio_main.run (fun env -> - Eio.Switch.run (fun sw -> - let graph_id = - Logseq_db_types.Graph_types.Uuid.of_string - "22000000-0000-4000-8000-000000000001" - |> Result.get_ok - in - let admitted, context = admitted_asset_state graph_id in - let state = ref admitted in - let worker_state=ref(Core.initial(Core.config ~worker:fixture.config ~sync:(asset_sync_config())) |> Result.get_ok) in - let on_sync_fence=ref(fun (_:int64*Sync.effect_scope)->()) in - let active_runner = ref None in - let io_runner = ref None in - let actions = ref [] in - let output_handler = ref (fun (_ : Sync.asset_output) -> false) in - let held = Queue.create () in - let held_path = ref None in - let fixture_output = ref None in - let barrier_handler=ref(fun (_:int64)->false) in - let asset_current=ref(fun (_:Logseq_sync_pure_reducer.Asset_transfer.ticket)->false) in - let asset_completed=ref(fun (_:Logseq_sync_pure_reducer.Asset_transfer.event)->()) in - let post = function - | Core.Sync_event((Sync.IO_fence_requested _|Sync.IO_fence_completed _)as event)-> - (match event with Sync.IO_fence_requested(id,scope)-> !on_sync_fence(id,scope)|_->()); - let transition=Core.step !worker_state(Core.Sync_event event) in - worker_state:=transition.next; - List.iter(Runner.submit(Option.get !active_runner))transition.effects - | Core.Sync_event event -> - (match event with - | Sync.Asset_requested request -> actions := request.action :: !actions - | _ -> ()); - let transition = Sync.step !state event in - state := transition.next; - List.iter - (function - | Sync.Run runner_effect -> - Sync_runner.submit (Option.get !io_runner) runner_effect - | Sync.Publish (Sync.Asset_finished output) -> - if String.starts_with ~prefix:"fixture:" output.request.operation - then fixture_output := Some output.result - else if not (!output_handler output) - then - Runner.submit - (Option.get !active_runner) - (Core.Deliver_asset_result output) - | Sync.Publish _ -> () - | Sync.Delegate _ -> Alcotest.fail "unexpected bridge graph effect") - transition.effects - | Core.Asset_completed(_,event)-> !asset_completed event - | Core.Asset_cleanup_barrier token->if not(!barrier_handler token) then Runner.submit(Option.get !active_runner)(Core.Asset_cleanup_barrier_reached token) - | _ -> () - in - io_runner - := Some - (asset_io_runner ~pending_budget_bytes - ~fork:(fun ~sw task->if asynchronous_io then Eio.Fiber.fork ~sw task else task()) - ~on_asset_diagnostic:(fun ~scope ~operation ~request_operation detail -> - Runner.report_asset_diagnostic (Option.get !active_runner) - ~scope ~operation ~request_operation detail) - ~env - ~sw - ~support:fixture.config.application_support_directory - ~post:(fun event -> post (Core.Sync_event event)) ()); - let sync_runner = - Runner.sync_runner ~retire_account:(Sync_runner.retire_account (Option.get !io_runner)) - ~submit:(fun runnable -> Sync_runner.submit (Option.get !io_runner) runnable) - ~shutdown:(fun () -> Sync_runner.shutdown (Option.get !io_runner)) - () - in - let dependencies = - Runner.dependencies - ~runtime: - (Runner.runtime - ~fork:(fun ~sw task -> if asynchronous then Eio.Fiber.fork ~sw task else task ()) - ~sleep:(fun seconds->if asynchronous then Eio.Time.sleep(Eio.Stdenv.clock env)seconds) - |> Result.get_ok) - ~config:fixture.config - ~overlay:fixture.overlay - ~sync_runner - ~publish:(fun _ -> ()) - |> Result.get_ok - in - let runner = - Runner.create - ~sw - dependencies - ~post - ~recovery_current:(fun _ -> false) - ~upload_current:(fun _ -> false) - ~asset_current:(fun ticket -> !asset_current ticket) - |> Result.get_ok - in - active_runner := Some runner; - let serial = ref 0 in - let fixture_result action = - incr serial; - fixture_output := None; - post - (Core.Sync_event - (Sync.Asset_requested - { scope = context.scope - ; operation = "fixture:" ^ string_of_int !serial - ; action - })); - while !fixture_output=None do Eio.Fiber.yield() done; - Option.get !fixture_output - in - let fixture_asset action=fixture_result action |> Result.get_ok in - let path_of_output (output : Sync.asset_output) = - match output.result with - | Ok (Sync.Asset_staged { file; _ }) -> - (match fixture_asset (Sync.Retain_staged_file file) with - | Sync.Asset_retained (Some (lease, path)) -> - ignore (fixture_asset (Sync.Release_asset_file lease)); - Some path - | _ -> Alcotest.fail "gated staged resource unavailable") - | Ok (Sync.Asset_retained (Some (_, path))) -> Some path - | _ -> None - in - let gate ?(hold_callback=false) predicate ~deliver_first = - let reached, resolve = Eio.Promise.create () in - let callback_done, callback_resolve = Eio.Promise.create () in - (output_handler - := fun output -> - if not (predicate output) - then false - else ( - held_path := path_of_output output; - Queue.add (output, deliver_first,callback_resolve) held; - if deliver_first - then Runner.submit runner (Core.Deliver_asset_result output); - ignore(Eio.Promise.try_resolve resolve ():bool); - if deliver_first || hold_callback then Eio.Promise.await callback_done; - true)); - reached - in - let deliver_held ?(keep_gate=false) () = - if not keep_gate then (output_handler := fun _ -> false); - Queue.iter - (fun (output, delivered,callback_resolve) -> - if not delivered - then Runner.submit runner (Core.Deliver_asset_result output); - ignore(Eio.Promise.try_resolve callback_resolve():bool)) - held; - Queue.clear held - in - let held_barriers=Queue.create() in - let hold_asset_barrier ()= - let reached,resolve=Eio.Promise.create() in - barrier_handler:=(fun token->Queue.add token held_barriers; - ignore(Eio.Promise.try_resolve resolve():bool);true); - reached in - let deliver_asset_barriers ()= - barrier_handler:=(fun _->false); - Queue.iter(fun token->Runner.submit runner(Core.Asset_cleanup_barrier_reached token))held_barriers; - Queue.clear held_barriers in - Fun.protect - ~finally:(fun () -> Runner.shutdown runner) - (fun () -> - run - { sw;runner - ; context - ; support = fixture.config.application_support_directory - ; gate - ; deliver_held - ; hold_asset_barrier - ; deliver_asset_barriers - ; set_asset_current=(fun callback->asset_current:=callback) - ; on_sync_fence=(fun callback->on_sync_fence:=callback) - ; on_asset_completed=(fun callback->asset_completed:=callback) - ; held_path - ; actions - ; fixture_result - ; fixture_asset - })))) -;; - let picker_import support ordinal : Logseq_db_types.Asset_import.t = let uuid n = Logseq_db_types.Graph_types.Uuid.of_string @@ -823,153 +531,227 @@ let picker_import support ordinal : Logseq_db_types.Asset_import.t = (* Local staging publication and lease ownership live in the runner, not the transfer reducer. Exercise real cache pressure and filesystem failures. *) -let test_wrapped_local_asset_fallback mode () = - with_asset_bridge ~asynchronous:true ~pending_budget_bytes:32L (fun fixture -> - let module Transfer=Logseq_sync_pure_reducer.Asset_transfer in - let scope=fixture.context.scope in - let source=picker_import fixture.support 400 in - let prepared=Runner.prepare_import fixture.runner ~context:fixture.context source ~current:(fun()->true) - |> Result.get_ok in - let filler=picker_import fixture.support 410 in - let remote_bytes=String.make 16 'r' in - Out_channel.with_open_bin filler.source_file(fun output->output_string output remote_bytes); - let checksum,file=match fixture.fixture_asset(Sync.Stage_asset_file - {operation=filler.operation;file_type=filler.file_type;source_file=filler.source_file}) with - | Sync.Asset_staged {checksum;file;_}->checksum,file | _->Alcotest.fail "remote cache staging failed" in - let version=Logseq_db_types.Asset_descriptor.version ~checksum ~file_type:"bin" |> Result.get_ok in - ignore(fixture.fixture_asset(Sync.Publish_staged_asset{asset=prepared.asset;version;file})); - let pinned=match fixture.fixture_asset(Sync.Check_asset_cache(prepared.asset,version)) with - | Sync.Asset_cached(Some handle)->handle | _->Alcotest.fail "remote cache did not fill budget" in - Alcotest.(check bool) "original version actually evicted" true - (fixture.fixture_asset(Sync.Check_asset_cache(prepared.asset,prepared.version))=Sync.Asset_cached None); - let descriptor=Logseq_db_types.Asset_descriptor.create ~uuid:prepared.asset - ~source:(Managed(Some version)) ~current_checksum:(Some prepared.version.checksum) - ~size:None ~dimensions:None |> Result.get_ok in - let transfer=Transfer.create(Transfer.config ~active:2 ~foreground_reserved:1 ~pending:3 ~retries:0 |> Result.get_ok) - ~scope ~online:true ~unlocked:true in - let _,work=Transfer.step transfer(Replace{consumer="wrapped-local-fallback";priority=Foreground;assets=[descriptor]}) in - let lookup=List.find_map(function Transfer.Check_cache ticket->Some ticket|_->None)work |> Option.get in - fixture.set_asset_current(fun ticket->ticket=Transfer.Lookup lookup); - let completed,resolve=Eio.Promise.create() in - fixture.on_asset_completed(function Transfer.Cache_checked(_,result)-> - ignore(Eio.Promise.try_resolve resolve result:bool)|_->()); - let publication_failure=ref None and retained_lease=ref None and gated=ref false in - let releases handle=List.fold_left(fun count->function - | Sync.Release_asset_file actual when actual=handle -> count+1 | _->count) 0 !(fixture.actions) in - let remote_checks ()=List.fold_left(fun count->function - | Sync.Check_asset_cache(asset,actual) when asset=prepared.asset && actual=version -> count+1 - | _->count) 0 !(fixture.actions) in - let before_remote=remote_checks () in - let reached=fixture.gate (fun output -> match output.Sync.request.action with - | Sync.Retain_staged_file file when file=prepared.staged_file && not !gated -> - retained_lease:=(match output.result with Ok(Sync.Asset_retained(Some(lease,_)))->Some lease|_->None); - gated:=true;true - | Sync.Publish_staged_asset _ -> - (match output.result with Error failure->publication_failure:=Some failure|_->());false - | _->false) ~deliver_first:false in - Runner.submit fixture.runner(Core.Run_asset(fixture.context,Transfer.Check_cache lookup)); - Eio.Promise.await reached; - let path=Option.get !(fixture.held_path) in - (match mode with - | `Full->() - | `Missing->Unix.unlink path - | `Checksum->Out_channel.with_open_bin path(fun output->output_string output "fake") - | `Oversize->Out_channel.with_open_bin path(fun output->output_string output (String.make 17 'x'))); - fixture.deliver_held ~keep_gate:true (); - let result=Eio.Promise.await completed in - let failure=Option.get !publication_failure in - let expected=match mode with `Full->Sync.Asset_storage_full | `Missing->Asset_missing_source - | `Checksum->Asset_checksum_mismatch | `Oversize->Asset_size_rejected in - Alcotest.(check bool) "actual publication error is wrapped with cause" true - (Sync.asset_failure_kind failure=expected && Option.is_some(Sync.asset_failure_cause failure)); - Alcotest.(check int) "outer staging lease released only on non-full branch" - (if mode=`Full then 0 else 1) (releases (Option.get !retained_lease)); - Alcotest.(check int) "only missing/checksum consult remote cache" - (if mode=`Missing || mode=`Checksum then 1 else 0) (remote_checks () - before_remote); - if mode=`Oversize then - Alcotest.(check bool) "nonfallback failure category reaches transfer unchanged" true - (result=Error(Transfer.Invalid_content "Asset size exceeds the limit")) - else ( - let handle=match result with Ok(Some handle)->handle - | _->Alcotest.fail "wrapped local publication failure broke preview/cache fallback" in - let lease,readable=match fixture.fixture_asset(Sync.Retain_asset_file handle) with - | Sync.Asset_retained(Some file)->file | _->Alcotest.fail "fallback returned an unreadable lease" in - Alcotest.(check string) "fallback owns correct readable bytes" - (if mode=`Full then "file" else remote_bytes) - (In_channel.with_open_bin readable In_channel.input_all); - ignore(fixture.fixture_asset(Sync.Release_asset_file lease)); - ignore(fixture.fixture_asset(Sync.Release_asset_file handle))); - ignore(fixture.fixture_asset(Sync.Release_asset_file pinned))) -;; +type owner_fixture = + { runner : Runner.t + ; context : Sync.asset_context + ; support : string + ; database : Logseq_overlay_db.Database.t + ; crypto : Logseq_sync_crypto.Core.t + ; identity : Sync_dsl.Core.identity + ; actions : Sync.asset_action list ref + ; fixture_result : Sync.asset_action -> Sync.asset_result + ; fixture_asset : Sync.asset_action -> Sync.asset_value + ; set_upload_current : (Logseq_db_worker_pure_reducer.Asset_upload.ticket -> bool) -> unit + ; upload_events : Logseq_db_worker_pure_reducer.Asset_upload.completion Queue.t + ; published : Core.output list ref + ; recover_uploads : unit -> (Logseq_db_types.Asset_upload_intent.t list, string) result + ; staged_path : string -> string option + ; gate : (Sync.asset_output -> bool) -> unit Eio.Promise.t + ; deliver_held : ?keep_gate:bool -> unit -> unit + } -let cancelled_stage_output deliver_first () = - with_asset_bridge (fun fixture -> - let source = picker_import fixture.support 10 in - let reached = - fixture.gate - (fun output -> - match output.Sync.request.action with - | Sync.Stage_asset_file _ -> true - | _ -> false) - ~deliver_first - in - Eio.Fiber.first +let with_direct_owner + ?(network : Execute.Core.worker_network = + {with_admission=(fun _ -> Alcotest.fail "Unexpected owner network admission")}) + ?(put = fun ~identity:_ ~asset:_ ~version:_ ~wire:_ -> Alcotest.fail "Unexpected network PUT") + run = + T.with_managed (fun fixture -> Eio_main.run (fun env -> Eio.Switch.run (fun sw -> + let module D = Sync_dsl.Core in + let module Assets = Logseq_sync_assets.Core in + let module Cache = Logseq_sync_assets.Cache in + let scope : Sync.graph_scope = + { account = account "user-1" + ; graph_id = Logseq_db_types.Graph_types.Uuid.of_string + "22000000-0000-4000-8000-000000000001" |> Result.get_ok + ; graph_generation = 1 + } in + let identity = D.identity ~origin:(Uri.to_string scope.account.managed_sync_origin) + ~account:scope.account.user_id ~graph:(Logseq_db_types.Graph_types.Uuid.to_string scope.graph_id) + ~account_generation:1L ~graph_generation:1L |> Result.get_ok in + let limits = D.limits ~maximum_items:8 ~maximum_bytes:32768 ~maximum_pending:8 + ~maximum_resources:8 ~steps_per_turn:8 ~maximum_tasks:8 ~maximum_messages:8 + ~maximum_message_bytes:32768 ~maximum_demands:8 ~maximum_active_assets:2 + ~foreground_reserve:1 ~foreground_demand_reserve:1 ~foreground_descriptor_reserve_bytes:128 + ~asset_cache_budget_bytes:16L ~asset_maximum_file_bytes:16 ~asset_pending_budget_bytes:32L + ~snapshot_maximum_artifact_bytes:4096L ~snapshot_staging_budget_bytes:4096L + ~snapshot_scratch_budget_bytes:4096L |> Result.get_ok in + let crypto = Logseq_sync_crypto.Core.create ~identity ~limits ~encrypted:false + ~backend:{crypto=Logseq_sync_native_crypto.Platform_crypto.crypto; + unlock=(fun ~identity:_ ~wrapped_graph_key:_ -> Error D.Locked)} in + let support = fixture.config.application_support_directory in + let assets = Assets.create ~root:support ~identity ~limits ~crypto + ~key:(Assets.native_key ~identity ~crypto) + ~network:{get=(fun ~identity:_ ~asset:_ ~version:_ ~maximum_wire_bytes:_ -> + Alcotest.fail "Unexpected network GET"); + put} + |> Result.get_ok in + ignore (Test_support.seed_mirror ~graph_id:scope.graph_id support); + let inspection = Logseq_overlay_db.Database.inspect_mirror + ~application_support_directory:support ~graph_id:scope.graph_id |> Result.get_ok in + let database = Logseq_overlay_db.Database.open_ ~sw fixture.overlay inspection + ~graph_name:"Direct import owner" |> Result.get_ok in + let database_owner = Execute.Core.database_owner database in + let cache = Assets.cache assets in + let actions = ref [] in + let owner_ref = ref None in + let await_owner () = match !owner_ref with + | None -> Ok () + | Some owner -> + Result.bind (Logseq_sync_worker_owner.Core.fence owner ~identity) + (fun () -> Logseq_sync_worker_owner.Core.await_io owner ~identity) + |> Result.map_error D.error_message in + let with_cause kind message = Sync.Asset_with_cause (kind, + Logseq_db_types.Failure_diagnostic.of_source ~operation:"assetPreview" + ~reason:"ownerFailed" message) in + let failure error = + let kind = match error with + | Cache.Full -> Sync.Asset_storage_full + | Checksum_mismatch -> Sync.Asset_checksum_mismatch + | Stale -> Sync.Asset_cancelled + | Invalid message | Io message -> Sync.Asset_invalid_content message in + with_cause kind (D.error_message (Assets.cache_error error)) in + let fixture_result action = + actions := action :: !actions; + match action with + | Sync.Stage_asset_file {operation;file_type;source_file} -> + Cache.stage cache ~operation ~file_type ~source_file ~pending_budget_bytes:32L + |> Result.map (fun (staged : Cache.staged) -> Sync.Asset_staged + {file=staged.file;checksum=staged.checksum;size=staged.size}) + |> Result.map_error failure + | Sync.Check_asset_cache (asset, version) -> + Cache.lookup cache ~asset ~version |> Result.map (fun handle -> Sync.Asset_cached handle) + |> Result.map_error failure + | Sync.Publish_staged_asset {asset;version;file} -> + let module Codec = Logseq_sync_native_crypto.Asset_codec in + (match Cache.staged_path cache ~file with + | None -> Error (with_cause Sync.Asset_missing_source "Staged original is missing") + | Some path when not (Sys.file_exists path) -> + Error (with_cause Sync.Asset_missing_source "Staged original is missing") + | Some path -> + let native = Logseq_sync_native_crypto.Platform_crypto.crypto in + let crypto : Codec.crypto = {encrypt=native.encrypt_aes_gcm;decrypt=native.decrypt_aes_gcm} in + let bytes = In_channel.with_open_bin path In_channel.input_all in + Codec.decode ~maximum_plaintext_bytes:16 ~crypto ~key:None ~expected:version bytes + |> Result.map_error (fun error -> + let kind = match error with + | Codec.Checksum_mismatch -> Sync.Asset_checksum_mismatch + | Wire_too_large | Plaintext_too_large -> Sync.Asset_size_rejected + | Invalid_limit | Malformed_envelope | Malformed_transit _ | Decryption_failed _ -> + Sync.Asset_invalid_content (Codec.decode_error_message error) in + with_cause kind (Codec.decode_error_message error)) + |> fun result -> Result.bind result (fun verified -> + Cache.publish cache ~asset ~current:(fun () -> true) ~verified + |> Result.map_error failure) + |> fun result -> Result.bind result (fun handle -> + Cache.release_checked cache handle |> Result.map (fun () -> Sync.Asset_unit) + |> Result.map_error failure)) + | Sync.Retain_staged_file file -> + Ok (Sync.Asset_retained (Option.bind (Cache.retain_staged cache ~file) + (fun handle -> Option.map (fun path -> handle,path) (Cache.path cache handle)))) + | Sync.Retain_asset_file handle -> + Ok (Sync.Asset_retained (Option.bind (Cache.retain cache handle) + (fun retained -> Option.map (fun path -> retained,path) (Cache.path cache retained)))) + | Sync.Release_staged_file file -> + Cache.release_staged cache ~file |> Result.map (fun () -> Sync.Asset_unit) |> Result.map_error failure + | Sync.Release_asset_file handle -> + Cache.release_checked cache handle |> Result.map (fun () -> Sync.Asset_unit) |> Result.map_error failure + | Sync.Close_asset_scope -> + Cache.close_checked cache |> Result.map (fun () -> Sync.Asset_unit) |> Result.map_error failure + | Sync.Await_asset_operations -> + await_owner () |> Result.map (fun () -> Sync.Asset_unit) + |> Result.map_error (fun message -> Sync.Asset_invalid_content message) + | _ -> Alcotest.fail "Unexpected direct cache operation" in + let runner_ref = ref None in + let result_hook = ref (fun (_ : Sync.asset_output) -> ()) in + let held = Queue.create () in + let gate predicate = + let reached, resolve = Eio.Promise.create () in + result_hook := (fun output -> if predicate output then ( + let released, release = Eio.Promise.create () in + Queue.add release held; + ignore (Eio.Promise.try_resolve resolve () : bool); + Eio.Promise.await released)); + reached in + let deliver_held ?(keep_gate=false) () = + if not keep_gate then result_hook := (fun _ -> ()); + while not (Queue.is_empty held) do + Eio.Promise.resolve (Queue.take held) () + done in + let direct_services : Runner.services = + { (services ()) with + owner=(fun ~scope:actual -> + if actual<>scope then Error "Wrong owner incarnation" + else match !owner_ref with + | Some owner -> Ok (identity, owner) + | None -> + let store = Runner.sync_upload_store (Option.get !runner_ref) |> Result.get_ok in + let namespace = Logseq_db_worker_contract.Worker_pure_mutation.namespace_for_owner + ~origin:scope.account.managed_sync_origin ~user:scope.account.user_id ~graph:scope.graph_id in + let owner = Logseq_sync_worker_owner.Core.create + ~clock:(Eio.Stdenv.mono_clock env) ~namespace ~assets ~store + ~notify_outbox_changed:(fun ~owner:_ ~identity:_ -> Ok ()) in + let module Owner = (val Logseq_sync_worker_owner.Core.worker_owner owner : Execute.Core.WORKER_OWNER) in + Owner.bind_runtime ~identity ~database_owner ~limits ~cleanup_seconds:2. + ~network + |> Result.map_error D.error_message + |> Result.map (fun () -> owner_ref := Some owner; identity, owner)) + ; asset=(fun request -> + if request.Sync.scope<>scope then Error Sync.Asset_cancelled + else let result=fixture_result request.action in + !result_hook {Sync.request;result}; + result) + ; await_io=(fun ~scope:_ -> await_owner ()) + } in + let published = ref [] in + let dependencies = Runner.dependencies ~config:fixture.config ~overlay:fixture.overlay + ~runtime:(Runner.runtime ~fork:(fun ~sw task -> Eio.Fiber.fork ~sw task) + ~sleep:(Eio.Time.sleep (Eio.Stdenv.clock env)) |> Result.get_ok) + ~services:direct_services ~publish:(fun output -> published := output :: !published) |> Result.get_ok in + let upload_current = ref (fun _ -> false) and upload_events = Queue.create () in + let recovery_results = Queue.create () in + let state = Core.initial (Core.config ~worker:fixture.config ~sync:(asset_sync_config ())) |> Result.get_ok in + let sync, coordination = lifecycle_projection fixture.config scope in + let opened = Core.database_opened ~database_id:"staging-recovery" ~graph_id:scope.graph_id in + let transition = Core.step state (Core.Lifecycle_snapshot (sync, coordination, [scope, opened])) in + let recovery_ticket = List.find_map (function Core.Read_uploads ticket -> Some ticket | _ -> None) + transition.effects |> Option.get in + let runner = Runner.create ~sw dependencies ~post:(function + | Core.Asset_cleanup_barrier token -> + Runner.submit (Option.get !runner_ref) (Core.Asset_cleanup_barrier_reached token) + | Core.Upload_completed (_, event) -> Queue.add event upload_events + | Core.Uploads_loaded (_, result) -> Queue.add result recovery_results + | _ -> ()) ~recovery_current:(fun ticket -> ticket = recovery_ticket) ~upload_current:(fun ticket -> !upload_current ticket) + ~asset_current:(fun _ -> false) |> Result.get_ok in + runner_ref := Some runner; + Runner.submit runner (Core.Publish (Core.Sync_output (Sync.State_changed sync))); + Runner.submit runner (Core.Publish (Core.Sync_output (Sync.Session_state_changed (scope, sync)))); + Fun.protect ~finally:(fun () -> + deliver_held (); + Runner.shutdown runner; + let closed = Assets.close assets in + Logseq_sync_crypto.Core.retire crypto; + Logseq_overlay_db.Database.close database |> Result.get_ok; + match closed with Ok () -> () | Error error -> Alcotest.fail (D.error_message error)) (fun () -> - ignore - (Runner.prepare_import - fixture.runner - ~context:fixture.context - source - ~current:(fun () -> true)); - Alcotest.fail "gated import unexpectedly finished") - (fun () -> Eio.Promise.await reached); - let path = Option.get !(fixture.held_path) in - if not deliver_first - then - Alcotest.(check bool) - "queued accepted staging still exists" - true - (Sys.file_exists path); - fixture.deliver_held (); - Alcotest.(check bool) - "canceled caller releases accepted staging" - false - (Sys.file_exists path); - Alcotest.(check bool) - "staging cleanup requested through reducer" - true - (List.exists - (function - | Sync.Release_staged_file _ -> true - | _ -> false) - !(fixture.actions)); - let db = - Sqlite3.db_open (Filename.concat fixture.support "asset-upload-intents.sqlite") - in - let durable = - Logseq_db_storage.Asset_upload_store.read db ~operation:source.operation - |> Result.get_ok - in - let namespace=Logseq_db_worker_contract.Worker_pure_mutation.namespace_for_owner - ~origin:fixture.context.scope.account.managed_sync_origin ~user:fixture.context.scope.account.user_id - ~graph:fixture.context.scope.graph_id in - let reserved=Logseq_db_storage.Asset_upload_store.Inputs.read db - ~owner:{namespace;origin=Uri.to_string fixture.context.scope.account.managed_sync_origin;account=fixture.context.scope.account.user_id;graph=fixture.context.scope.graph_id} ~operation:source.operation |> Result.get_ok in - let expected=Logseq_db_types.Asset_import.Pure_intent.create ~namespace source |> Result.get_ok in - Alcotest.(check bool) "canceled staging retains complete durable Unknown input" true - (Option.exists(fun(row:Logseq_db_storage.Asset_upload_store.Inputs.record)->row.prepared=None && row.evidence=None - && Logseq_db_types.Asset_import.Pure_intent.encode row.input=Logseq_db_types.Asset_import.Pure_intent.encode expected)reserved); - ignore (Sqlite3.db_close db); - Alcotest.(check bool) - "cancel before staging acceptance persists no import" - true - (durable = None)) + try + Eio.Time.with_timeout_exn (Eio.Stdenv.clock env) 10. (fun () -> + run {runner;context={scope;encrypted=false;key=None};support;database;crypto;identity;actions; + fixture_result;fixture_asset=(fun action -> fixture_result action |> Result.get_ok); + gate;deliver_held;upload_events;published;set_upload_current=(fun current -> upload_current := current); + staged_path=(fun file -> Cache.staged_path cache ~file); + recover_uploads=(fun () -> + Runner.submit runner (Core.Read_uploads recovery_ticket); + let rec await turns = + if not (Queue.is_empty recovery_results) then Queue.take recovery_results + else if turns = 0 then Alcotest.fail "Recovery completion did not arrive" + else (Eio.Fiber.yield (); await (turns - 1)) in + await 1000)}) + with Eio.Time.Timeout -> + Alcotest.fail "Direct owner did not settle within 10s; check physical IO and fixture gate ownership")))) ;; -(* Only this actual interpreter can establish that source recovery opens an - existing SQLite file read-only and never stages, acquires, or replays work. *) let test_import_input_readonly_recovery_and_proof_ack () = - with_asset_bridge(fun fixture-> + with_direct_owner(fun fixture-> let module P=Logseq_db_types.Asset_import.Pure_intent in let module W=Logseq_db_worker_contract.Worker_pure_mutation in let scope=fixture.context.scope in @@ -1008,67 +790,43 @@ let test_import_input_readonly_recovery_and_proof_ack () = ;; let test_worker_sync_io_fence_correlates_real_child () = - with_asset_bridge ~asynchronous:true ~asynchronous_io:true(fun fixture-> - let source=picker_import fixture.support 950 in - let reached=fixture.gate ~hold_callback:true - (fun output->match output.Sync.request.action with Sync.Stage_asset_file _->true|_->false) - ~deliver_first:false in - Eio.Fiber.first - (fun()->ignore(Runner.prepare_import fixture.runner ~context:fixture.context source ~current:(fun()->true)); - Alcotest.fail "held source unexpectedly finished") - (fun()->Eio.Promise.await reached); - let outcome=ref None and request=ref None in - fixture.on_sync_fence(fun value->request:=Some value); - let scope=Sync.effect_scope_of_graph fixture.context.scope in - Eio.Fiber.fork ~sw:fixture.sw(fun()->outcome:=Some(Runner.await_sync_io fixture.runner ~scope)); - while !request=None do Eio.Fiber.yield() done; - let id,actual_scope=Option.get !request in - let wrong={actual_scope with graph_generation=Some 9999} in - Runner.submit fixture.runner(Core.Sync_io_fence_reached(id,wrong,Ok())); - Runner.submit fixture.runner(Core.Sync_io_fence_reached(Int64.succ id,actual_scope,Ok())); - for _=1 to 20 do Eio.Fiber.yield() done; - let premature= !outcome<>None in - fixture.deliver_held(); - while !outcome=None do Eio.Fiber.yield() done; - Alcotest.(check bool) "wrong scope/ID and canceled parent cannot fake I/O ACK" false premature; - Alcotest.(check bool) "actual Sync and Worker callback outlets settle exact fence" true(!outcome=Some(Ok())); - Runner.shutdown fixture.runner; + T.with_managed (fun fixture -> Eio_main.run (fun _ -> Eio.Switch.run (fun sw -> + let reached,reach=Eio.Promise.create () and physical,settle=Eio.Promise.create () in + let posted=ref [] and outcome=ref None in + let account : Sync.account_scope = + {managed_sync_origin=Uri.of_string "https://api.logseq.io";user_id="user-1"; + account_generation=3;presentation_generation=4;lifecycle_generation=5L} in + let scope=Sync.effect_scope_of_account account in + let services : Runner.services = + {post=(fun _ -> Alcotest.fail "Unexpected lifecycle intent"); + owner=(fun ~scope:_->Error "No graph owner is installed"); + asset=(fun _->Error Sync.Asset_cancelled); + await_io=(fun ~scope:actual-> + Alcotest.(check bool) "original account scope" true (actual=scope); + Eio.Promise.resolve reach (); Eio.Promise.await physical); + retire_account=(fun ~account:_->Ok ()); shutdown=(fun()->())} in + let dependencies=Runner.dependencies + ~runtime:(Runner.runtime ~fork:(fun ~sw task->Eio.Fiber.fork ~sw task) + ~sleep:(fun _->()) |> Result.get_ok) + ~config:fixture.config ~overlay:fixture.overlay ~services ~publish:(fun _->()) |> Result.get_ok in + let runner=Runner.create ~sw dependencies ~post:(fun event->posted:=event:: !posted) + ~recovery_current:(fun _->false) ~upload_current:(fun _->false) + ~asset_current:(fun _->false) |> Result.get_ok in + Eio.Fiber.fork ~sw (fun()->outcome:=Some(Runner.await_sync_io runner ~scope)); + Eio.Promise.await reached; + Alcotest.(check bool) "physical completion is required" true (!outcome=None); + Alcotest.(check int) "no reducer ticket or fence event" 0 (List.length !posted); + Eio.Promise.resolve settle (Error "physical cleanup failed"); + while !outcome=None do Eio.Fiber.yield () done; + Alcotest.(check bool) "exact cleanup failure survives forwarding" true + (!outcome=Some(Error "physical cleanup failed")); + Runner.shutdown runner; Alcotest.(check bool) "stopped Worker cannot mint an I/O ACK" true - (Result.is_error(Runner.await_sync_io fixture.runner ~scope))) -;; - -let test_sync_io_fence_joins_nonasset_callback () = - T.with_managed(fun fixture->Eio_main.run(fun env->Eio.Switch.run(fun sw-> - let reached,reached_resolve=Eio.Promise.create() and release,release_resolve=Eio.Promise.create() in - let fenced=ref None in - let runner=asset_io_runner ~env ~sw ~support:fixture.config.application_support_directory - ~fork:(fun ~sw task->Eio.Fiber.fork ~sw task) - ~post:(function - |Sync.Runner_completed _->ignore(Eio.Promise.try_resolve reached_resolve():bool);Eio.Promise.await release - |Sync.IO_fence_completed(id,_,outcome) when id=1L->fenced:=Some outcome - |_->())() in - Fun.protect ~finally:(fun()->ignore(Eio.Promise.try_resolve release_resolve():bool);Sync_runner.shutdown runner)(fun()-> - let limits=Sync.limits ~maximum_response_bytes:1024 ~maximum_artifact_bytes:1024 ~submission_batch_size:1 |> Result.get_ok in - let state=Sync.config ~managed_sync_origin:(Uri.of_string "https://api.logseq.io") ~limits |> Result.get_ok |> Sync.initial |> Result.get_ok in - let state=(Sync.step state(Sync.Sign_in_requested{account_generation=0;presentation_generation=0})).next in - let authenticated=Sync.step state(Sync.Account_authenticated{user_id=Some "user-1"}) in - let request=List.find_map(function Sync.Run(Sync.Request(_,Sync.Fetch_catalog _) as request)->Some request|_->None)authenticated.effects |> Option.get in - Sync_runner.submit runner request; - Eio.Promise.await reached; - let scope:Sync.effect_scope={account_generation=None;graph_generation=None;connection_generation=None; - presentation_generation=None;lifecycle_generation=None} in - let barrier=Sync.step authenticated.next(Sync.IO_fence_requested(1L,scope)) in - List.iter(function Sync.Run instruction->Sync_runner.submit runner instruction|_->())barrier.effects; - for _=1 to 20 do Eio.Fiber.yield() done; - let premature= !fenced<>None in - ignore(Eio.Promise.try_resolve release_resolve():bool); - while !fenced=None do Eio.Fiber.yield() done; - Alcotest.(check bool) "generic Sync callback belongs to its actual I/O fence" false premature; - Alcotest.(check bool) "generic Sync callback completion settles fence" true(!fenced=Some(Ok())))))) + (Result.is_error(Runner.await_sync_io runner ~scope))))) ;; let test_scope_close_keeps_live_staged_lease () = - with_asset_bridge(fun fixture-> + with_direct_owner(fun fixture-> let source=picker_import fixture.support 960 in let staged=match fixture.fixture_asset(Sync.Stage_asset_file {operation=source.operation;file_type=source.file_type;source_file=source.source_file})with @@ -1090,7 +848,7 @@ let test_scope_close_keeps_live_staged_lease () = (* Physical cache/lease ownership is outside Core.step; exercise the real Sync storage owner through the Worker interpreter, not an injected successful ACK. *) let test_host_scope_settlement_preserves_live_lease () = - with_asset_bridge ~asynchronous:true ~asynchronous_io:true(fun fixture-> + with_direct_owner(fun fixture-> let source=picker_import fixture.support 961 in let prepared=Runner.prepare_import fixture.runner ~context:fixture.context source ~current:(fun()->true) |> Result.get_ok in let lease,path=Runner.retain_imported_file fixture.runner ~scope:fixture.context.scope ~operation:source.operation |> Option.get in @@ -1119,38 +877,30 @@ let test_observation_release_waits_for_initial_snapshot () = T.with_managed(fun fixture-> ignore(Fixture.seed_mirror fixture.config.application_support_directory); Eio_main.run(fun env->Eio.Switch.run(fun sw-> - let worker=ref None in - let post event=Option.iter(fun db->Db.post db(Core.Sync_event event))!worker in - let graph:Sync.graph={graph_id=Fixture.graph_uuid;name="Observation snapshot fixture"; - schema={major=65;minor=33;exact=true};encrypted=false}in - let sync_runner=Runner.sync_runner ~retire_account:(fun ~account:_->Ok()) ~shutdown:(fun()->()) - ~submit:(function - |Sync.Request(ticket,Sync.Load_catalog _)->post(Sync.Runner_completed(Sync.Completion(ticket,Ok(Some - (Sync.catalog_cache ~user_id:"fixture-user" ~graphs:[graph] ~selected_graph:(Some graph.graph_id)))))) - |Sync.Request(ticket,Sync.Save_catalog _)->post(Sync.Runner_completed(Sync.Completion(ticket,Ok()))) - |Sync.Asset_io(ticket,request)->let value=match request.Sync.action with - |Sync.Prune_asset_staging _->Sync.Asset_pruned 0 - |Close_asset_scope|Release_asset_file _|Release_staged_file _|Cancel_asset_operation _->Sync.Asset_unit - |_->failwith "Unexpected asset operation in observation fixture"in - post(Sync.Runner_completed(Sync.Asset_completion(ticket,Ok value))) - |_->())()in - let dependencies=Runner.dependencies ~config:fixture.config ~overlay:fixture.overlay ~sync_runner + let scope : Sync.graph_scope = {account=account "fixture-user"; + graph_id=Fixture.graph_uuid;graph_generation=1} in + let services = services () in + let dependencies=Runner.dependencies ~config:fixture.config ~overlay:fixture.overlay ~services ~runtime:(Runner.runtime ~fork:(fun ~sw task->Eio.Fiber.fork ~sw task) ~sleep:(Eio.Time.sleep(Eio.Stdenv.clock env)) |> Result.get_ok) ~publish:(fun _->()) |> Result.get_ok in - let db=Db.create ~sw ~config:(Core.config ~worker:fixture.config ~sync:(asset_sync_config())) + let db=Db.create_lifecycle ~sw ~config:(Core.config ~worker:fixture.config ~sync:(asset_sync_config())) ~runner_dependencies:dependencies |> Result.get_ok in - worker:=Some db; - Fun.protect ~finally:(fun()->Db.shutdown db)(fun()-> + let inspection=Logseq_overlay_db.Database.inspect_mirror + ~application_support_directory:fixture.config.application_support_directory ~graph_id:scope.graph_id |> Result.get_ok in + let database=Logseq_overlay_db.Database.open_ ~sw fixture.overlay inspection + ~graph_name:"Observation snapshot fixture" |> Result.get_ok in + Fun.protect ~finally:(fun()->Db.shutdown db; + Logseq_overlay_db.Database.close database |> Result.get_ok)(fun()-> + let opened=Db.attach_database_owner db ~scope (Execute.Core.database_owner database) |> Result.get_ok in + let sync,coordination=lifecycle_projection fixture.config scope in + Db.post db (Core.Lifecycle_snapshot(sync,coordination,[scope,opened])); let latest=ref None in ignore(Db.register_coordination db ~registration:801L ~notify:(fun value->latest:=Some value) |> Result.get_ok); - post(Sync.Restore_local_account{user_id="fixture-user"}); Eio.Time.with_timeout_exn(Eio.Stdenv.clock env)3.(fun()-> while not(Option.exists(fun(snapshot:Db.coordination_snapshot)-> List.exists(fun(_,state)->state.Db.phase=Db.Graph_open)snapshot.state.residents)!latest) do Eio.Fiber.yield()done); - let snapshot=Option.get !latest in - let scope,_=List.find(fun(_,state)->state.Db.phase=Db.Graph_open)snapshot.state.residents in - let owner=Option.get snapshot.state.sync.selected in + let owner=Option.get coordination.selected in let binding=F.binding ~serial:801L ~incarnation:(Read.worker_incarnation ~runtime_epoch:(Journal_worker_ids.Runtime.Epoch.of_int64 29000L) ~worker_generation:Journal_worker_ids.Worker.Generation.one) @@ -1300,176 +1050,38 @@ let test_observation_release_waits_for_initial_snapshot () = let test_host_followup_survives_full_actor_queue () = T.with_managed(fun fixture->Eio_main.run(fun env->Eio.Switch.run(fun sw-> let module Db=Logseq_db_worker in - let worker=ref None in - let post event=Option.iter(fun worker->Db.post worker(Core.Sync_event event))!worker in - let sync_runner=Runner.sync_runner ~retire_account:(fun ~account:_->Alcotest.fail "Unexpected account retirement in fixture") ~shutdown:(fun()->()) ~submit:(function - |Sync.Request(ticket,Sync.Load_catalog _)->post(Sync.Runner_completed(Sync.Completion(ticket,Ok None))) - |Sync.Await_io(id,scope)->post(Sync.IO_fence_completed(id,scope,Ok())) - |_->())() in - let dependencies=Runner.dependencies ~config:fixture.config ~overlay:fixture.overlay ~sync_runner + let forwarded=ref 0 in + let dependencies=Runner.dependencies ~config:fixture.config ~overlay:fixture.overlay + ~services:(services ~post:(function Core.Foreground_intent false -> incr forwarded | _ -> ()) ()) ~runtime:(Runner.runtime ~fork:(fun ~sw task->Eio.Fiber.fork ~sw task) - ~sleep:(Eio.Time.sleep(Eio.Stdenv.clock env)) |> Result.get_ok) ~publish:(fun _->()) |> Result.get_ok in - let db=Db.create ~sw ~config:(Core.config ~worker:fixture.config ~sync:(asset_sync_config())) + ~sleep:(Eio.Time.sleep(Eio.Stdenv.clock env)) |> Result.get_ok) + ~publish:(fun _->()) |> Result.get_ok in + let db=Db.create_lifecycle ~sw ~config:(Core.config ~worker:fixture.config ~sync:(asset_sync_config())) ~runner_dependencies:dependencies |> Result.get_ok in - worker:=Some db; Fun.protect ~finally:(fun()->Db.shutdown db)(fun()-> - let latest=ref None and armed=ref false and filled=ref false in + let scope : Sync.graph_scope = {account=account "host-queue-next"; + graph_id=Test_support.graph_uuid;graph_generation=1} in + let sync,coordination=lifecycle_projection fixture.config scope in + let filled=ref false and latest=ref None in let notify (snapshot:Db.coordination_snapshot)= latest:=Some snapshot; - if !armed && not !filled && snapshot.state.sync.account=None then ( + if not !filled then ( filled:=true; for _=1 to 256 do Db.post db(Core.Set_foreground false) done; - Eio.Fiber.fork ~sw(fun()->post(Sync.Restore_local_account{user_id="host-queue-next"}))) in + Eio.Fiber.fork ~sw(fun()->Db.post db(Core.Lifecycle_snapshot(sync,coordination,[])))) in ignore(Db.register_coordination db ~registration:79L ~notify |> Result.get_ok); - post(Sync.Restore_local_account{user_id="host-queue-old"}); - Eio.Time.with_timeout_exn(Eio.Stdenv.clock env) 2.(fun()-> - while not(Option.exists(fun(snapshot:Db.coordination_snapshot)->snapshot.state.sync.account<>None)!latest)do Eio.Fiber.yield()done; - let snapshot=Option.get !latest in - let owner=Option.get snapshot.state.sync.account in - let plan:Db.host_plan={version=snapshot.version;registration=79L;owner; - id=Logseq_db_types.Graph_types.Uuid.of_string "10000000-0000-4000-8000-000000079079" |> Result.get_ok; - change=Core.Host_handoff.Replace_account;expected=Rrbvec.empty} in - let receipt=Db.prepare_host_change db plan |> Result.get_ok in - ignore(Db.apply_host_change db receipt |> Result.get_ok); - armed:=true;post(Sync.Account_authenticated{user_id=None}); - while not(Option.exists(fun(snapshot:Db.coordination_snapshot)-> - Option.exists(fun(account:Sync.account_scope)->account.user_id="host-queue-next")snapshot.state.sync.account)!latest) - do Eio.Fiber.yield()done; - Alcotest.(check bool) "consumer drains full queue across internal follow-up" true !filled))))) -;; - -let test_asset_io_barrier_does_not_block_serial_submit () = - T.with_managed(fun fixture->Eio_main.run(fun env->Eio.Switch.run(fun sw-> - let tasks=Queue.create() and events=ref [] in - let runner=asset_io_runner ~env ~sw ~support:fixture.config.application_support_directory - ~fork:(fun ~sw:_ task->Queue.add task tasks) ~post:(fun event->events:=event::!events)() in - Fun.protect ~finally:(fun()->Sync_runner.shutdown runner)(fun()-> - let graph_id=Logseq_db_types.Graph_types.Uuid.of_string "22000000-0000-4000-8000-000000000001" |> Result.get_ok in - let initial,context=admitted_asset_state graph_id in - let source=picker_import fixture.config.application_support_directory 980 in - let issue state operation action= - let transition=Sync.step state(Sync.Asset_requested{scope=context.scope;operation;action}) in - let instruction=List.find_map(function Sync.Run instruction->Some instruction|_->None)transition.effects |> Option.get in - transition.next,instruction in - let pending,stage=issue initial "actual-stage-before-barrier" - (Sync.Stage_asset_file{operation=source.operation;file_type=source.file_type;source_file=source.source_file}) in - Sync_runner.submit runner stage; - let barrier_state,barrier=issue pending "actual-io-barrier" Sync.Await_asset_operations in - let returned=ref false in - Eio.Fiber.fork ~sw(fun()->Sync_runner.submit runner barrier;returned:=true); - Eio.Fiber.yield(); - let blocked=not !returned in - let later_source=picker_import fixture.config.application_support_directory 990 in - let _,later=issue barrier_state "later-unrelated-stage" - (Sync.Stage_asset_file{operation=later_source.operation;file_type=later_source.file_type;source_file=later_source.source_file}) in - Sync_runner.submit runner later; - (Queue.take tasks)(); (* actual first source finishes; later source stays queued *) - let barrier_finished=ref false in - if not(Queue.is_empty tasks) then ( - let task=Queue.take tasks in - Eio.Fiber.fork ~sw(fun()->task();barrier_finished:=true)); - for _=1 to 4 do Eio.Fiber.yield() done; - let joined_later=not !barrier_finished in - while not(Queue.is_empty tasks)do (Queue.take tasks)() done; - while not !returned || not !barrier_finished do Eio.Fiber.yield() done; - Alcotest.(check bool) "Await submit must leave the serial callback consumer free" false blocked; - Alcotest.(check bool) "barrier cannot capture later unrelated I/O" false joined_later; - Alcotest.(check bool) "actual asset owner still reports completion" true(List.length !events>=3))))) -;; - -let test_writer_fence_joins_actual_canceled_asset_callback () = - with_asset_bridge ~asynchronous:true ~asynchronous_io:true(fun fixture-> - let source=picker_import fixture.support 970 in - let reached=fixture.gate ~hold_callback:true - (fun output->match output.Sync.request.action with Sync.Stage_asset_file _->true|_->false) - ~deliver_first:false in - Eio.Fiber.first - (fun()->ignore(Runner.prepare_import fixture.runner ~context:fixture.context source ~current:(fun()->true)); - Alcotest.fail "held actual staging callback unexpectedly returned") - (fun()->Eio.Promise.await reached); - let outcome=ref None in - Eio.Fiber.fork ~sw:fixture.sw (fun()-> - outcome:=Some(Runner.quiesce_writers fixture.runner ~scope:fixture.context.scope)); - for _=1 to 20 do Eio.Fiber.yield() done; - let premature= !outcome<>None in - fixture.deliver_held(); - while !outcome=None do Eio.Fiber.yield() done; - Alcotest.(check bool) "actual canceled Sync task callback must finish before writer ACK" false premature; - Alcotest.(check bool) "actual child callback allows writer barrier" true(!outcome=Some(Ok()))) -;; - -let test_cancelled_retained_output () = - with_asset_bridge (fun fixture -> - let source = picker_import fixture.support 20 in - let prepared = - Runner.prepare_import - fixture.runner - ~context:fixture.context - source - ~current:(fun () -> true) - |> Result.get_ok - in - let reached = - fixture.gate - (fun output -> - match output.Sync.request.action with - | Sync.Retain_staged_file _ -> true - | _ -> false) - ~deliver_first:false - in - Eio.Fiber.first - (fun () -> - ignore - (Runner.retain_imported_file - fixture.runner - ~scope:fixture.context.scope - ~operation:source.operation); - Alcotest.fail "gated lease acquisition unexpectedly finished") - (fun () -> Eio.Promise.await reached); - let path = Option.get !(fixture.held_path) in - fixture.deliver_held (); - Alcotest.(check bool) - "canceled lease acquisition requests release" - true - (List.exists - (function - | Sync.Release_asset_file _ -> true - | _ -> false) - !(fixture.actions)); - ignore (fixture.fixture_asset (Sync.Release_staged_file prepared.staged_file)); - Alcotest.(check bool) - "released lease no longer pins staged bytes" - false - (Sys.file_exists path)) -;; - -let test_checked_release_exact_physical_completion () = - with_asset_bridge(fun fixture-> - let source=picker_import fixture.support 60 in - ignore(Runner.prepare_import fixture.runner ~context:fixture.context source - ~current:(fun()->true) |> Result.get_ok); - let lease,path=Runner.retain_imported_file fixture.runner ~scope:fixture.context.scope - ~operation:source.operation |> Option.get in - let wrong_scope={fixture.context.scope with graph_generation=fixture.context.scope.graph_generation+1} in - Alcotest.(check bool) "cancelled scoped release is not ACK" true - (Result.is_error(Runner.release_asset_file_checked fixture.runner ~scope:wrong_scope ~handle:lease)); - Alcotest.(check bool) "old scope release leaves staged original intact" true (Sys.file_exists path); - let reached=fixture.gate (fun output->match output.Sync.request.action with - | Sync.Release_asset_file handle->handle=lease | _->false) ~deliver_first:false in - let completed=ref None in - Eio.Fiber.both - (fun()->completed:=Some(Runner.release_asset_file_checked fixture.runner ~scope:fixture.context.scope ~handle:lease)) - (fun()->Eio.Promise.await reached; - Alcotest.(check bool) "release ACK waits for actual owner completion" true (!completed=None); - fixture.deliver_held()); - Alcotest.(check bool) "exact physical release completes" true (!completed=Some(Ok())); - Runner.shutdown fixture.runner; - Alcotest.(check bool) "stopped runner cannot acknowledge release" true - (Result.is_error(Runner.release_asset_file_checked fixture.runner ~scope:fixture.context.scope ~handle:lease))) + let previous_scope={scope with account=account "host-queue-old"} in + let previous_sync,previous_coordination=lifecycle_projection fixture.config previous_scope in + Db.post db(Core.Lifecycle_snapshot(previous_sync,previous_coordination,[])); + Eio.Time.with_timeout_exn(Eio.Stdenv.clock env)2.(fun()-> + while !forwarded<256 || not(Option.exists(fun(snapshot:Db.coordination_snapshot)-> + snapshot.state.sync.account=Some scope.account)!latest) do Eio.Fiber.yield()done); + Alcotest.(check bool) "consumer drains full queue across lifecycle follow-up" true !filled; + Alcotest.(check int) "every queued foreground intent reaches lifecycle owner" 256 !forwarded)))) ;; let test_checked_release_storage_retry () = - with_asset_bridge(fun fixture-> + with_direct_owner(fun fixture-> let source=picker_import fixture.support 80 in let prepared=Runner.prepare_import fixture.runner ~context:fixture.context source ~current:(fun()->true) |> Result.get_ok in @@ -1488,7 +1100,7 @@ let test_checked_release_storage_retry () = ;; let test_discarded_retain_retries_real_storage_cleanup () = - with_asset_bridge ~asynchronous:true(fun fixture-> + with_direct_owner(fun fixture-> let source=picker_import fixture.support 81 in let prepared=Runner.prepare_import fixture.runner ~context:fixture.context source ~current:(fun()->true) |> Result.get_ok in @@ -1513,20 +1125,72 @@ let test_discarded_retain_retries_real_storage_cleanup () = (* The public pure cleanup owner correctly queues both releases. Its state cannot reproduce the interpreter's job finishing before queued physical IO starts. *) +let test_asset_cleanup_snapshot_retains_physical_failure () = + with_direct_owner(fun fixture-> + let source=picker_import fixture.support 83 in + let prepared=Runner.prepare_import fixture.runner ~context:fixture.context source ~current:(fun()->true) |> Result.get_ok in + let lease,path=Runner.retain_imported_file fixture.runner ~scope:fixture.context.scope ~operation:source.operation |> Option.get in + ignore(fixture.fixture_asset(Sync.Release_staged_file prepared.staged_file)); + let directory=Filename.dirname path in + Unix.chmod directory 0o500; + Fun.protect ~finally:(fun()->Unix.chmod directory 0o700)(fun()-> + Runner.submit fixture.runner(Core.Run_asset(fixture.context,Logseq_sync_asset_transfer.Core.Release_handle lease)); + let snapshot=Runner.asset_cleanup_snapshot fixture.runner ~scope:fixture.context.scope in + Alcotest.(check bool) "a failed physical resource cannot produce settlement ACK" true + (Result.is_error(Runner.await_asset_cleanup fixture.runner snapshot)); + Alcotest.(check bool) "failure retains actual bytes" true(Sys.file_exists path)); + let deadline=Unix.gettimeofday()+.2. in + while Runner.unsettled_orphan_retains fixture.runner>0 && Unix.gettimeofday() retained + | _ -> Alcotest.fail "Concrete cache owner did not retain staged file" +;; + +let test_checked_release_exact_physical_completion () = + with_direct_owner(fun fixture-> + let source=picker_import fixture.support 60 in + let prepared=Runner.prepare_import fixture.runner ~context:fixture.context source + ~current:(fun()->true) |> Result.get_ok in + let lease,path=retain_fixture_staged_file fixture prepared.staged_file in + let wrong_scope={fixture.context.scope with graph_generation=fixture.context.scope.graph_generation+1} in + Alcotest.(check bool) "cancelled scoped release is not ACK" true + (Result.is_error(Runner.release_asset_file_checked fixture.runner ~scope:wrong_scope ~handle:lease)); + Alcotest.(check bool) "old scope release leaves staged original intact" true (Sys.file_exists path); + let reached=fixture.gate (fun output->match output.Sync.request.action with + | Sync.Release_asset_file handle->handle=lease | _->false) in + let completed=ref None in + Eio.Fiber.both + (fun()->completed:=Some(Runner.release_asset_file_checked fixture.runner ~scope:fixture.context.scope ~handle:lease)) + (fun()->Eio.Promise.await reached; + Alcotest.(check bool) "release ACK waits for actual owner completion" true (!completed=None); + fixture.deliver_held()); + Alcotest.(check bool) "exact physical release completes" true (!completed=Some(Ok())); + Runner.shutdown fixture.runner; + Alcotest.(check bool) "stopped runner cannot acknowledge release" true + (Result.is_error(Runner.release_asset_file_checked fixture.runner ~scope:fixture.context.scope ~handle:lease))) +;; + let test_asset_cleanup_snapshot_joins_queued_release () = - with_asset_bridge ~asynchronous:true ~asynchronous_io:true(fun fixture-> + with_direct_owner(fun fixture-> let source=picker_import fixture.support 87 in let prepared=Runner.prepare_import fixture.runner ~context:fixture.context source ~current:(fun()->true) |> Result.get_ok in - let retain()=Runner.retain_imported_file fixture.runner ~scope:fixture.context.scope ~operation:source.operation |> Option.get |> fst in + let retain()=retain_fixture_staged_file fixture prepared.staged_file |> fst in let first=retain() and second=retain() in ignore(fixture.fixture_asset(Sync.Release_staged_file prepared.staged_file)); let second_reached,resolve_second=Eio.Promise.create() in let reached=fixture.gate(fun output->match output.Sync.request.action with | Sync.Release_asset_file handle when handle=first->true | Sync.Release_asset_file handle when handle=second->Eio.Promise.resolve resolve_second();true - | _->false) ~deliver_first:false in + | _->false) in let release handle=Runner.submit fixture.runner(Core.Run_asset(fixture.context, - Logseq_sync_pure_reducer.Asset_transfer.Release_handle handle)) in + Logseq_sync_asset_transfer.Core.Release_handle handle)) in release first; Eio.Promise.await reached; release second; @@ -1543,14 +1207,14 @@ let test_asset_cleanup_snapshot_joins_queued_release () = ;; let test_asset_cleanup_snapshot_joins_physical_release () = - with_asset_bridge ~asynchronous:true(fun fixture-> + with_direct_owner(fun fixture-> let source=picker_import fixture.support 82 in let prepared=Runner.prepare_import fixture.runner ~context:fixture.context source ~current:(fun()->true) |> Result.get_ok in - let lease,path=Runner.retain_imported_file fixture.runner ~scope:fixture.context.scope ~operation:source.operation |> Option.get in + let lease,path=retain_fixture_staged_file fixture prepared.staged_file in ignore(fixture.fixture_asset(Sync.Release_staged_file prepared.staged_file)); let reached=fixture.gate(fun output->match output.Sync.request.action with - | Sync.Release_asset_file handle->handle=lease|_->false) ~deliver_first:false in - Runner.submit fixture.runner(Core.Run_asset(fixture.context,Logseq_sync_pure_reducer.Asset_transfer.Release_handle lease)); + | Sync.Release_asset_file handle->handle=lease|_->false) in + Runner.submit fixture.runner(Core.Run_asset(fixture.context,Logseq_sync_asset_transfer.Core.Release_handle lease)); let snapshot=Runner.asset_cleanup_snapshot fixture.runner ~scope:fixture.context.scope in let completed=ref None in Eio.Fiber.both @@ -1562,83 +1226,15 @@ let test_asset_cleanup_snapshot_joins_physical_release () = Alcotest.(check bool) "retired actual staging bytes were removed" false(Sys.file_exists path)) ;; -let test_asset_cleanup_ignores_later_unrelated_work () = - with_asset_bridge ~asynchronous:true(fun fixture-> - let module Transfer=Logseq_sync_pure_reducer.Asset_transfer in - let asset=Logseq_db_types.Asset_descriptor.create ~uuid:(Logseq_db_types.Graph_types.Uuid.of_string "22000000-0000-4000-8000-000000000085" |> Result.get_ok) - ~source:(Managed None) ~current_checksum:(Some(String.make 64 'a')) ~size:None ~dimensions:None |> Result.get_ok in - let transfer=Transfer.create(Transfer.config ~active:2 ~foreground_reserved:1 ~pending:3 ~retries:0 |> Result.get_ok) - ~scope:fixture.context.scope ~online:true ~unlocked:true in - let _,work=Transfer.step transfer(Replace{consumer="cancel-before-unrelated";priority=Foreground;assets=[asset]}) in - let lookup=List.find_map(function Transfer.Check_cache ticket->Some ticket|_->None)work |> Option.get in - let reached=fixture.hold_asset_barrier() in - Runner.submit fixture.runner(Core.Run_asset(fixture.context,Transfer.Cancel(Lookup lookup))); - let snapshot=Runner.asset_cleanup_snapshot fixture.runner ~scope:fixture.context.scope in - let source=picker_import fixture.support 85 in - let prepared=Runner.prepare_import fixture.runner ~context:fixture.context source ~current:(fun()->true) |> Result.get_ok in - let lease,_=Runner.retain_imported_file fixture.runner ~scope:fixture.context.scope ~operation:source.operation |> Option.get in - ignore(fixture.fixture_asset(Sync.Release_staged_file prepared.staged_file)); - let physical=fixture.gate(fun output->match output.Sync.request.action with Sync.Release_asset_file handle->handle=lease|_->false) ~deliver_first:false in - let completed=ref None in - Eio.Fiber.both - (fun()->completed:=Some(Runner.await_asset_cleanup fixture.runner snapshot)) - (fun()->Eio.Promise.await reached; - Runner.submit fixture.runner(Core.Run_asset(fixture.context,Transfer.Release_handle lease)); - Eio.Promise.await physical; - fixture.deliver_asset_barriers(); - for _=0 to 9 do Eio.Fiber.yield() done; - let unrelated= !completed=Some(Ok()) in - fixture.deliver_held(); - Alcotest.(check bool) "later unrelated cleanup cannot join an earlier consumer ACK" true unrelated)) -;; - -let test_asset_cleanup_joins_queued_result_child () = - with_asset_bridge ~asynchronous:true(fun fixture-> - let module Transfer=Logseq_sync_pure_reducer.Asset_transfer in - let source=picker_import fixture.support 86 in - let prepared=Runner.prepare_import fixture.runner ~context:fixture.context source ~current:(fun()->true) |> Result.get_ok in - ignore(fixture.fixture_asset(Sync.Publish_staged_asset{asset=prepared.asset;version=prepared.version;file=prepared.staged_file})); - let asset=Logseq_db_types.Asset_descriptor.create ~uuid:prepared.asset ~source:(Managed(Some prepared.version)) - ~current_checksum:None ~size:None ~dimensions:None |> Result.get_ok in - let transfer=Transfer.create(Transfer.config ~active:2 ~foreground_reserved:1 ~pending:3 ~retries:0 |> Result.get_ok) - ~scope:fixture.context.scope ~online:true ~unlocked:true in - let active,work=Transfer.step transfer(Replace{consumer="queued-result-child";priority=Foreground;assets=[asset]}) in - let active=ref active in - fixture.set_asset_current(fun ticket->Transfer.ticket_current !active ticket); - let received,received_resolve=Eio.Promise.create() in - fixture.on_asset_completed(fun event->Eio.Promise.resolve received_resolve event); - List.iter(fun instruction->match instruction with Transfer.Check_cache _->Runner.submit fixture.runner(Core.Run_asset(fixture.context,instruction))|_->())work; - let result=Eio.Promise.await received in - let released,cancel=Transfer.step !active(Release "queued-result-child") in - active:=released; - let barrier=fixture.hold_asset_barrier() in - List.iter(fun instruction->match instruction with Transfer.Cancel _->Runner.submit fixture.runner(Core.Run_asset(fixture.context,instruction))|_->())cancel; - let snapshot=Runner.asset_cleanup_snapshot fixture.runner ~scope:fixture.context.scope in - let physical=fixture.gate(fun output->match output.Sync.request.action with Sync.Release_asset_file _->true|_->false) ~deliver_first:false in - let completed=ref None in - Eio.Fiber.both - (fun()->completed:=Some(Runner.await_asset_cleanup fixture.runner snapshot)) - (fun()->Eio.Promise.await barrier; - let stale,children=Transfer.step !active result in active:=stale; - List.iter(fun instruction->match instruction with Transfer.Release_handle _->Runner.submit fixture.runner(Core.Run_asset(fixture.context,instruction))|_->())children; - Eio.Promise.await physical; - fixture.deliver_asset_barriers(); - for _=0 to 9 do Eio.Fiber.yield() done; - let waiting= !completed=None in - fixture.deliver_held(); - Alcotest.(check bool) "queued successful result's actual release child joins cancellation" true waiting); - Alcotest.(check bool) "exact causal resource ACK settles after real child IO" true(!completed=Some(Ok()))) -;; - let test_asset_cleanup_stop_does_not_wait_for_lost_io () = - with_asset_bridge ~asynchronous:true(fun fixture-> + with_direct_owner(fun fixture-> let source=picker_import fixture.support 84 in let prepared=Runner.prepare_import fixture.runner ~context:fixture.context source ~current:(fun()->true) |> Result.get_ok in - let lease,_=Runner.retain_imported_file fixture.runner ~scope:fixture.context.scope ~operation:source.operation |> Option.get in + let lease,_=retain_fixture_staged_file fixture prepared.staged_file in ignore(fixture.fixture_asset(Sync.Release_staged_file prepared.staged_file)); let reached=fixture.gate(fun output->match output.Sync.request.action with - | Sync.Release_asset_file handle->handle=lease|_->false) ~deliver_first:false in - Runner.submit fixture.runner(Core.Run_asset(fixture.context,Logseq_sync_pure_reducer.Asset_transfer.Release_handle lease)); + | Sync.Release_asset_file handle->handle=lease|_->false) in + Runner.submit fixture.runner(Core.Run_asset(fixture.context,Logseq_sync_asset_transfer.Core.Release_handle lease)); let snapshot=Runner.asset_cleanup_snapshot fixture.runner ~scope:fixture.context.scope in let completed=ref None in Eio.Fiber.both @@ -1650,1952 +1246,284 @@ let test_asset_cleanup_stop_does_not_wait_for_lost_io () = Alcotest.(check bool) "stopped cleanup reports failure without waiting for lost callback" true returned)) ;; -let test_asset_cleanup_snapshot_retains_physical_failure () = - with_asset_bridge ~asynchronous:true(fun fixture-> - let source=picker_import fixture.support 83 in - let prepared=Runner.prepare_import fixture.runner ~context:fixture.context source ~current:(fun()->true) |> Result.get_ok in - let lease,path=Runner.retain_imported_file fixture.runner ~scope:fixture.context.scope ~operation:source.operation |> Option.get in - ignore(fixture.fixture_asset(Sync.Release_staged_file prepared.staged_file)); - let directory=Filename.dirname path in - Unix.chmod directory 0o500; - Fun.protect ~finally:(fun()->Unix.chmod directory 0o700)(fun()-> - Runner.submit fixture.runner(Core.Run_asset(fixture.context,Logseq_sync_pure_reducer.Asset_transfer.Release_handle lease)); - let snapshot=Runner.asset_cleanup_snapshot fixture.runner ~scope:fixture.context.scope in - Alcotest.(check bool) "a failed physical resource cannot produce settlement ACK" true - (Result.is_error(Runner.await_asset_cleanup fixture.runner snapshot)); - Alcotest.(check bool) "failure retains actual bytes" true(Sys.file_exists path)); - let deadline=Unix.gettimeofday()+.2. in - while Runner.unsettled_orphan_retains fixture.runner>0 && Unix.gettimeofday() + let source = picker_import fixture.support 982 in + let intent = Runner.prepare_import fixture.runner ~context:fixture.context source + ~current:(fun () -> true) |> Result.get_ok in + let path = fixture.staged_path intent.staged_file |> Option.get in + let directory = Filename.dirname path in + let saved = directory ^ "-synthetic-failure" in + Unix.rename directory saved; + let channel = open_out_bin directory in + close_out channel; + Fun.protect ~finally:(fun () -> Sys.remove directory; Unix.rename saved directory) + (fun () -> + match fixture.recover_uploads () with + | Ok _ -> Alcotest.fail "Physical staging error must not acknowledge reconciliation" + | Error message -> + let cause = "Path component is not a directory" in + let rec contains offset = + offset + String.length cause <= String.length message + && (String.sub message offset (String.length cause) = cause || contains (offset + 1)) in + Alcotest.(check bool) "physical failure retains ENOTDIR cause" true (contains 0)); + Alcotest.(check bool) "failed recovery preserves original physical source" true (Sys.file_exists path); + Alcotest.(check bool) "only actual successful prune settles recovery" true + (fixture.recover_uploads () = Ok [intent])) ;; -let test_db_shutdown_accepts_late_staging_completion () = - let module Db = Logseq_db_worker in - T.with_managed (fun fixture -> - Eio_main.run (fun env -> - Eio.Switch.run (fun sw -> - let support = fixture.config.application_support_directory in - let source = picker_import support 40 in - let graph : Sync.graph = - { graph_id = source.target - ; name = "Shutdown fixture" - ; schema = { major = 65; minor = 33; exact = true } - ; encrypted = false - } - in - let db = ref None in - let io_runner = ref None in - let catalog_ready, catalog_resolve = Eio.Promise.create () in - let scope_ready, scope_resolve = Eio.Promise.create () in - let stage_ready, stage_resolve = Eio.Promise.create () in - let path_ready, path_resolve = Eio.Promise.create () in - let stage_completion = ref None in - let cleanup_actions = ref [] in - let post event = - match event with - | Sync.Runner_completed (Sync.Asset_completion (_, Ok (Sync.Asset_staged _))) -> - stage_completion := Some event; - Eio.Promise.resolve stage_resolve () - | Sync.Runner_completed - (Sync.Asset_completion (_, Ok (Sync.Asset_retained (Some (_, path))))) -> - Eio.Promise.resolve path_resolve path; - Db.post (Option.get !db) (Core.Sync_event event) - | event -> Db.post (Option.get !db) (Core.Sync_event event) - in - io_runner := Some (asset_io_runner ~env ~sw ~support ~post ()); - let sync_runner = - Runner.sync_runner ~retire_account:(Sync_runner.retire_account (Option.get !io_runner)) - ~submit:(function - | Sync.Request (ticket, Sync.Fetch_catalog _) -> - Db.post - (Option.get !db) - (Core.Sync_event - (Sync.Runner_completed (Sync.Completion (ticket, Ok [ graph ])))); - Eio.Promise.resolve catalog_resolve () - | Sync.Request (_, Sync.Fetch_snapshot_metadata scope) -> - Eio.Promise.resolve scope_resolve scope - | Sync.Request (_, Sync.Fetch_snapshot_baseline _) - | Sync.Request (_, Sync.Download_snapshot _) -> () - | Sync.Asset_io (_, request) as runnable -> - (match request.action with - | Sync.Release_staged_file _ -> - cleanup_actions := request.action :: !cleanup_actions - | _ -> ()); - Sync_runner.submit (Option.get !io_runner) runnable - | runnable -> Sync_runner.submit (Option.get !io_runner) runnable) - ~shutdown:(fun () -> Sync_runner.shutdown (Option.get !io_runner)) - () - in - let runner_dependencies = - Runner.dependencies - ~runtime: - (Runner.runtime ~fork:(fun ~sw:_ task -> task ()) ~sleep:(fun _ -> ()) - |> Result.get_ok) - ~config:fixture.config - ~overlay:fixture.overlay - ~sync_runner - ~publish:(fun _ -> ()) - |> Result.get_ok - in - let limits = - Sync.limits - ~maximum_response_bytes:1024 - ~maximum_artifact_bytes:1024 - ~submission_batch_size:1 - |> Result.get_ok - in - let sync = - Sync.config ~managed_sync_origin:(Uri.of_string "https://api.logseq.io") ~limits - |> Result.get_ok - in - let worker = - Db.create - ~sw - ~config:(Core.config ~worker:fixture.config ~sync) - ~runner_dependencies - |> Result.get_ok - in - db := Some worker; - Fun.protect - ~finally:(fun () -> Db.shutdown worker) - (fun () -> - Db.post worker (Core.Sync_event (Sync.Sign_in_requested {account_generation=0; presentation_generation=0})); - Db.post - worker - (Core.Sync_event (Sync.Account_authenticated { user_id = Some "user-1" })); - Eio.Promise.await catalog_ready; - Db.post worker (Core.Sync_event (Sync.Graph_selected graph.graph_id)); - let scope = Eio.Promise.await scope_ready in - Db.post - worker - (Core.Sync_event - (Sync.Asset_requested - { scope - ; operation = "db-shutdown:stage" - ; action = - Sync.Stage_asset_file - { operation = source.operation - ; file_type = source.file_type - ; source_file = source.source_file - } - })); - Eio.Promise.await stage_ready; - let file = - match Option.get !stage_completion with - | Sync.Runner_completed - (Sync.Asset_completion (_, Ok (Sync.Asset_staged { file; _ }))) -> file - | _ -> Alcotest.fail "missing real staging completion" - in - Db.post - worker - (Core.Sync_event - (Sync.Asset_requested - { scope - ; operation = "db-shutdown:path" - ; action = Sync.Retain_staged_file file - })); - let path = Eio.Promise.await path_ready in - Alcotest.(check bool) - "actual runner staged file before shutdown" - true - (Sys.file_exists path); - Db.shutdown worker; - Alcotest.(check bool) "actual worker stopped" true (Db.view worker).shutdown; - Db.post worker (Core.Sync_event (Option.get !stage_completion)); - Alcotest.(check bool) - "late resource completion reaches cleanup submit" - true - (List.exists - (function - | Sync.Release_staged_file released -> released = file - | _ -> false) - !cleanup_actions); - Alcotest.(check bool) - "late completion leaves no staged orphan" - false - (Sys.file_exists path); - Eio.Fiber.yield ())))) +let test_recovery_prunes_original_owner_staging () = + with_direct_owner (fun fixture -> + let source = picker_import fixture.support 980 in + let intent = Runner.prepare_import fixture.runner ~context:fixture.context source + ~current:(fun () -> true) |> Result.get_ok in + let durable_path = fixture.staged_path intent.staged_file |> Option.get in + let orphan_source = picker_import fixture.support 981 in + let orphan = match fixture.fixture_asset (Sync.Stage_asset_file + {operation=orphan_source.operation;file_type=orphan_source.file_type;source_file=orphan_source.source_file}) with + | Sync.Asset_staged value -> value.file + | _ -> Alcotest.fail "Expected real staged orphan" in + let orphan_path = fixture.staged_path orphan |> Option.get in + let before = !(fixture.actions) in + let recovered = match fixture.recover_uploads () with + | Ok rows -> rows + | Error message -> Alcotest.fail message in + Alcotest.(check bool) "real Read_uploads returns original durable intent" true (recovered = [intent]); + Alcotest.(check bool) "unpublished durable source survives" true (Sys.file_exists durable_path); + Alcotest.(check bool) "unproven staging is physically pruned" false (Sys.file_exists orphan_path); + Alcotest.(check bool) "recovery never reaches registration fallback" true (before = !(fixture.actions)); + Alcotest.(check bool) "recovery is idempotent without PUT or metadata" true + (fixture.recover_uploads () = Ok [intent])) ;; -(* Pure reducers cannot execute filesystem cleanup or warning-output failures. - Reuse real public Runner staging cleanup with an unwritable stderr capability. *) -let test_upload_checkpoint_io ?(closed_stderr = false) () = +let test_upload_owner_network_classification () = + let module D = Sync_dsl.Core in let module U = Logseq_db_worker_pure_reducer.Asset_upload in let module I = Logseq_db_types.Asset_upload_intent in - let module Store = Logseq_db_storage.Asset_upload_store in - let uuid n = - Logseq_db_types.Graph_types.Uuid.of_string - (Printf.sprintf "00000000-0000-4000-8000-%012d" n) - |> Result.get_ok - in - let admitted, context = admitted_asset_state (uuid 1) in - let sync_state = ref admitted in - let scope = context.Sync.scope in - let intent = - I.prepare - ~replace_reference:None - ~operation_id:(uuid 2) - ~origin:(Uri.to_string scope.account.managed_sync_origin) - ~account:"user-1" - ~graph:(uuid 1) - ~asset:(uuid 3) - ~version: - (Logseq_db_types.Asset_descriptor.version - ~checksum:(String.make 64 'a') - ~file_type:"png" - |> Result.get_ok) - ~title:"Imported image" - ~size:4L - ~staged_file:"fixture.bin" - ~target:(uuid 4) - ~local_mutation:(uuid 5) - ~metadata_mutation:(uuid 6) - |> Result.get_ok - in - let state, instructions = U.step (U.create ~scope ~available:true) (Start intent) in - let active_upload = ref state in - let instruction = List.hd instructions in - T.with_managed (fun fixture -> - Eio_main.run (fun env -> - Eio.Switch.run (fun sw -> - let source_file = - Filename.concat fixture.config.application_support_directory "picker.bin" - in - Out_channel.with_open_bin source_file (fun output -> output_string output "file"); - let published = ref [] in - let posted = ref [] in - let current = ref true in - let runtime = - Runner.runtime - ~fork:(fun ~sw:_ f -> f ()) - ~sleep:(fun _ -> failwith "unexpected wait") - |> Result.get_ok - in - let stage_calls = ref 0 in - let active_runner = ref None in - let source_diagnostic ~scope ~operation ~request_operation detail = - let runner = Option.get !active_runner in - Runner.report_asset_diagnostic runner ~scope ~operation ~request_operation detail; - let wrong = Logseq_db_types.Failure_diagnostic.create ~operation ~reason:"unknown" () in - Runner.report_asset_diagnostic runner ~scope:{scope with graph_id=uuid 90} - ~operation ~request_operation wrong; - Runner.report_asset_diagnostic runner ~scope ~operation - ~request_operation:(request_operation ^ "-unrelated") wrong; - Runner.report_asset_diagnostic runner ~scope ~operation:"workerOperation" - ~request_operation wrong in - let bridge_requests = ref 0 in - let bridge_outputs = ref 0 in - let io_runner = ref None in - let fixture_output = ref None in - let transfer_ticket = ref None in - let submit_sync = ref (fun _ -> failwith "sync interpreter not initialized") in - let post = function - | Core.Sync_event event -> - (match event with - | Sync.Asset_requested _ -> incr bridge_requests - | _ -> ()); - let transition = Sync.step !sync_state event in - sync_state := transition.next; - List.iter - (function - | Sync.Run runner_effect -> !submit_sync runner_effect - | Sync.Publish (Sync.Asset_finished output) -> - incr bridge_outputs; - if String.starts_with ~prefix:"fixture:" output.request.operation - then fixture_output := Some output.result - else - Runner.submit - (Option.get !active_runner) - (Core.Deliver_asset_result output) - | Sync.Publish _ -> () - | Sync.Delegate _ -> Alcotest.fail "unexpected graph work in asset bridge") - transition.effects - | event -> posted := event :: !posted - in - io_runner - := Some - (asset_io_runner - ~on_asset_diagnostic:source_diagnostic - ~token_failure:(Printexc.to_string (Unix.Unix_error (Unix.EACCES,"open","PRIVATE_TOKEN_SOURCE"))) - ~env - ~sw - ~support:fixture.config.application_support_directory - ~post:(fun event -> post (Core.Sync_event event)) ()); - let sync_runner = - Runner.sync_runner ~retire_account:(Sync_runner.retire_account (Option.get !io_runner)) - ~submit:(fun runner_effect -> - (match runner_effect with - | Sync.Asset_io (_, { action = Sync.Stage_asset_file _; _ }) -> - incr stage_calls - | _ -> ()); - Sync_runner.submit (Option.get !io_runner) runner_effect) - ~shutdown:(fun () -> Sync_runner.shutdown (Option.get !io_runner)) - () - in - (submit_sync - := fun runner_effect -> - Runner.submit (Option.get !active_runner) (Core.Run_sync runner_effect)); - let dependencies = - Runner.dependencies - ~runtime - ~config:fixture.config - ~overlay:fixture.overlay - ~sync_runner - ~publish:(fun output -> published := output :: !published) - |> Result.get_ok - in - let runner = - Runner.create - ~sw - dependencies - ~post - ~recovery_current:(fun _ -> false) - ~upload_current:(fun ticket -> !current && U.ticket_current !active_upload ticket) - ~asset_current:(fun ticket -> !transfer_ticket = Some ticket) - |> Result.get_ok - in - active_runner := Some runner; - let fixture_serial = ref 0 in - let fixture_asset action = - incr fixture_serial; - fixture_output := None; - post - (Core.Sync_event - (Sync.Asset_requested - { scope - ; operation = "fixture:" ^ string_of_int !fixture_serial - ; action - })); - Option.get !fixture_output |> Result.get_ok - in - let staged_path file = - match fixture_asset (Sync.Retain_staged_file file) with - | Sync.Asset_retained (Some (lease, path)) -> - ignore (fixture_asset (Sync.Release_asset_file lease)); - path - | _ -> Alcotest.fail "staged resource unavailable" - in - let orphan_file = - match - fixture_asset - (Sync.Stage_asset_file - { operation = uuid 88; file_type = "bin"; source_file }) - with - | Sync.Asset_staged { file; _ } -> file - | _ -> Alcotest.fail "orphan staging failed" - in - let orphan_path = staged_path orphan_file in - stage_calls := 0; - Runner.submit - runner - (Core.Run_upload ({ scope; encrypted = false; key = None }, instruction)); - Alcotest.(check bool) - "durable acknowledgement" - true - (match !posted with - | [ Core.Upload_completed (_, U.Persisted) ] -> true - | _ -> false); - let db = - Sqlite3.db_open - (Filename.concat - fixture.config.application_support_directory - "asset-upload-intents.sqlite") - in - let stored = - Logseq_db_storage.Asset_upload_store.read db ~operation:intent.operation_id - |> Result.get_ok - in - ignore (Sqlite3.db_close db); - Alcotest.(check bool) - "survives independent database reopen" - true - (stored = Some intent); - current := false; - posted := []; - Runner.submit - runner - (Core.Run_upload ({ scope; encrypted = false; key = None }, instruction)); - Alcotest.(check int) "stale ticket performs no completion" 0 (List.length !posted); - let source : Logseq_db_types.Asset_import.t = - { operation = uuid 10 - ; asset = uuid 11 - ; target = uuid 4 - ; local_mutation = uuid 12 - ; replace_reference = None - ; metadata_mutation = uuid 13 - ; source_file - ; title = "Imported image" - ; file_type = "png" - } - in - Alcotest.(check bool) - "stale import does not stage" - true - (Result.is_error - (Runner.prepare_import runner ~context source ~current:(fun () -> false))); - Alcotest.(check int) "stale selection has no IO" 0 !stage_calls; - Alcotest.(check bool) - "stale import leaves orphan untouched" - true - (Sys.file_exists orphan_path); - let prepared = - Runner.prepare_import runner ~context source ~current:(fun () -> true) - |> Result.get_ok - in - let db = - Sqlite3.db_open - (Filename.concat - fixture.config.application_support_directory - "asset-upload-intents.sqlite") - in - Alcotest.(check bool) - "import reconciles orphan before staging" - false - (Sys.file_exists orphan_path); - let durable = - Logseq_db_storage.Asset_upload_store.read db ~operation:source.operation - |> Result.get_ok - in - ignore (Sqlite3.db_close db); - Alcotest.(check bool) - "import is durable before success" - true - (durable = Some prepared && prepared.phase = I.Prepared); - (* Worker import owns publishing the staged original. This exercises its - public prepare operation and the sealed Core/runner bridge. *) - let normal_handle = - match fixture_asset (Sync.Check_asset_cache (prepared.asset, prepared.version)) with - | Sync.Asset_cached (Some handle) -> handle - | _ -> Alcotest.fail "imported original must be in normal cache before upload" - in - let normal_lease, normal_path = - match fixture_asset (Sync.Retain_asset_file normal_handle) with - | Sync.Asset_retained (Some file) -> file - | _ -> Alcotest.fail "normal imported original must be retainable" - in - Alcotest.(check string) "normal cache contains the original" "file" - (In_channel.with_open_bin normal_path In_channel.input_all); - ignore (fixture_asset (Sync.Release_asset_file normal_lease)); - ignore (fixture_asset (Sync.Release_asset_file normal_handle)); - let local_descriptor = Logseq_db_types.Asset_descriptor.create - ~uuid:prepared.asset ~source:(Managed None) - ~current_checksum:(Some prepared.version.checksum) ~size:None ~dimensions:None - |> Result.get_ok in - let module Transfer = Logseq_sync_pure_reducer.Asset_transfer in - let transfer = Transfer.create - (Transfer.config ~active:2 ~foreground_reserved:1 ~pending:3 ~retries:0 |> Result.get_ok) - ~scope ~online:false ~unlocked:false in - let _, instructions = Transfer.step transfer - (Replace { consumer = "local-original"; priority = Foreground; assets = [local_descriptor] }) in - let lookup = List.find_map (function Transfer.Check_cache ticket -> Some ticket | _ -> None) instructions |> Option.get in - transfer_ticket := Some (Transfer.Lookup lookup); - posted := []; - Runner.submit runner (Core.Run_asset (context, Transfer.Check_cache lookup)); - let local_handle = List.find_map (function - | Core.Asset_completed (actual, Transfer.Cache_checked (_, Ok (Some handle))) when actual = scope -> Some handle - | _ -> None) !posted |> Option.get in - ignore (fixture_asset (Sync.Release_asset_file local_handle)); - posted := []; - let before_wrong_scope = !bridge_requests in - Runner.submit runner (Core.Run_asset ({ context with scope = { scope with graph_id = uuid 90 } }, Transfer.Check_cache lookup)); - Alcotest.(check int) "mismatched transfer context performs no asset IO" before_wrong_scope !bridge_requests; - Alcotest.(check bool) "mismatched transfer context has no completion" true (!posted = []); - transfer_ticket := None; - let lease, preview = - match Runner.retain_imported_file runner ~scope ~operation:source.operation with - | Some preview -> preview - | None -> Alcotest.fail "durable import preview unavailable" - in - Alcotest.(check bool) "preview uses staged bytes" true (Sys.file_exists preview); - Alcotest.(check bool) - "another graph cannot acquire import" - true - (Runner.retain_imported_file - runner - ~scope:{ scope with graph_id = uuid 90 } - ~operation:source.operation - = None); - Alcotest.(check bool) - "another account cannot acquire import" - true - (Runner.retain_imported_file - runner - ~scope:{ scope with account = { scope.account with user_id = "other" } } - ~operation:source.operation - = None); - Runner.release_asset_file runner ~scope ~handle:lease; - Alcotest.(check bool) "same operation cannot replace its complete original source" true - (Result.is_error(Runner.prepare_import runner ~context - {source with source_file="picker-no-longer-available"} ~current:(fun()->true))); - let duplicate = - Runner.prepare_import runner ~context source ~current:(fun () -> true) - |> Result.get_ok - in - Alcotest.(check bool) - "same identity restores its intent" - true - (duplicate = prepared); - Alcotest.(check int) - "duplicate import never recopies picker source" - 1 - !stage_calls; - (* A real changed staged file fails in the sync interpreter before PUT; - the Worker must keep that reason while retaining Invalid_content. *) - let uploading = I.advance prepared Local_committed |> Result.get_ok - |> fun i -> I.advance i Uploading |> Result.get_ok in - let upload_state, observe = U.step (U.create ~scope ~available:true) (Restore uploading) in - let observe = match observe with [U.Inspect (ticket, _)] -> ticket - | _ -> Alcotest.fail "upload restore must inspect" in - let upload_state, put = U.step upload_state (Completed (observe, Inspected Local_present)) in - let put = match put with [U.Put _ as instruction] -> instruction - | _ -> Alcotest.fail "upload restore must resume PUT" in - active_upload := upload_state; - current := true; - Runner.submit runner (Core.Publish (Core.Sync_output (Sync.State_changed (Sync.state !sync_state)))); - published := []; posted := []; - let original_path = staged_path prepared.staged_file in - Out_channel.with_open_bin original_path (fun output -> output_string output "changed fixture bytes"); - Fun.protect - ~finally:(fun () -> Out_channel.with_open_bin original_path (fun output -> output_string output "file")) - (fun () -> - Runner.submit runner (Core.Run_upload (context, put)); - Alcotest.(check bool) "PUT failure keeps original result category" true - (List.exists (function Core.Upload_completed (_, U.Failed U.Invalid_content) -> true | _ -> false) !posted); - if not closed_stderr then Alcotest.(check bool) "failure detail waits for its owner notice" false - (List.exists (fun detail -> Logseq_db_types.Failure_diagnostic.operation detail = "assetUpload") - (failure_details !published)); - Runner.submit runner (Core.Publish (Core.Asset_notice (scope, Core.Upload_status - {operation=prepared.operation_id;asset=prepared.asset;target=prepared.target;title=prepared.title; - status=U.Failed_upload U.Invalid_content}))); - if not closed_stderr then Alcotest.(check bool) "PUT failure has a safe scoped reason" true - (List.exists (fun detail -> Logseq_db_types.Failure_diagnostic.operation detail = "assetUpload") - (failure_details !published))); - let put_ticket = match put with U.Put (ticket, _) -> ticket | _ -> assert false in - let retry_state, _ = U.step upload_state (Completed (put_ticket, Failed Invalid_content)) in - let retry_state, retried = U.step retry_state Retry in - let retried = match retried with [U.Put _ as instruction] -> instruction - | _ -> Alcotest.fail "PUT retry must remain PUT" in - active_upload := retry_state; - published := []; posted := []; - Runner.submit runner (Core.Run_upload (context, retried)); - Alcotest.(check bool) "token failure keeps authentication result" true - (List.exists (function Core.Upload_completed (_, U.Failed U.Authentication) -> true | _ -> false) !posted); - Runner.submit runner (Core.Publish (Core.Asset_notice (scope, Core.Upload_status - {operation=prepared.operation_id;asset=prepared.asset;target=prepared.target;title=prepared.title; - status=U.Failed_upload U.Authentication}))); - Alcotest.(check bool) "exact active request keeps safe token source class" true - (List.exists (fun detail -> Logseq_db_types.Failure_diagnostic.operation detail="assetUpload" - && Logseq_db_types.Failure_diagnostic.source_class detail="io-permission") (failure_details !published)); - (* Real transfer policy fans a terminal failure out to every consumer. - Every notice must be followed by the same current safe cause. *) - let remote_asset = uuid 71 in - let remote_version = Logseq_db_types.Asset_descriptor.version - ~checksum:(String.make 64 'b') ~file_type:"png" |> Result.get_ok in - let descriptor = Logseq_db_types.Asset_descriptor.create ~uuid:remote_asset - ~source:(Managed (Some remote_version)) ~current_checksum:None ~size:None ~dimensions:None - |> Result.get_ok in - let transfer = Transfer.create - (Transfer.config ~active:2 ~foreground_reserved:1 ~pending:3 ~retries:0 |> Result.get_ok) - ~scope ~online:true ~unlocked:true in - let transfer, work = Transfer.step transfer (Replace {consumer="first-view";priority=Foreground;assets=[descriptor]}) in - let transfer, _ = Transfer.step transfer (Replace {consumer="second-view";priority=Foreground;assets=[descriptor]}) in - let remote_lookup = List.find_map (function Transfer.Check_cache ticket -> Some ticket | _ -> None) work |> Option.get in - transfer_ticket := Some (Transfer.Lookup remote_lookup); posted := []; - Runner.submit runner (Core.Run_asset (context, Transfer.Check_cache remote_lookup)); - let completion = List.find_map (function Core.Asset_completed (_, event) -> Some event | _ -> None) !posted |> Option.get in - let transfer, work = Transfer.step transfer completion in - let fetch = List.find_map (function Transfer.Fetch ticket -> Some ticket | _ -> None) work |> Option.get in - transfer_ticket := Some (Transfer.Download fetch); posted := []; - Runner.submit runner (Core.Run_asset (context, Transfer.Fetch fetch)); - let completion = List.find_map (function Core.Asset_completed (_, event) -> Some event | _ -> None) !posted |> Option.get in - let _, work = Transfer.step transfer completion in - let failures = List.filter_map (function - | Transfer.Notify {consumer;asset;descriptor;availability=Failed _ as availability} -> - Some (Core.Asset_availability {consumer;asset;descriptor;availability}) - | _ -> None) work in - Alcotest.(check int) "real terminal failure notifies both consumers" 2 (List.length failures); - List.iter (fun notice -> - published := []; - Runner.submit runner (Core.Publish (Core.Asset_notice (scope, notice))); - let latest = match !published with output::_ -> failure_details [output] | [] -> [] in - Alcotest.(check bool) "each failed consumer notice keeps its exact safe cause last" true - (List.exists (fun detail -> Logseq_db_types.Failure_diagnostic.operation detail="assetDownload" - && Logseq_db_types.Failure_diagnostic.source_class detail="io-permission") latest)) failures; - transfer_ticket := None; - active_upload := state; - current := false; - Alcotest.(check bool) - "changed identity payload is rejected" - true - (Result.is_error - (Runner.prepare_import - runner - ~context - { source with asset = uuid 99 } - ~current:(fun () -> true))); - let source_path = staged_path prepared.staged_file in - Sys.remove source_file; - let checkpoint_path = - Filename.concat - fixture.config.application_support_directory - "asset-upload-intents.sqlite" - in - let db = Sqlite3.db_open checkpoint_path in - let complete = - List.fold_left - (fun previous phase -> - let next = I.advance previous phase |> Result.get_ok in - Store.save db ~expected:(Some previous.I.revision) next |> Result.get_ok; - next) - prepared - [ I.Local_committed; Uploading; Remote_stored; Metadata_pending; Complete ] - in - ignore (Sqlite3.db_close db); - let restore_terminal () = - let db = Sqlite3.db_open checkpoint_path in - let restored = - Store.list - db - ~origin:complete.origin - ~account:complete.account - ~graph:complete.graph - ~after:None - ~limit:16 - |> Result.get_ok - in - ignore (Sqlite3.db_close db); - let terminal = - List.find (fun i -> i.I.operation_id = complete.operation_id) restored - in - Alcotest.(check bool) - "terminal checkpoint survives database reopen" - true - (terminal = complete); - let _, instructions = - U.step (U.create ~scope ~available:false) (Restore terminal) - in - match instructions with - | [ (U.Release_staging _ as instruction) ] -> instruction - | _ -> Alcotest.fail "terminal recovery attempted non-cleanup work" - in - let staging_directory = Filename.dirname source_path in - Unix.chmod staging_directory 0o500; - Fun.protect - ~finally:(fun () -> Unix.chmod staging_directory 0o700) - (fun () -> - Runner.submit runner (Core.Publish (Core.Sync_output (Sync.State_changed (Sync.state !sync_state)))); - let terminal_cleanup = restore_terminal () in - let cleanup () = Runner.submit runner (Core.Run_upload (context, terminal_cleanup)) in - if not closed_stderr then cleanup () else ( - flush stderr; - let saved = Unix.dup Unix.stderr in - let read_only = Unix.openfile "/dev/null" [Unix.O_RDONLY] 0 in - Fun.protect ~finally:(fun () -> - Unix.dup2 saved Unix.stderr; Unix.close saved; Unix.close read_only) - (fun () -> Unix.dup2 read_only Unix.stderr; cleanup ())); - Alcotest.(check bool) - "unavailable cache retains pending source" - true - (Sys.file_exists source_path); - Alcotest.(check bool) - "failed cleanup is observable" - true - (List.exists - (function - | Core.Diagnostic _ -> true - | _ -> false) - !published); - let signed_out = Sync.step !sync_state (Sync.Account_authenticated {user_id=None}) in - Runner.submit runner (Core.Publish (Core.Sync_output (Sync.State_changed (Sync.state signed_out.next)))); - published := []; - Runner.submit runner (Core.Run_upload (context, restore_terminal ())); - Alcotest.(check bool) "old scope cleanup cannot create a current diagnostic" false - (List.exists (function Core.Diagnostic _ -> true | _ -> false) !published); - Runner.shutdown runner); - sync_state := admitted; - io_runner - := Some - (asset_io_runner - ~on_asset_diagnostic:(fun ~scope ~operation ~request_operation detail -> - Runner.report_asset_diagnostic (Option.get !active_runner) - ~scope ~operation ~request_operation detail) - ~env - ~sw - ~support:fixture.config.application_support_directory - ~post:(fun event -> post (Core.Sync_event event)) ()); - let restarted = - Runner.create - ~sw - dependencies - ~post - ~recovery_current:(fun _ -> false) - ~upload_current:(fun _ -> false) - ~asset_current:(fun ticket -> !transfer_ticket = Some ticket) - |> Result.get_ok - in - active_runner := Some restarted; - transfer_ticket := Some (Transfer.Lookup lookup); - posted := []; - Runner.submit restarted (Core.Run_asset (context, Transfer.Check_cache lookup)); - let cold_handle = List.find_map (function - | Core.Asset_completed (actual, Transfer.Cache_checked (_, Ok (Some handle))) when actual = scope -> Some handle - | _ -> None) !posted |> Option.get in - ignore (fixture_asset (Sync.Release_asset_file cold_handle)); - transfer_ticket := None; - let restored_import = - Runner.prepare_import restarted ~context source ~current:(fun () -> true) - |> Result.get_ok - in - Alcotest.(check bool) - "reconciliation retains durable terminal staging" - true - (restored_import = complete && Sys.file_exists source_path); - let cleanup = restore_terminal () in - Runner.submit restarted (Core.Run_upload (context, cleanup)); - Alcotest.(check bool) - "restart completes durable terminal cleanup" - false - (Sys.file_exists source_path); - published := []; - Runner.submit restarted (Core.Run_upload (context, restore_terminal ())); - Alcotest.(check int) "repeated cleanup is idempotent" 0 (List.length !published); - Alcotest.(check int) - "every asset request returns through reducer output" - !bridge_requests - !bridge_outputs; - Alcotest.(check bool) "asset bridge exercised" true (!bridge_requests > 0); - (* Real malformed private checkpoint schema: optional preview stays None - while the current graph receives only a safe storage class. *) - let db = Sqlite3.db_open checkpoint_path in - Sqlite3.Rc.check (Sqlite3.exec db - "DROP TABLE asset_upload_intents; CREATE TABLE asset_upload_intents (PRIVATE_COLUMN TEXT)"); - ignore (Sqlite3.db_close db); - Runner.submit restarted (Core.Publish (Core.Sync_output (Sync.State_changed (Sync.state !sync_state)))); - published := []; - Alcotest.(check bool) "bad metadata keeps optional preview unavailable" true - (Runner.retain_imported_file restarted ~scope ~operation:source.operation = None); - Alcotest.(check bool) "bad metadata keeps its storage reason" true - (List.exists (fun detail -> - Logseq_db_types.Failure_diagnostic.operation detail = "assetMetadata" - && Logseq_db_types.Failure_diagnostic.source_class detail = "sqlite-schema") - (failure_details !published)); - List.iter (fun detail -> - Alcotest.(check bool) "metadata diagnostic excludes raw column/path" false - (let encoded = Logseq_db_types.Failure_diagnostic.encode detail in - String.contains encoded '/' || String.contains encoded 'P')) (failure_details !published); - Runner.shutdown restarted))) -;; - -let deletion_runtime_fixture ?deletion_case ?(journal_failure = false) - ?(confirmation_failure = false) ~busy () = - let module Db = Logseq_db_worker in - let module G = Logseq_db_types.Graph_types in - T.with_managed (fun fixture -> - Eio_main.run (fun env -> Eio.Switch.run (fun sw -> - let graph_id suffix = G.Uuid.of_string ("94000000-0000-4000-8000-" ^ suffix) |> Result.get_ok in - let first = graph_id "000000000201" and second = graph_id "000000000202" in - let graph graph_id : Sync.graph = - { graph_id; name = "Deletion waiter fixture"; encrypted = false - ; schema = { major = 65; minor = 33; exact = true } } in - let root = fixture.config.application_support_directory in - let worker_root = Filename.concat root "logseq-db-worker" in - Unix.mkdir worker_root 0o700; - let mirrors = Filename.concat worker_root "synced-graphs" in - Unix.mkdir mirrors 0o700; - let directory = Filename.concat mirrors (G.Uuid.to_string first) in - Unix.mkdir directory 0o700; - let first_database_path = Filename.concat directory "db.sqlite" in - let sqlite = Sqlite3.db_open first_database_path in - let checkpoint = Logseq_db_types.Sync_checkpoint.create ~graph_id:first - ~schema:{ major = 65; minor = 33 } - ~applied_server_t:(Logseq_db_types.Server_cursor.of_int64 0L |> Result.get_ok) - ~checksum:"0123456789abcdef" |> Result.get_ok in - Logseq_db_storage.Sync_checkpoint_store.initialize_database sqlite checkpoint |> Result.get_ok; - ignore (Sqlite3.db_close sqlite); - (match deletion_case with - | None | Some `Legacy -> () - | Some `Corrupt_sqlite -> - Out_channel.with_open_bin first_database_path (fun output -> output_string output "not a SQLite database") - | Some `Corrupt_checkpoint -> - let db = Sqlite3.db_open first_database_path in - Sqlite3.Rc.check (Sqlite3.exec db "DROP TABLE sync_meta; CREATE TABLE sync_meta (broken TEXT)"); - ignore (Sqlite3.db_close db) - | Some `Missing_database -> Unix.unlink first_database_path - | Some `Linked_database -> - let external_database = Filename.concat root "unrelated-database.sqlite" in - Unix.rename first_database_path external_database; - Unix.symlink external_database first_database_path); - let second_directory = Filename.concat mirrors (G.Uuid.to_string second) in - Unix.mkdir second_directory 0o700; - let sqlite = Sqlite3.db_open (Filename.concat second_directory "db.sqlite") in - let open Yojson.Safe.Util in - let storage = T.read_json (T.fixture "storage/logseq-65.33-create-page.json") in - Sqlite3.Rc.check (Sqlite3.exec sqlite (storage |> member "tableSql" |> to_string)); - let statement = Sqlite3.prepare sqlite "INSERT INTO kvs(addr, content, addresses) VALUES(?, ?, ?)" in - storage |> member "rows" |> to_list |> List.iter (fun row -> - Sqlite3.Rc.check (Sqlite3.reset statement); - Sqlite3.Rc.check (Sqlite3.bind_int statement 1 (row |> member "addr" |> to_int)); - Sqlite3.Rc.check (Sqlite3.bind_text statement 2 (row |> member "content" |> to_string)); - Sqlite3.Rc.check (Sqlite3.bind statement 3 (match member "addresses" row with - | `Null -> Sqlite3.Data.NULL | `String value -> TEXT value | _ -> assert false)); - Sqlite3.Rc.check (Sqlite3.step statement)); - Sqlite3.Rc.check (Sqlite3.finalize statement); - let checkpoint = Logseq_db_types.Sync_checkpoint.create ~graph_id:second - ~schema:{ major = 65; minor = 33 } - ~applied_server_t:(Logseq_db_types.Server_cursor.of_int64 0L |> Result.get_ok) - ~checksum:"0123456789abcdef" |> Result.get_ok in - Logseq_db_storage.Sync_checkpoint_store.initialize_database sqlite checkpoint |> Result.get_ok; - Logseq_db_storage.Sync_outbox_store.initialize_database sqlite |> Result.get_ok; - ignore (Sqlite3.db_close sqlite); - let worker = ref None and refreshed = ref false and held_cleanup = ref None in - let destructive_requests = ref 0 in - let published = ref [] and blocked_states = ref None in - let post event = Db.post (Option.get !worker) (Core.Sync_event event) in - let sync_runner = Runner.sync_runner ~retire_account:(fun ~account:_->Alcotest.fail "Unexpected account retirement in fixture") ~submit:(function - | Sync.Request (ticket, Sync.Load_catalog _) -> - post (Sync.Runner_completed (Sync.Completion (ticket, Ok None))) - | Sync.Request (ticket, Sync.Save_catalog _) -> - post (Sync.Runner_completed (Sync.Completion (ticket, Ok ()))) - | Sync.Request (ticket, Sync.Fetch_catalog _) -> - post (Sync.Runner_completed (Sync.Completion - (ticket, Ok (if !refreshed then [graph first] else [graph first; graph second])))) - | Sync.Request (ticket, Sync.Fetch_snapshot_baseline _) -> - post (Sync.Runner_completed (Sync.Completion - (ticket, Error (Sync.Effect_failed "Synthetic bootstrap unavailable.")))) - | Sync.Request (ticket, Sync.Fetch_snapshot_metadata _) -> - post (Sync.Runner_completed (Sync.Completion - (ticket, Error (Sync.Effect_failed "Synthetic bootstrap unavailable.")))) - | Sync.Asset_io (ticket, { context; action = Sync.Prepare_graph_asset_deletion retirement; _ }) -> - incr destructive_requests; - let deletion = Sync.asset_deletion_reference ~scope:context.scope ~retirement in - if confirmation_failure then ( - (* Preparation has already written the uncommitted receipt. Keep - it readable so the canonical rename can commit, but prevent - the receipt rewrite that confirms durability afterwards. *) - let states = Filename.concat mirrors - (Filename.concat ".mirror-deletions" - (Filename.concat (G.Uuid.to_string first) "states")) in - Unix.chmod states 0o500; - blocked_states := Some states; - post (Sync.Runner_completed (Sync.Asset_completion - (ticket, Ok (Sync.Asset_deletion_prepared deletion))))) - else if busy || Option.is_some deletion_case then post (Sync.Runner_completed (Sync.Asset_completion (ticket, Ok (Sync.Asset_deletion_prepared deletion)))) - else held_cleanup := Some (ticket, deletion) - | Sync.Asset_io (ticket, _) -> - post (Sync.Runner_completed (Sync.Asset_completion (ticket, Ok Sync.Asset_unit))) - | _ -> ()) ~shutdown:(fun () -> ()) () in - let dependencies = Runner.dependencies - ~runtime:(Runner.runtime ~fork:(fun ~sw task -> Eio.Fiber.fork ~sw task) - ~sleep:(Eio.Time.sleep (Eio.Stdenv.clock env)) |> Result.get_ok) - ~config:fixture.config ~overlay:fixture.overlay ~sync_runner - ~publish:(fun output -> published := output :: !published) - |> Result.get_ok in - let limits = Sync.limits ~maximum_response_bytes:1024 ~maximum_artifact_bytes:1024 - ~submission_batch_size:1 |> Result.get_ok in - let sync = Sync.config ~managed_sync_origin:(Uri.of_string "https://api.logseq.io") ~limits - |> Result.get_ok in - let db = Db.create ~sw ~config:(Core.config ~worker:fixture.config ~sync) - ~runner_dependencies:dependencies |> Result.get_ok in - worker := Some db; - let wait predicate = - match Eio.Time.with_timeout_exn (Eio.Stdenv.clock env) 3. (fun () -> - while not (predicate ()) do Eio.Fiber.yield () done) with - | () -> () - | exception error -> - let snapshot = (Db.view db).sync.snapshot in - Alcotest.failf "Wait failed: %s selected=%s bootstrap=%b restore=%b failure=%b error=%s" - (Printexc.to_string error) - (Option.fold ~none:"none" ~some:G.Uuid.to_string snapshot.selected_graph) - snapshot.startup.bootstrapping snapshot.startup.restoring_local - (Option.is_some snapshot.startup.failure) - (Option.value snapshot.last_error ~default:"none") in - Fun.protect ~finally:(fun () -> - Option.iter (fun states -> Unix.chmod states 0o700) !blocked_states; - Option.iter (fun (ticket, _) -> post (Sync.Runner_completed - (Sync.Asset_completion (ticket, Error Sync.Asset_cancelled)))) !held_cleanup; - Db.shutdown db) - (fun () -> - post (Sync.Sign_in_requested {account_generation=0; presentation_generation=0}); - post (Sync.Account_authenticated { user_id = Some "user-1" }); - wait (fun () -> (Db.view db).sync.snapshot.catalog <> []); - post (Sync.Graph_selected second); - wait (fun () -> let snapshot = (Db.view db).sync.snapshot in - snapshot.selected_graph = Some second && (Db.graph_state db).phase = Db.Graph_open - && not snapshot.startup.bootstrapping && not snapshot.startup.restoring_local); - let generation = (Db.view db).sync.snapshot.startup.account_generation in - if confirmation_failure then ( - published := []; - ignore (Db.delete_local_copy db ~account_generation:generation ~graph_id:first); - Alcotest.(check bool) "canonical deletion reached the caught confirmation fault" false - (Sys.file_exists directory); - Alcotest.(check bool) "current other graph stays open" true - ((Db.graph_state db).phase = Db.Graph_open); - let states = Option.get !blocked_states in - let receipts = Sys.readdir states |> Array.to_list in - Alcotest.(check bool) "unconfirmed receipt remains replayable" true - (List.exists (fun name -> Filename.check_suffix name ".json" - && not (T.read_json (Filename.concat states name) - |> Yojson.Safe.Util.member "committed" |> Yojson.Safe.Util.to_bool)) receipts); - Alcotest.(check bool) "caught current confirmation failure publishes Diagnostic" true - (List.exists (function - | Core.Diagnostic "worker.confirmMirrorRetirement" -> true - | _ -> false) !published)) - else if Option.is_some deletion_case then ( - let result = Db.delete_local_copy db ~account_generation:generation ~graph_id:first in - (match result with Ok () -> () | Error message -> Alcotest.failf "unopened local copy must delete: %s" message); - Alcotest.(check bool) "selected target directory is absent" false (Sys.file_exists directory); - Alcotest.(check bool) "other graph database survives" true - (Sys.file_exists (Filename.concat second_directory "db.sqlite")); - Alcotest.(check bool) "current Worker remains open" true ((Db.graph_state db).phase = Db.Graph_open); - Alcotest.(check bool) "opening current graph creates no account-owner JSON" false - (Sys.file_exists (Filename.concat second_directory ".journal-local-owner.json")); - if deletion_case = Some `Linked_database then - Alcotest.(check bool) "linked external database survives deletion" true - (Sys.file_exists (Filename.concat root "unrelated-database.sqlite"))) - else if journal_failure then ( - let blocked = Filename.concat worker_root "local-copy-retirements" in - Out_channel.with_open_bin blocked (fun output -> output_string output "synthetic mkdir fault"); - Alcotest.(check bool) "journal IO failure returns a deletion error" true - (Result.is_error (Db.delete_local_copy db ~account_generation:generation ~graph_id:first)); - Alcotest.(check int) "journal failure never retires assets" 0 !destructive_requests; - Alcotest.(check bool) "journal failure preserves target mirror" true (Sys.file_exists first_database_path); - Alcotest.(check bool) "current graph Worker stays open" true ((Db.graph_state db).phase = Db.Graph_open); - Alcotest.(check bool) "journal IO failure has a safe deletion reason" true - (has_failure_reason !published "deleteMirror" "deleteFailed"); - post (Sync.Catalog_refresh_requested - (let f=(Db.view db).sync.snapshot.startup in - {account_generation=f.account_generation;presentation_generation=f.presentation_generation})); - wait (fun () -> (Db.view db).sync.snapshot.catalog <> [])) - else if busy then ( - let lock = Sqlite3.db_open (Filename.concat directory ".logseq-overlay-owner.sqlite") in - Fun.protect ~finally:(fun () -> ignore (Sqlite3.exec lock "ROLLBACK"); ignore (Sqlite3.db_close lock)) (fun () -> - Sqlite3.Rc.check (Sqlite3.exec lock "BEGIN IMMEDIATE"); - Alcotest.(check bool) "busy mirror reports failure" true - (Result.is_error (Db.delete_local_copy db ~account_generation:generation ~graph_id:first)); - Alcotest.(check int) "busy mirror never starts destructive asset work" 0 !destructive_requests; - Alcotest.(check bool) "busy mirror retains safe ownership reason" true - (has_failure_reason !published "deleteMirror" "busy"); - Alcotest.(check bool) "busy mirror remains usable" true (Sys.file_exists first_database_path))) - else ( - let result, resolve = Eio.Promise.create () in - Eio.Fiber.fork ~sw (fun () -> Eio.Promise.resolve resolve - (Db.delete_local_copy db ~account_generation:generation ~graph_id:first)); - wait (fun () -> Option.is_some !held_cleanup); - refreshed := true; - post (Sync.Catalog_refresh_requested - (let f=(Db.view db).sync.snapshot.startup in - {account_generation=f.account_generation;presentation_generation=f.presentation_generation})); - wait (fun () -> (Db.view db).sync.snapshot.selected_graph = None); - Alcotest.(check bool) "catalog refresh cannot report deletion success" true - (match Eio.Promise.peek result with Some (Ok ()) -> false | _ -> true); - Option.iter (fun (ticket, deletion) -> post (Sync.Runner_completed - (Sync.Asset_completion (ticket, Ok (Sync.Asset_deletion_prepared deletion))))) !held_cleanup; - held_cleanup := None; - Alcotest.(check bool) "matching deletion finishes with cancellation failure" true - (match Eio.Promise.await result with Error _ -> true | Ok () -> false); - Alcotest.(check bool) "unauthorized deletion preserves its mirror" true - (Sys.file_exists first_database_path)))))) -;; - -let test_catalog_refresh_cannot_complete_another_deletion () = deletion_runtime_fixture ~busy:false () -let test_deletion_checks_ownership_before_assets () = deletion_runtime_fixture ~busy:true () -let test_deletion_journal_failure_preserves_worker () = deletion_runtime_fixture ~journal_failure:true ~busy:false () -let test_deletion_confirmation_failure_reports () = - deletion_runtime_fixture ~confirmation_failure:true ~busy:false () - -(* Producer mapping lives in the runner. Real cursor/format failures reach it - through the public reducer and database; no failed external result is injected. *) -let authoritative_detail_cases = - [ "empty", "[]", [ "stage=prepare"; "reason=empty-transaction"; "transaction=1" ] - ; "top-level", "{}", [ "stage=prepare"; "reason=transaction-shape"; "type=map" ] - ; "syntax", "[", [ "stage=prepare"; "reason=transit-syntax" ] - ; "entry", "[42]", [ "reason=operation-shape"; "operationIndex=1"; "type=integer" ] - ; ( "keyword" - , "[[\"PRIVATE_SYNTHETIC_PAYLOAD\",99999]]" - , [ "reason=operation-keyword"; "type=string" ] ) - ; ( "attribute" - , "[[\"~:db/add\",99999,\"PRIVATE_SYNTHETIC_PAYLOAD\",1]]" - , [ "reason=operation-fields"; "operation=add"; "arity=4" ] ) - ; ( "protected" - , "[[\"~:db/add\",99999,\"~:block/title\",false]]" - , [ "reason=protected-value-type"; "attribute=block/title"; "type=bool" ] ) - ; ( "extra-fields" - , "[[\"~:db/add\",99999,\"~:block/PRIVATE_SYNTHETIC_PAYLOAD\",1,2,3]]" - , [ "stage=decode"; "reason=operation-fields"; "arity=6"; "attribute=custom" ] ) - ; ( "source-tx" - , "[[\"~:db/add\",99999,\"~:block/foo\",1,\"PRIVATE_SYNTHETIC_PAYLOAD\"]]" - , [ "reason=source-tx-type"; "field=source-tx"; "type=string" ] ) - ; ( "entity-range" - , "[[\"~:db/add\",\"~i4611686018427387904\",\"~:block/foo\",1]]" - , [ "reason=integer-range"; "field=entity"; "type=int64" ] ) - ; ( "big-integer" - , "[[\"~:db/add\",99999,\"~:block/foo\",\"~n9223372036854775808\"]]" - , [ "reason=integer-value"; "type=big-integer" ] ) - ; ( "reference" - , "[[\"~:db/add\",[\"~:block/uuid\",\"~u22222222-2222-4222-8222-222222222222\"],\"~:block/parent\",null]]" - , [ "reason=reference-shape"; "field=value"; "type=null" ] ) - ; ( "missing-lookup" - , "[[\"~:db/add\",[\"~:block/uuid\",\"~u99999999-9999-4999-8999-999999999999\"],\"~:block/foo\",42]]" - , [ "stage=application" - ; "reason=application-exception" - ; "exception=invalid-argument" - ; "lookupRefs=1" - ; "operationCount=1" - ] ) - ; ( "application-nil" - , "[[\"~:db/add\",99999,\"~:block/PRIVATE_SYNTHETIC_PAYLOAD\",null]]" - , [ "stage=application" - ; "reason=application-exception" - ; "exception=invalid-argument" - ; "operationCount=1" - ] ) - ; ( "cas-expected" - , "[[\"~:db.fn/cas\",99999,\"~:block/title\",false,true]]" - , [ "stage=prepare" - ; "reason=protected-value-type" - ; "field=expected" - ; "operation=cas" - ] ) - ; ( "cas-value" - , "[[\"~:db.fn/cas\",99999,\"~:block/title\",null,false]]" - , [ "stage=prepare"; "reason=protected-value-type"; "field=value" ] ) - ; ( "second-transaction" - , "[]" - , [ "stage=prepare"; "reason=empty-transaction"; "transaction=2" ] ) - ; ( "second-operation" - , "[[\"~:db/retractEntity\",99998],[\"~:PRIVATE_SYNTHETIC_PAYLOAD\",99999]]" - , [ "transaction=1" - ; "operationIndex=2" - ; "operation=other" - ; "reason=unsupported-operation" - ] ) - ; ( "application-bounded-stats" - , "[[\"~:db/add\",99999,\"~:block/foo\",null],[\"~:db/add\",99999,\"~:block/bar\",[" - ^ String.concat "," (List.init 3000 (fun _ -> "0")) - ^ "]]]" - , [ "stage=application" - ; "reason=application-exception" - ; "operationCount=2" - ; "!lookupRefs=" - ] ) - ] -;; - -let test_authoritative_rejection_diagnostics kind () = - let module G = Logseq_db_types.Graph_types in - let module Error = Logseq_db_worker.Error in - T.with_managed (fun fixture -> - Eio_main.run (fun _ -> - Eio.Switch.run (fun sw -> - let graph_id = - G.Uuid.of_string "94000000-0000-4000-8000-000000000707" |> Result.get_ok - in - let graph : Sync.graph = - { graph_id - ; name = "Synthetic authoritative diagnostics" - ; encrypted = false - ; schema = { major = 65; minor = 33; exact = true } - } - in - let root = fixture.config.application_support_directory in - let worker_root = Filename.concat root "logseq-db-worker" in - Unix.mkdir worker_root 0o700; - let mirrors = Filename.concat worker_root "synced-graphs" in - Unix.mkdir mirrors 0o700; - let directory = Filename.concat mirrors (G.Uuid.to_string graph_id) in - Unix.mkdir directory 0o700; - let sqlite = Sqlite3.db_open (Filename.concat directory "db.sqlite") in - let open Yojson.Safe.Util in - let storage = T.read_json (T.fixture "storage/logseq-65.33-create-page.json") in - Sqlite3.Rc.check (Sqlite3.exec sqlite (storage |> member "tableSql" |> to_string)); - let stmt = - Sqlite3.prepare sqlite "INSERT INTO kvs(addr,content,addresses) VALUES(?,?,?)" - in - storage - |> member "rows" - |> to_list - |> List.iter (fun row -> - Sqlite3.Rc.check (Sqlite3.reset stmt); - Sqlite3.Rc.check (Sqlite3.bind_int stmt 1 (row |> member "addr" |> to_int)); - Sqlite3.Rc.check - (Sqlite3.bind_text stmt 2 (row |> member "content" |> to_string)); - Sqlite3.Rc.check - (Sqlite3.bind - stmt - 3 - (match member "addresses" row with - | `Null -> Sqlite3.Data.NULL - | `String value -> TEXT value - | _ -> assert false)); - Sqlite3.Rc.check (Sqlite3.step stmt)); - Sqlite3.Rc.check (Sqlite3.finalize stmt); - let cursor n = Logseq_db_types.Server_cursor.of_int64 n |> Result.get_ok in - let checkpoint = - Logseq_db_types.Sync_checkpoint.create - ~graph_id - ~schema:{ major = 65; minor = 33 } - ~applied_server_t:(cursor 0L) - ~checksum:"0123456789abcdef" - |> Result.get_ok - in - Logseq_db_storage.Sync_checkpoint_store.initialize_database sqlite checkpoint - |> Result.get_ok; - Logseq_db_storage.Sync_outbox_store.initialize_database sqlite |> Result.get_ok; - ignore (Sqlite3.db_close sqlite); - (match kind with - | `Integrity -> - let module Storage = Logseq_db_storage.Logseq_sqlite_storage in - let module Session = Logseq_db_storage.Storage_session in - let storage = - Storage.open_database (Filename.concat directory "db.sqlite") - |> Result.get_ok - in - let db = Storage.restore_database storage |> Result.get_ok in - let session = - Session.create - ~tail: - (Datascript.Storage.restore_tail_groups - (Storage.datascript_storage storage)) - ~callbacks:(Storage.connection_callbacks storage) - in - let block = Datascript.Temp_id "diagnostic-synthetic-block" in - let page = - Datascript.Lookup_ref - ("block/uuid", Datascript.Uuid "11111111-1111-4111-8111-111111111111") - in - let staged = - Session.stage_transact - session - ~authoritative_before:db - [ Datascript.Add - (block, "block/uuid", Uuid "22222222-2222-4222-8222-222222222224") - ; Add (block, "block/title", String "Synthetic initial block") - ; Add (block, "block/parent", Ref_to page) - ; Add (block, "block/page", Ref_to page) - ; Add (block, "block/order", String "a00000001") - ; Add (block, "block/created-at", Int64 1700000000000L) - ; Add (block, "block/updated-at", Int64 1700000000000L) - ] - |> Result.get_ok - in - Session.commit_staged session staged |> Result.get_ok; - Session.close session |> Result.get_ok; - let module O = Logseq_overlay_db.Types in - let module DB = Logseq_overlay_db.Database in - let inspection = - DB.inspect_mirror ~application_support_directory:root ~graph_id - |> Result.get_ok - in - let database = - DB.open_ ~sw fixture.overlay inspection ~graph_name:graph.name - |> Result.get_ok - in - Fun.protect - ~finally:(fun () -> ignore (DB.close database)) - (fun () -> - let block = - G.Uuid.of_string "22222222-2222-4222-8222-222222222224" |> Result.get_ok - in - let mutation_id = - G.Uuid.of_string "94000000-0000-4000-8000-000000000709" |> Result.get_ok - in - let snapshot = DB.current_snapshot database |> Result.get_ok in - let revision = - match DB.get_outline_blocks snapshot [ block ] |> Result.get_ok with - | [ O.Present_block { revision; _ } ] -> revision - | _ -> Alcotest.fail "synthetic block missing" - in - DB.release_snapshot snapshot; - let expected = - DB.write_precondition ~blocks:[ block, revision ] ~pages:[] ~scopes:[] - |> Result.get_ok - in - ignore - (DB.commit_local - database - ~expected - (O.Save_block - { mutation_id; block; title = "PRIVATE_SYNTHETIC_PAYLOAD" }) - |> Result.get_ok); - let sync = DB.inspect_sync database |> Result.get_ok in - let prepared, crypto = - DB.begin_outbox_transition - database - ~expected:(O.sync_view_token sync) - (O.Submit_group [ mutation_id ]) - |> Result.get_ok - in - let crypto = Option.get crypto in - let encrypted = - DB.protection_plaintexts crypto - |> List.map (fun (id, plain) -> id, "synthetic:" ^ plain) - in - let commit = - DB.apply_outbox_transition - database - prepared - ~encrypted:(Some (crypto, encrypted)) - |> Result.get_ok - in - let batch = Option.get commit.submission_batch in - let sync = DB.inspect_sync database |> Result.get_ok in - let barrier = - O. - { through = cursor 1L - ; checksum = - Checksum.of_string "checksum:v1:0123456789abcdef" |> Result.get_ok - } - in - let prepared, _ = - DB.begin_outbox_transition - database - ~expected:(O.sync_view_token sync) - (O.Accept_group { batch_id = O.submission_batch_id batch; barrier }) - |> Result.get_ok - in - ignore - (DB.apply_outbox_transition database prepared ~encrypted:None - |> Result.get_ok)) - | _ -> ()); - let limits = - Sync.limits - ~maximum_response_bytes: - (match kind with - | `Detail ("application-bounded-stats", _, _) -> 16384 - | _ -> 1024) - ~maximum_artifact_bytes:1048576 - ~submission_batch_size:32 - |> Result.get_ok - in - let sync = - Sync.config ~managed_sync_origin:(Uri.of_string "https://api.logseq.io") ~limits - |> Result.get_ok - in - let state = - ref (Core.initial (Core.config ~worker:fixture.config ~sync) |> Result.get_ok) - in - let events = Queue.create () - and emitted = ref [] - and failures = ref [] - and connection = ref None in - let post event = Queue.add event events in - let sync_runner = - Runner.sync_runner ~retire_account:(fun ~account:_->Alcotest.fail "Unexpected account retirement in fixture") - ~shutdown:(fun () -> ()) - ~submit:(function - | Sync.Request (ticket, Sync.Fetch_catalog _) -> - post - (Core.Sync_event - (Sync.Runner_completed (Sync.Completion (ticket, Ok [ graph ])))) - | Sync.Request (ticket, Sync.Save_catalog _) -> - post - (Core.Sync_event - (Sync.Runner_completed (Sync.Completion (ticket, Ok ())))) - | Sync.Start_websocket request -> connection := Some request.scope - | _ -> ()) - () - in - let dependencies = - Runner.dependencies - ~runtime: - (Runner.runtime ~fork:(fun ~sw:_ task -> task ()) ~sleep:(fun _ -> ()) - |> Result.get_ok) - ~config:fixture.config - ~overlay:fixture.overlay - ~sync_runner - ~publish:(fun output -> emitted := output :: !emitted) - |> Result.get_ok - in - let runner = - Runner.create - ~sw - dependencies - ~post - ~recovery_current:(fun _ -> false) - ~upload_current:(fun _ -> false) - ~asset_current:(fun _ -> false) - |> Result.get_ok - in - let drive event = - post event; - while not (Queue.is_empty events) do - let event = Queue.take events in - (match event with - | Core.Runner_completed (Sync_worker_effect_completed (_, Error error)) -> - failures := error :: !failures - | _ -> ()); - let transition = Core.step !state event in - state := transition.next; - List.iter - (fun instruction -> - (match instruction with - | Core.Publish (Core.Reply _ as reply) -> emitted := reply :: !emitted - | _ -> ()); - Runner.submit runner instruction) - transition.effects - done - in - Fun.protect - ~finally:(fun () -> Runner.shutdown runner) - (fun () -> - drive - (Core.Sync_event - (Sync.Sign_in_requested - { account_generation = 0; presentation_generation = 0 })); - drive - (Core.Sync_event - (Sync.Account_authenticated { user_id = Some "synthetic-user" })); - drive (Core.Sync_event (Sync.Graph_selected graph_id)); - Alcotest.(check bool) - "actual mirror opens" - true - ((Core.view !state).graph.phase = Graph_open); - let connection = Option.get !connection in - drive (Core.Sync_event (Sync.Websocket_opened connection)); - let t, payload, code, category, message, checks = - match kind with - | `Integrity -> - ( 1L - , "[]" - , Error.Ambiguous_sync_state - , "authoritativeIntegrityFailure" - , None - , [ "reason=payload-mismatch" - ; "checkpoint=0" - ; "through=1" - ; "expectedCursor=1" - ; "tBefore=0" - ; "ordinal=0" - ; "count=1" - ; "outboxState=accepted" - ; "mutation=save-block" - ; "expectedDatoms=" - ; "actualDatoms=" - ] ) - | `Cursor -> - ( 2L - , "[[\"~:db/retractEntity\",99999]]" - , Error.Ambiguous_sync_state - , "authoritativeCursorDiscontinuous" - , Some "Authoritative transaction cursors are not continuous." - , [] ) - | `Format -> - ( 1L - , "[[\"~:PRIVATE_SYNTHETIC_PAYLOAD\",99999]]" - , Error.Unsupported_semantics - , "authoritativeDecodeFailed" - , None - , [ "stage=prepare"; "reason=unsupported-operation"; "operation=other" ] - ) - | `Apply_format -> - ( 1L - , "[[\"~:db/add\",null,\"~:block/foo\",\"PRIVATE_SYNTHETIC_PAYLOAD\"]]" - , Error.Unsupported_semantics - , "authoritativeDecodeFailed" - , None - , [ "stage=decode"; "reason=entity-shape"; "field=entity"; "type=null" ] - ) - | `Detail (name, payload, checks) -> - ( (if name = "second-transaction" then 2L else 1L) - , payload - , Error.Unsupported_semantics - , "authoritativeDecodeFailed" - , None - , checks ) - in - let tx : Logseq_sync_pure_reducer.Sync_protocol.Server.pull_transaction = - { t = cursor t; tx = payload; outliner_op = None } - in - let txs = - match kind with - | `Detail ("second-transaction", _, _) -> - [ { tx with t = cursor 1L; tx = "[[\"~:db/retractEntity\",99998]]" } - ; tx - ] - | _ -> [ tx ] - in - drive - (Core.Sync_event - (Sync.Websocket_message - (connection, Pull_ok { t = cursor t; checksum = None; txs }))); - let empty_success = - match kind with - | `Detail (("empty" | "second-transaction"), _, _) -> true - | _ -> false - in - if empty_success - then ( - Alcotest.(check bool) - "legal empty authoritative batch succeeds" - true - (!failures = []); - Alcotest.(check bool) - "empty cursor reaches actual manager" - true - ((Core.view !state).sync.snapshot.applied_server_t = Some (cursor t)); - Alcotest.(check (option string)) - "successful empty pull has no Last error" - None - (Core.view !state).sync.snapshot.last_error; - Alcotest.(check bool) - "successful empty pull keeps graph open" - true - ((Core.view !state).graph.phase = Graph_open)) - else ( - let error = - match !failures with - | [ error ] -> error - | _ -> Alcotest.fail "expected actual authoritative failure" - in - Alcotest.(check bool) - "failure code reflects actual category" - true - (Error.code error = code); - let message_text = Error.message error in - let includes text needle = - let rec loop i = - i + String.length needle <= String.length text - && (String.sub text i (String.length needle) = needle || loop (i + 1)) - in - loop 0 - in - Option.iter - (fun expected -> - Alcotest.(check string) - "producer retains a fixed safe reason" - expected - message_text) - message; - List.iter - (fun expected -> - let absent = String.starts_with ~prefix:"!" expected in - let part = - if absent - then String.sub expected 1 (String.length expected - 1) - else expected - in - Alcotest.(check bool) - ("safe detail " ^ expected) - (not absent) - (includes message_text part)) - checks; - Alcotest.(check (option string)) - "typed origin retains category" - (Some category) - (Error.trace error).origin.code; - Alcotest.(check (option string)) - "actual Last error retains category" - (Some message_text) - (Core.view !state).sync.snapshot.last_error; - Alcotest.(check bool) - "actual Worker diagnostic retains category" - true - (List.exists - (function - | Core.Diagnostic signal -> - String.starts_with - ~prefix:("worker.applyAuthoritative." ^ category) - signal - | _ -> false) - !emitted); - if checks <> [] - then - Alcotest.(check bool) - "validated detail travels in actual diagnostic signal" - true - (List.exists - (function - | Core.Diagnostic signal -> - includes - signal - (match kind with - | `Integrity -> "authoritative-integrity:v1:" - | _ -> "authoritative-decode:v1:") - | _ -> false) - !emitted); - Alcotest.(check bool) - "failure precedes checkpoint advancement" - true - ((Core.view !state).sync.snapshot.applied_server_t = Some (cursor 0L)); - (match kind with - | `Integrity -> - let module Read=Worker_pure_read in - let module F=Read.For_worker in - let scope=List.hd(Core.graph_scopes !state) in - let binding=F.binding ~serial:1L ~incarnation:(Read.worker_incarnation - ~runtime_epoch:(Journal_worker_ids.Runtime.Epoch.of_int64 1L) - ~worker_generation:(Journal_worker_ids.Worker.Generation.of_int64 1L)) - ~origin:(Uri.to_string scope.account.managed_sync_origin) ~user:scope.account.user_id in - let owner:Sync.timeline_scope={startup={account_generation=scope.account.account_generation; - presentation_generation=scope.account.presentation_generation};graph_id=scope.graph_id;graph_generation=scope.graph_generation}in - let limits:Worker_pure_data.capability_limits={response_budget_bytes=fixture.config.response_budget_bytes; - outbox_max_records=4096;outbox_max_bytes=8388608;change_max_items=4096;change_max_bytes=1048576; - dispatcher_capacity=256;wire_batch_max_bytes=262144}in - let session=F.session ~binding ~graph:scope.graph_id ~account:scope.account.account_generation - ~lifecycle:scope.account.lifecycle_generation ~generation:scope.graph_generation ~scope:owner ~asset_scope:scope ~limits in - let budget=Read.read_budget session ~maximum_items:1 ~maximum_bytes:limits.response_budget_bytes|>Result.get_ok in - drive(Core.Graph_request_for_scope{scope;id=Core.request_id_of_int64 3L; - read=F.pack_query budget Read.Admission}); - let active = - List.find_map - (function - |Core.Reply(_,Read.Query_answer(_,Read.Admission,Ok inspection))->Some inspection.active_records - | _ -> None) - !emitted - |> Option.get - in - Alcotest.(check int) "failed validation retains durable outbox" 1 active; - Option.iter - (fun path -> - let signal = - List.find_map - (function - | Core.Diagnostic s - when String.starts_with - ~prefix: - "worker.applyAuthoritative.authoritativeIntegrityFailure:" - s -> Some s - | _ -> None) - !emitted - |> Option.get - in - let groups = - (Core.view !state).sync.diagnostics.groups - |> List.filter_map (fun (g : Sync.diagnostic_group) -> - if g.title = "Sync transactions" - then - Some - (`Assoc - [ "title", `String g.title - ; ( "entries" - , `List - (List.map - (fun (k, v) -> `List [ `String k; `String v ]) - g.entries) ) - ]) - else None) - in - Yojson.Safe.to_file - path - (`Assoc - [ "lastError", `String message_text - ; "signal", `String signal - ; "outboxRecords", `Int active - ; "groups", `List groups - ])) - (Sys.getenv_opt "JOURNAL_DIAGNOSTIC_FIXTURE_OUTPUT") - | _ -> ()); - let encoded = - Yojson.Safe.to_string (Error.to_yojson error) - ^ "\n" - ^ String.concat - "\n" - (List.filter_map - (function - | Core.Diagnostic signal -> Some signal - | _ -> None) - !emitted) - in - let contains needle = - let rec loop i = - i + String.length needle <= String.length encoded - && (String.sub encoded i (String.length needle) = needle || loop (i + 1) - ) - in - loop 0 - in - List.iter - (fun secret -> - Alcotest.(check bool) - "source content excluded" - false - (contains secret)) - ([ root - ; "PRIVATE_SYNTHETIC_PAYLOAD" - ; "99999999-9999-4999-8999-999999999999" - ; "Nothing found" - ; "Cannot store nil" - ] - @ if String.length payload > 30 then [ payload ] else [])))))) + List.iter (fun (error, expected) -> + let admission = ref (Some error) and sent = ref [] in + let network : Execute.Core.worker_network = + {with_admission=(fun callback -> match !admission with + | Some error -> Error error + | None -> callback ())} in + with_direct_owner ~network + ~put:(fun ~identity:_ ~asset ~version ~wire -> + sent := (asset, version, wire) :: !sent; Ok ()) (fun fixture -> + let source = { (picker_import fixture.support 990) with target = Test_support.page_uuid } in + let intent = Runner.prepare_import fixture.runner ~context:fixture.context source + ~current:(fun () -> true) |> Result.get_ok in + let state = ref (U.create ~scope:fixture.context.scope ~available:true) in + fixture.set_upload_current (fun ticket -> U.ticket_current !state ticket); + let run instruction = + Runner.submit fixture.runner (Core.Run_upload (fixture.context, instruction)); + let rec await turns = + if not (Queue.is_empty fixture.upload_events) then Queue.take fixture.upload_events + else if turns = 0 then Alcotest.fail "Owner upload completion did not arrive" + else (Eio.Fiber.yield (); await (turns - 1)) in + await 1000 in + let step event = let next, effects = U.step !state event in state := next; effects in + let rec reach_put = function + | [U.Put _ as put] -> put + | [instruction] -> + let ticket = match instruction with + | U.Persist (ticket, _, _) | Apply_local (ticket, _) -> ticket + | _ -> Alcotest.fail "Unexpected instruction before PUT" in + let completion = run instruction in + (match completion with U.Failed _ -> Alcotest.fail "Local owner setup failed" | _ -> ()); + reach_put (step (Completed (ticket, completion))) + | _ -> Alcotest.fail "Expected one upload instruction" in + let put = reach_put (step (Start intent)) in + let ticket, uploading = match put with U.Put (ticket, intent) -> ticket, intent | _ -> assert false in + Alcotest.(check bool) "Uploading checkpoint precedes network admission" true + (uploading.phase = I.Uploading && uploading.revision = 2); + let completion = run put in + Alcotest.(check bool) "typed owner failure is classified" true (completion = U.Failed expected); + ignore (step (Completed (ticket, completion))); + Runner.submit fixture.runner (Core.Publish (Core.Asset_notice (fixture.context.scope, + Core.Upload_status {operation=intent.operation_id; asset=intent.asset; target=intent.target; + title=intent.title; status=U.Failed_upload expected}))); + let module Diagnostic = Logseq_db_types.Failure_diagnostic in + let detail = List.find_map (function + | Core.Diagnostic text when String.starts_with ~prefix:Diagnostic.signal_prefix text -> + Diagnostic.decode (String.sub text (String.length Diagnostic.signal_prefix) + (String.length text - String.length Diagnostic.signal_prefix)) + | _ -> None) !(fixture.published) |> Option.get in + let encoded = Diagnostic.encode detail in + Alcotest.(check bool) "safe diagnostic carrier stays bounded" true (String.length encoded <= 1024); + Alcotest.(check bool) "safe diagnostic message stays bounded" true + (String.length (Diagnostic.message detail) <= 512); + Alcotest.(check bool) "raw paths never reach public diagnostic" false (String.contains encoded '/'); + let reason = match error with + | D.Network_closed | Network_failure _ | Cleanup_failure _ -> "networkFailed" + | Authentication_failure _ -> "authenticationFailed" + | Storage_failure _ -> "persistenceFailed" + | Crypto_failure _ -> "cryptoInvalid" + | _ -> Alcotest.fail "Unexpected fixture error" in + Alcotest.(check string) "specific owner reason survives safe public output" + reason (Diagnostic.reason detail); + (match error with + | D.Cleanup_failure _ -> + Alcotest.(check bool) "cleanup cause is retained independently" true + (Option.map Diagnostic.reason (Diagnostic.related detail) = Some "persistenceFailed") + | _ -> ()); + Alcotest.(check int) "closed admission sends no bytes" 0 (List.length !sent); + let path = fixture.staged_path intent.staged_file |> Option.get in + Alcotest.(check string) "original source survives failure" "file" + (In_channel.with_open_bin path In_channel.input_all); + if expected = U.Network then ( + admission := None; + let retried = match step Retry with [U.Put (_, intent) as put] -> + Alcotest.(check bool) "same operation and publication IDs on retry" true (intent = uploading); put + | _ -> Alcotest.fail "Network retry did not issue original PUT" in + Alcotest.(check bool) "original owner PUT succeeds" true (run retried = U.Put_succeeded); + Alcotest.(check bool) "plain source bytes unchanged on wire" true + (!sent = [intent.asset, intent.version, "file"])))) + [D.Network_closed, U.Network; D.Network_failure "Disconnected", U.Network; + D.Authentication_failure "Denied", U.Authentication; + D.Storage_failure ("/PRIVATE_SYNTHETIC_PAYLOAD/" ^ String.make 8192 'x'), U.Invalid_content; + D.Crypto_failure "Encode failed", U.Invalid_content; + D.Cleanup_failure {primary=Some D.Network_closed; + failures=Rrbvec.singleton (D.Storage_failure "Release failed")}, U.Invalid_content] ;; -(* The pure reducer can consume an outbox notice, but cannot reproduce a missing - notice from the physical commit owner. Exercise that public runner boundary - against isolated synthetic mirrors, then drain its real events. *) -let test_durable_commit_notifies_own_graph ~resident () = - let module Mutation=Logseq_db_worker_contract.Worker_pure_mutation in - let require condition message=Alcotest.(check bool) message true condition in - T.with_managed(fun fixture-> - let graph_id=Test_support.graph_uuid in - let other_id=Test_support.mutation_uuid 29800 in - List.iter(fun graph_id->ignore(Test_support.seed_mirror ~graph_id fixture.config.application_support_directory)) - (if resident then [graph_id;other_id] else [graph_id]); - Eio_main.run(fun _->Eio.Switch.run(fun sw-> - let graph graph_id : Sync.graph={graph_id;name="Durable notification fixture"; - encrypted=false;schema={major=65;minor=33;exact=true}} in - let state=ref(Core.initial(Core.config ~worker:fixture.config ~sync:(asset_sync_config())) |> Result.get_ok) in - let events=Queue.create() and connections=ref [] and inspections=ref [] in - let authority=ref true and retire_on_projection=ref false in - let post event= - (match event with Core.Projection_push_for_scope _ when !retire_on_projection->authority:=false|_->()); - Queue.add event events in - let sync_runner=Runner.sync_runner ~retire_account:(fun ~account:_->Ok()) ~shutdown:(fun()->()) - ~submit:(function - |Sync.Request(ticket,Sync.Fetch_catalog _)->post(Core.Sync_event(Sync.Runner_completed - (Sync.Completion(ticket,Ok(List.map graph (if resident then [graph_id;other_id] else [graph_id])))))) - |Sync.Request(ticket,Sync.Save_catalog _)->post(Core.Sync_event(Sync.Runner_completed(Sync.Completion(ticket,Ok())))) - |Sync.Start_websocket request->connections:=request.scope :: !connections - |_->())() in - let dependencies=Runner.dependencies ~config:fixture.config ~overlay:fixture.overlay ~sync_runner - ~runtime:(Runner.runtime ~fork:(fun ~sw:_ task->task()) ~sleep:(fun _->()) |> Result.get_ok) - ~publish:(fun _->()) |> Result.get_ok in - let runner=Runner.create ~sw dependencies ~post ~recovery_current:(fun _->false) - ~upload_current:(fun _->false) ~asset_current:(fun _->false) |> Result.get_ok in - let drain ()=while not(Queue.is_empty events)do - let transition=Core.step !state(Queue.take events)in state:=transition.next; - List.iter(fun instruction-> - (match instruction with - |Core.Run_worker(Core.Request(_,Core.Handle_sync_worker_effect(Sync.Inspect_sync scope)))-> - inspections:=scope :: !inspections - |_->()); - Runner.submit runner instruction)transition.effects done in - let drive event=post(Core.Sync_event event);drain() in - let notices ()=Queue.fold(fun scopes->function - |Core.Sync_event(Sync.Local_graph_outbox_changed scope)->scope::scopes - |Core.Sync_event Sync.Local_outbox_changed->Alcotest.fail "durable mutation emitted an unscoped notice" - |_->scopes)[] events in - let intent ?(title="Durable notification") n= - Mutation.create_append ~origin:(Uri.of_string "https://api.logseq.io") ~user:"fixture-user" ~graph:graph_id - ~mutation_id:(Test_support.mutation_uuid n) ~page:(Test_support.mutation_uuid 29801) - ~day:(Logseq_db_worker_contract.Worker_pure_day.create 20261107 |> Result.get_ok) - ~root:{block_id=Test_support.mutation_uuid(n+100);title;task_status=None;children=Rrbvec.empty} - |> Result.get_ok |> fun value->Mutation.Append_intent value in - Fun.protect ~finally:(fun()->Runner.shutdown runner)(fun()-> - drive(Sync.Sign_in_requested{account_generation=0;presentation_generation=0}); - drive(Sync.Account_authenticated{user_id=Some "fixture-user"}); - if resident then ( - drive(Sync.Resident_graphs_changed[graph_id;other_id]); - drive(Sync.Visible_graph_changed other_id)) - else drive(Sync.Graph_selected graph_id); - List.iter(fun connection->drive(Sync.Websocket_opened connection))!connections; - let connection=List.find(fun (value:Sync.connection_scope)->value.graph.graph_id=graph_id)!connections in - let scope=connection.graph in - if resident then require((Core.view !state).sync.snapshot.selected_graph=Some other_id) - "mutation fixture must keep another resident visible"; - inspections:=[]; - let mutate intent=Runner.durable_mutation runner ~scope ~intent ~current:(fun()-> !authority) in - let first=intent 29810 in - let proof=match mutate first with Mutation.Committed proof->proof|_->Alcotest.fail "synthetic append did not commit" in - require(notices()=[scope]) "committed append posts exactly its own scoped sync notice"; - drain(); - require(!inspections=[scope]) "committed append automatically inspects only its own graph"; - require(match mutate first with Mutation.Committed replay->replay=proof|_->false) - "idempotent replay retains original commit proof"; - require(notices()=[scope]) "replayed commit can repair a lost sync notification"; - drain(); - require(match mutate(intent ~title:"Conflicting payload" 29810)with Mutation.Rejected _->true|_->false) - "conflicting intent is rejected"; - require(notices()=[]) "rejected mutation never notifies sync"; - authority:=false; - require(mutate(intent 29811)=Mutation.Unknown) "retired authority cannot commit"; - require(notices()=[]) "unknown mutation never notifies sync"; - authority:=true;retire_on_projection:=true; - require(match mutate(intent 29812)with Mutation.Committed _->true|_->false) - "physical commit retains evidence when presentation retires"; - require(not !authority) "fixture retires authority during actual commit publication"; - require(notices()=[scope]) "caller retirement cannot strand a successfully committed outbox"; - let cancelled=Runner.durable_mutation runner ~scope ~intent:(intent 29813) - ~current:(fun()->raise(Eio.Cancel.Cancelled(Failure "synthetic cancellation"))) in - require(cancelled=Mutation.Unknown) "cancellation retains legacy Unknown"; - let raised=ref false in - let outcome=Runner.durable_mutation runner ~scope ~intent:(intent 29814) - ~current:(fun()->if !raised then true else ( - raised:=true;raise(Unix.Unix_error(Unix.EACCES,"open","PRIVATE_SYNTHETIC_PAYLOAD")))) in - match outcome with - | Mutation.Unknown_with_cause cause -> - let module D=Logseq_db_types.Failure_diagnostic in - Alcotest.(check string) "resource exception operation is request bound" "commitLocal" (D.operation cause); - Alcotest.(check string) "resource exception reason is request bound" "resourceFailed" (D.reason cause); - Alcotest.(check string) "resource exception has safe classification" "io-permission" (D.source_class cause); - require(not(String.contains(D.encode cause) '/')) "resource exception excludes private paths" - | _ -> Alcotest.fail "non-cancel resource exception must return a request-bound cause")))) +let test_upload_checkpoint_io () = + let module Upload = Logseq_db_worker_pure_reducer.Asset_upload in + with_direct_owner (fun fixture -> + let source = picker_import fixture.support 900 in + let intent = Runner.prepare_import fixture.runner ~context:fixture.context source + ~current:(fun () -> true) |> Result.get_ok in + let state, instructions = Upload.step + (Upload.create ~scope:fixture.context.scope ~available:true) (Upload.Start intent) in + let current = ref true in + fixture.set_upload_current (fun ticket -> !current && Upload.ticket_current state ticket); + let instruction = List.hd instructions in + Runner.submit fixture.runner (Core.Run_upload (fixture.context, instruction)); + let rec await turns = + if not (Queue.is_empty fixture.upload_events) then Queue.take fixture.upload_events + else if turns=0 then Alcotest.fail "Upload checkpoint completion did not arrive" + else (Eio.Fiber.yield (); await (turns-1)) in + Alcotest.(check bool) "durable acknowledgement" true (await 1000 = Upload.Persisted); + let db = Sqlite3.db_open ~mode:`READONLY + (Filename.concat fixture.support "asset-upload-intents.sqlite") in + let stored = Fun.protect ~finally:(fun () -> + Alcotest.(check bool) "independent checkpoint reader closes" true (Sqlite3.db_close db)) + (fun () -> Logseq_db_storage.Asset_upload_store.read db ~operation:intent.operation_id + |> Result.get_ok) in + Alcotest.(check bool) "survives independent database reopen" true (stored=Some intent); + current := false; + Runner.submit fixture.runner (Core.Run_upload (fixture.context, instruction)); + for _=1 to 8 do Eio.Fiber.yield () done; + Alcotest.(check bool) "stale upload ticket performs no completion" true + (Queue.is_empty fixture.upload_events)) ;; -let test_durable_commit_during_protection ?(resident=false) mode () = - let module Mutation=Logseq_db_worker_contract.Worker_pure_mutation in - let require condition message=Alcotest.(check bool) message true condition in - T.with_managed(fun fixture-> - let graph_id=Test_support.graph_uuid in - let other_id=Test_support.mutation_uuid 29800 in - List.iter(fun graph_id->ignore(Test_support.seed_mirror ~graph_id fixture.config.application_support_directory)) - (if resident then [graph_id;other_id] else [graph_id]); - Eio_main.run(fun env->Eio.Switch.run(fun sw-> - let graph graph_id : Sync.graph={graph_id;name="Durable notification fixture"; - encrypted=true;schema={major=65;minor=33;exact=true}} in - let state=ref(Core.initial(Core.config ~worker:fixture.config ~sync:(asset_sync_config())) |> Result.get_ok) in - let events=Queue.create() and connections=ref [] and inspections=ref [] in - let authority=ref true and retire_on_projection=ref false in - let held=ref None and running=ref 0 and failures=ref [] and inspected=ref None and closes=ref 0 in - let hold_crypto=ref(mode<>`Ack) and hold_ack=ref(mode=`Ack) and held_ack=ref None in - let sends=ref 0 and superseded=ref 0 and applied=ref None in - let post event= - (match event with Core.Projection_push_for_scope _ when !retire_on_projection->authority:=false|_->()); - Queue.add event events in - let sync_runner=Runner.sync_runner ~retire_account:(fun ~account:_->Ok()) ~shutdown:(fun()->()) - ~submit:(function - |Sync.Request(ticket,Sync.Fetch_catalog _)->post(Core.Sync_event(Sync.Runner_completed - (Sync.Completion(ticket,Ok(List.map graph (if resident then [graph_id;other_id] else [graph_id])))))) - |Sync.Request(ticket,Sync.Save_catalog _)->post(Core.Sync_event(Sync.Runner_completed(Sync.Completion(ticket,Ok())))) - |Sync.Request(ticket,Sync.Load_and_unlock_graph_key scope)-> - post(Core.Sync_event(Sync.Runner_completed(Sync.Completion(ticket,Ok(Sync.graph_key_handle ~id:"synthetic-key" ~scope))))) - |Sync.Request(ticket,Sync.Fetch_and_unlock_graph_key request)-> - post(Core.Sync_event(Sync.Runner_completed(Sync.Completion(ticket,Ok(Sync.graph_key_handle ~id:"synthetic-key" ~scope:request.scope))))) - |Sync.Protected_io(ticket,request)-> - if !hold_crypto then held:=Some(ticket,request) else - let values=match request.Sync.protected_action with Sync.Encrypt_values(_,values)->values|_->assert false in - post(Core.Sync_event(Sync.Runner_completed(Sync.Protected_completion(ticket, - Ok(Sync.Encrypted_values(List.map(fun value->String.make 12 'i',value)values)))))) - |Sync.Send_websocket {message=Logseq_sync_pure_reducer.Sync_protocol.Client.Tx_batch _;_}->incr sends - |Sync.Close_websocket _->incr closes - |Sync.Start_websocket request->connections:=request.scope :: !connections - |_->())() in - let dependencies=Runner.dependencies ~config:fixture.config ~overlay:fixture.overlay ~sync_runner - ~runtime:(Runner.runtime ~fork:(fun ~sw task->incr running;Eio.Fiber.fork ~sw(fun()->Fun.protect ~finally:(fun()->decr running)task)) ~sleep:(fun _->()) |> Result.get_ok) - ~publish:(fun _->()) |> Result.get_ok in - let runner=Runner.create ~sw dependencies ~post ~recovery_current:(fun _->false) - ~upload_current:(fun _->false) ~asset_current:(fun _->false) |> Result.get_ok in - let drain ()=while not(Queue.is_empty events)do - let event=Queue.take events in - (match event with - |Core.Runner_completed(Core.Sync_worker_effect_completed(_,Error error))->failures:=error:: !failures - |Core.Runner_completed(Core.Sync_worker_effect_completed(_,Ok{event=Some(Sync.Sync_inspected{sync;_});_}))->inspected:=Some sync - |Core.Runner_completed(Core.Sync_worker_effect_completed(_,Ok{event=Some(Sync.Outbox_transition_superseded _);_}))->incr superseded - |Core.Runner_completed(Core.Sync_worker_effect_completed(_,Ok{event=Some(Sync.Outbox_transition_applied result);_}))->applied:=Some result - |_->()); - let transition=Core.step !state event in state:=transition.next; - List.iter(fun instruction-> - (match instruction with - |Core.Run_worker(Core.Request(_,Core.Handle_sync_worker_effect(Sync.Inspect_sync scope)))-> - inspections:=scope :: !inspections - |_->()); - match instruction with - |Core.Run_worker(Core.Request(_,Core.Handle_sync_worker_effect(Sync.Apply_outbox_transition request))) - when !hold_ack && (match request.transition with Logseq_overlay_db.Types.Accept_group _->true|_->false)-> - held_ack:=Some(instruction,request) - |_->Runner.submit runner instruction)transition.effects done in - let until condition=Eio.Time.with_timeout_exn(Eio.Stdenv.clock env)3.(fun()-> - while not(condition())do drain();Eio.Fiber.yield()done;drain()) in - let settle ()=until(fun()-> !running=0 && Queue.is_empty events) in - let drive event=post(Core.Sync_event event);settle() in - let intent ?(title="Durable notification") n= - Mutation.create_append ~origin:(Uri.of_string "https://api.logseq.io") ~user:"fixture-user" ~graph:graph_id - ~mutation_id:(Test_support.mutation_uuid n) ~page:(Test_support.mutation_uuid 29801) - ~day:(Logseq_db_worker_contract.Worker_pure_day.create 20261107 |> Result.get_ok) - ~root:{block_id=Test_support.mutation_uuid(n+100);title;task_status=None;children=Rrbvec.empty} - |> Result.get_ok |> fun value->Mutation.Append_intent value in - Fun.protect ~finally:(fun()->Runner.shutdown runner)(fun()-> - drive(Sync.Sign_in_requested{account_generation=0;presentation_generation=0}); - drive(Sync.Account_authenticated{user_id=Some "fixture-user"}); - if resident then ( - drive(Sync.Resident_graphs_changed[graph_id;other_id]); - drive(Sync.Visible_graph_changed other_id)) - else drive(Sync.Graph_selected graph_id); - List.iter(fun connection->drive(Sync.Websocket_opened connection))!connections; - let connection=List.find(fun (value:Sync.connection_scope)->value.graph.graph_id=graph_id)!connections in - let scope=connection.graph in - if resident then require((Core.view !state).sync.snapshot.selected_graph=Some other_id) - "mutation fixture must keep another resident visible"; - inspections:=[]; - let mutate intent=Runner.durable_mutation runner ~scope ~intent ~current:(fun()-> !authority) in - require(match mutate(intent 29810)with Mutation.Committed _->true|_->false) "first append commits"; - (match mode with - |`Ack-> - settle();require(!sends=1) "first capture reaches actual submission"; - drive(Sync.Websocket_message(connection,Logseq_sync_pure_reducer.Sync_protocol.Server.Tx_batch_ok - {t=Logseq_db_types.Server_cursor.of_int64 1L |> Result.get_ok;checksum=Some "0123456789abcdef"})); - let instruction,request=Option.get !held_ack in - require(match mutate(intent 29811)with Mutation.Committed _->true|_->false) "second append commits before ACK preparation starts"; - settle();hold_ack:=false;Runner.submit runner instruction;settle(); - let result=Option.get !applied in - require(result.commit.transition=request.transition) "ACK recovery preserves its exact server barrier"; - let records=Logseq_overlay_db.Types.sync_view_submissions result.sync in - require(List.length records=2 && List.exists(fun (item:Logseq_overlay_db.Types.submission_descriptor)-> - match item.state with Accepted_pending_authoritative _->true|_->false)records) - "ACK settles only its submitted capture while later capture stays durable" - |`Crypto|`Malformed|`Crypto_error-> - until(fun()-> !held<>None); - require(!running>0) "first outbox preparation is suspended in protected IO"; - if mode=`Crypto then ( - require(match mutate(intent 29811)with Mutation.Committed _->true|_->false) "second append commits while protection waits"; - until(fun()-> !running=1 && Queue.is_empty events); - let queued=Logseq_overlay_db.Types.sync_view_submissions(Option.get !inspected) in - require(List.length queued=2 && List.for_all(fun (item:Logseq_overlay_db.Types.submission_descriptor)->item.state=Queued)queued) - "both rapid captures remain durably queued before crypto release"); - let ticket,request=Option.get !held in - let encrypted=match request.Sync.protected_action with - |Sync.Encrypt_values(_,values)->List.map(fun value->String.make 12 'i',value)values - |_->Alcotest.fail "expected encryption request" in - hold_crypto:=false; - let result=match mode with - |`Crypto_error->Error(Sync.Crypto_failed(Sync.Invalid_key_material,"synthetic invalid key")) - |`Malformed->Ok(Sync.Encrypted_values []) - |_->Ok(Sync.Encrypted_values encrypted) in - post(Core.Sync_event(Sync.Runner_completed(Sync.Protected_completion(ticket,result)))); - settle()); - if mode=`Malformed || mode=`Crypto_error then ( - require(List.length !failures=1 && !closes=1) "genuine crypto failure remains visible and closes connection"; - require(!superseded=0 && !sends=0) "genuine crypto failure is never retried or sent") - else ( - require(!failures=[] && !closes=0) "concurrent durable commit recovers without failing or closing sync"; - require(!superseded=1) "exactly one revision-driven recovery"; - require(!sends=1) "capture submits once after recovery"); - require((Core.view !state).graph.phase=Core.Graph_open) "concurrent durable commit preserves attached graph")))) +let test_plaintext_submission_encoding () = + let module D = Sync_dsl.Core in + let module C = Logseq_sync_crypto.Core in + let module Database = Logseq_overlay_db.Database in + let module Types = Logseq_overlay_db.Types in + with_direct_owner (fun fixture -> + let require = function Ok value -> value | Error _ -> Alcotest.fail "submission setup failed" in + let uuid value = Logseq_db_types.Graph_types.Uuid.of_string value |> require in + let page = uuid "70000000-0000-4000-8000-000000000001" in + let mutation_id = uuid "70000000-0000-4000-8000-000000000002" in + let title = "~:plain-中文-\"quoted\"" in + let snapshot = Database.current_snapshot fixture.database |> require in + let expected = Fun.protect ~finally:(fun () -> Database.release_snapshot snapshot) + (fun () -> match Database.get_pages snapshot [page] |> require with + | [Types.Missing_page {revision;_}] -> + Database.write_precondition ~blocks:[] ~pages:[page,revision] ~scopes:[] |> require + | _ -> Alcotest.fail "new page exists") in + Database.commit_local fixture.database ~expected + (Types.Create_journal_page {mutation_id;page;title;journal_day=20261119}) |> require |> ignore; + let view = Database.inspect_sync fixture.database |> require in + let prepared, request = Database.begin_outbox_transition fixture.database + ~expected:(Types.sync_view_token view) (Types.Submit_group [mutation_id]) |> require in + let request = Option.get request in + let run crypto operation = C.run crypto ~identity:fixture.identity + ~check_current:(fun () -> Ok ()) operation in + let locked = C.create ~identity:fixture.identity ~limits:(D.limits + ~maximum_items:8 ~maximum_bytes:32768 ~maximum_pending:8 ~maximum_resources:8 + ~steps_per_turn:8 ~maximum_tasks:8 ~maximum_messages:8 ~maximum_message_bytes:32768 + ~maximum_demands:8 ~maximum_active_assets:2 ~foreground_reserve:1 + ~foreground_demand_reserve:1 ~foreground_descriptor_reserve_bytes:128 + ~asset_cache_budget_bytes:16L ~asset_maximum_file_bytes:16 ~asset_pending_budget_bytes:32L + ~snapshot_maximum_artifact_bytes:4096L ~snapshot_staging_budget_bytes:4096L + ~snapshot_scratch_budget_bytes:4096L |> require) ~encrypted:true ~backend:C.apple_backend in + Fun.protect ~finally:(fun () -> C.retire locked) (fun () -> + Alcotest.(check bool) "encrypted graph cannot downgrade without key" true + (run locked (D.Protect request) = Error D.Locked)); + let values = match run fixture.crypto (D.Protect request) with + | Ok values -> values + | Error error -> Alcotest.fail (D.error_message error) in + Alcotest.(check bool) "plain upload decodes the serialized title" true + (List.mem title (List.map snd values)); + let result = Database.apply_outbox_transition fixture.database prepared + ~encrypted:(Some (request,values)) |> require in + let wire = Option.get result.submission_batch |> Types.submission_batch_wires + |> List.hd |> Types.submission_wire_protected_transaction in + let operations = match Logseq_db_storage.Logseq_sqlite_codec.decode_transit wire with + | Ok (Array operations) -> operations + | Ok _ -> Alcotest.fail "submission wire is not a Transit transaction array" + | Error message -> Alcotest.fail message in + let select_operations select = List.filter_map select operations in + let page_entities = select_operations (function + | Array [Keyword "db/add"; entity; Keyword "block/uuid"; Uuid value] + when value = Logseq_db_types.Graph_types.Uuid.to_string page -> Some entity + | _ -> None) in + let entity = match page_entities with + | [entity] -> entity + | _ -> Alcotest.fail "submission must identify exactly one target page entity" in + let titles = select_operations (function + | Array [Keyword "db/add"; target; Keyword "block/title"; value] + when target = entity -> Some value + | _ -> None) in + match titles with + | [String value] -> + Alcotest.(check string) "target page title survives Transit decoding" title value + | _ -> Alcotest.fail "target page must have exactly one Transit String block/title") ;; let () = - Alcotest.run - "logseq db worker effect runner" - [ ( "durable protection race", [ - Alcotest.test_case "second append during protected preparation" `Quick (test_durable_commit_during_protection `Crypto); - Alcotest.test_case "second append before ACK preparation" `Quick (test_durable_commit_during_protection `Ack); - Alcotest.test_case "genuine crypto failure is not retried" `Quick (test_durable_commit_during_protection `Crypto_error); - Alcotest.test_case "malformed crypto result is not retried" `Quick (test_durable_commit_during_protection `Malformed); - Alcotest.test_case "hidden resident protection race" `Quick (test_durable_commit_during_protection ~resident:true `Crypto)]) - ; ( "durable sync notification", [ - Alcotest.test_case "single graph commit and replay" `Quick (test_durable_commit_notifies_own_graph ~resident:false); - Alcotest.test_case "hidden resident commit and replay" `Quick (test_durable_commit_notifies_own_graph ~resident:true) ]) - ; ( "authoritative diagnostics", [ - Alcotest.test_case "real own echo mismatch retains safe integrity reason" `Quick (test_authoritative_rejection_diagnostics `Integrity); - Alcotest.test_case "real cursor rejection reaches Diagnostics" `Quick (test_authoritative_rejection_diagnostics `Cursor); - Alcotest.test_case "real format rejection reaches Diagnostics" `Quick (test_authoritative_rejection_diagnostics `Format); - Alcotest.test_case "apply format failure excludes raw source" `Quick (test_authoritative_rejection_diagnostics `Apply_format) ] @ - List.map (fun ((name,_,_) as detail) -> Alcotest.test_case ("safe detail "^name) `Quick - (test_authoritative_rejection_diagnostics (`Detail detail))) authoritative_detail_cases) - ; ( "ID token cache" - , [ Alcotest.test_case - "caps reuse at twenty-four hours" - `Quick - test_id_token_cache_caps_reuse_at_twenty_four_hours - ; Alcotest.test_case - "caps reuse one hour before expiration" - `Quick - test_id_token_cache_caps_reuse_one_hour_before_expiration - ; Alcotest.test_case - "does not retain immediately non-reusable token" - `Quick - test_id_token_cache_does_not_retain_immediately_non_reusable_token - ; Alcotest.test_case - "rejects invalid responses without retaining them" - `Quick - test_id_token_cache_rejects_invalid_responses_without_retaining_them - ; Alcotest.test_case - "coalesces concurrent misses" - `Quick - test_id_token_cache_coalesces_concurrent_misses - ; Alcotest.test_case - "propagates Flutter failure without caching" - `Quick - test_id_token_cache_propagates_flutter_failure_without_caching - ; Alcotest.test_case - "invalidation is credential-specific" - `Quick - test_id_token_cache_invalidation_is_credential_specific - ; Alcotest.test_case - "sign-out clears entry and cancels waiters" - `Quick - test_id_token_cache_sign_out_clears_entry_and_cancels_waiters - ; Alcotest.test_case - "account replacement makes delayed response inert" - `Quick - test_id_token_cache_account_replacement_makes_delayed_response_inert - ; Alcotest.test_case - "shutdown clears entry and cancels waiters" - `Quick - test_id_token_cache_shutdown_clears_entry_and_cancels_waiters - ] ) - ; ( "unlock diagnostics", [Alcotest.test_case "typed finite runner observations" `Quick test_unlock_runner_logging] ) - ; ( "contract" - , [ Alcotest.test_case "wrapped local publication storage full retains staging" `Quick (test_wrapped_local_asset_fallback `Full) - ; Alcotest.test_case "wrapped missing staging falls back to remote cache" `Quick (test_wrapped_local_asset_fallback `Missing) - ; Alcotest.test_case "wrapped checksum failure falls back to remote cache" `Quick (test_wrapped_local_asset_fallback `Checksum) - ; Alcotest.test_case "wrapped nonfallback publication failure releases lease" `Quick (test_wrapped_local_asset_fallback `Oversize) - ; Alcotest.test_case "worker scheduling failure completes" `Quick (fun () -> test_worker_schedule_failure_completes ()) - ; Alcotest.test_case "worker encoded exception preserves reason" `Quick - (test_worker_schedule_failure_completes - ~failure:(Invalid_argument (Logseq_db_storage.Logseq_sqlite_codec.diagnostic_error - (Logseq_db_storage.Logseq_sqlite_codec.Unsupported_tag "PRIVATE_SYNTHETIC_PAYLOAD"))) - ~expected_source:"unknown" ~expected_reason:"codecUnsupportedTag") - ; Alcotest.test_case "readonly import input recovery and exact proof ACK" `Quick test_import_input_readonly_recovery_and_proof_ack - ; Alcotest.test_case "initial observation snapshot joins real release" `Quick test_observation_release_waits_for_initial_snapshot - ; Alcotest.test_case "host internal followup survives full actor queue" `Quick test_host_followup_survives_full_actor_queue - ; Alcotest.test_case "host scope settlement preserves real lease" `Quick test_host_scope_settlement_preserves_live_lease - ; Alcotest.test_case "Worker Sync IO fence correlates actual child" `Quick test_worker_sync_io_fence_correlates_real_child - ; Alcotest.test_case "Sync IO fence joins nonasset callback" `Quick test_sync_io_fence_joins_nonasset_callback - ; Alcotest.test_case "scope close keeps live staged lease" `Quick test_scope_close_keeps_live_staged_lease - ; Alcotest.test_case "asset IO barrier submit stays nonblocking" `Quick test_asset_io_barrier_does_not_block_serial_submit - ; Alcotest.test_case "writer fence joins actual canceled asset callback" `Quick test_writer_fence_joins_actual_canceled_asset_callback - ; Alcotest.test_case "asset cleanup joins queued result child" `Quick test_asset_cleanup_joins_queued_result_child - ; Alcotest.test_case "asset cleanup ignores later unrelated work" `Quick test_asset_cleanup_ignores_later_unrelated_work - ; Alcotest.test_case "asset cleanup stop returns failure" `Quick test_asset_cleanup_stop_does_not_wait_for_lost_io - ; Alcotest.test_case "asset cleanup snapshot retains physical failure" `Quick test_asset_cleanup_snapshot_retains_physical_failure - ; Alcotest.test_case "asset cleanup snapshot joins queued release" `Quick test_asset_cleanup_snapshot_joins_queued_release - ; Alcotest.test_case "asset cleanup snapshot joins physical release" `Quick test_asset_cleanup_snapshot_joins_physical_release - ; Alcotest.test_case "discarded retain retries actual storage cleanup" `Quick - test_discarded_retain_retries_real_storage_cleanup - ; Alcotest.test_case - "checked release preserves storage failure for same-handle retry" `Quick - test_checked_release_storage_retry - ; Alcotest.test_case - "checked release awaits exact physical completion" `Quick - test_checked_release_exact_physical_completion - ; Alcotest.test_case - "cancel queued staging output" - `Quick - (cancelled_stage_output false) - ; Alcotest.test_case - "cancel resolved staging output" - `Quick - (cancelled_stage_output true) - ; Alcotest.test_case - "cancel queued retained lease output" - `Quick - test_cancelled_retained_output - ; Alcotest.test_case - "actual Db shutdown late staging completion" - `Quick - test_db_shutdown_accepts_late_staging_completion - ; Alcotest.test_case - "durable upload checkpoint IO" - `Quick - test_upload_checkpoint_io - ; Alcotest.test_case "staging cleanup survives unavailable warning output" `Quick - (test_upload_checkpoint_io ~closed_stderr:true) - ; Alcotest.test_case - "sync and publish delegation" - `Quick - test_sync_and_publish_instructions_are_not_reduced_recursively - ; Alcotest.test_case - "catalog refresh cannot complete another deletion" - `Quick - test_catalog_refresh_cannot_complete_another_deletion - ; Alcotest.test_case "busy mirror preserves attachment sources" `Quick - test_deletion_checks_ownership_before_assets - ; Alcotest.test_case "journal IO failure preserves current Worker" `Quick - test_deletion_journal_failure_preserves_worker - ; Alcotest.test_case "caught retirement confirmation failure reports Diagnostic" `Quick - test_deletion_confirmation_failure_reports - ; Alcotest.test_case "delete never-opened legacy copy" `Quick - (deletion_runtime_fixture ~deletion_case:`Legacy ~busy:false) - ; Alcotest.test_case "delete non-SQLite copy" `Quick - (deletion_runtime_fixture ~deletion_case:`Corrupt_sqlite ~busy:false) - ; Alcotest.test_case "delete invalid checkpoint copy" `Quick - (deletion_runtime_fixture ~deletion_case:`Corrupt_checkpoint ~busy:false) - ; Alcotest.test_case "delete residual directory without database" `Quick - (deletion_runtime_fixture ~deletion_case:`Missing_database ~busy:false) - ; Alcotest.test_case "delete contained database link without following it" `Quick - (deletion_runtime_fixture ~deletion_case:`Linked_database ~busy:false) - ] ) - ] + Alcotest.run "logseq db worker effect runner" + [ "ID token cache", [ + Alcotest.test_case "caps reuse at twenty four hours" `Quick test_id_token_cache_caps_reuse_at_twenty_four_hours; + Alcotest.test_case "caps reuse one hour before expiration" `Quick test_id_token_cache_caps_reuse_one_hour_before_expiration; + Alcotest.test_case "does not retain immediately non reusable token" `Quick test_id_token_cache_does_not_retain_immediately_non_reusable_token; + Alcotest.test_case "rejects invalid responses without retaining them" `Quick test_id_token_cache_rejects_invalid_responses_without_retaining_them; + Alcotest.test_case "coalesces concurrent misses" `Quick test_id_token_cache_coalesces_concurrent_misses; + Alcotest.test_case "propagates flutter failure without caching" `Quick test_id_token_cache_propagates_flutter_failure_without_caching; + Alcotest.test_case "invalidation is credential specific" `Quick test_id_token_cache_invalidation_is_credential_specific; + Alcotest.test_case "sign out clears entry and cancels waiters" `Quick test_id_token_cache_sign_out_clears_entry_and_cancels_waiters; + Alcotest.test_case "account replacement makes delayed response inert" `Quick test_id_token_cache_account_replacement_makes_delayed_response_inert; + Alcotest.test_case "shutdown clears entry and cancels waiters" `Quick test_id_token_cache_shutdown_clears_entry_and_cancels_waiters + ]; + "contract", [ + Alcotest.test_case "sync and publish instructions are not reduced recursively" `Quick test_sync_and_publish_instructions_are_not_reduced_recursively; + Alcotest.test_case "direct owner io failure has no event bridge" `Quick test_direct_owner_io_failure_has_no_event_bridge; + Alcotest.test_case "worker schedule failure completes" `Quick (fun () -> test_worker_schedule_failure_completes ()); + Alcotest.test_case "import input readonly recovery and proof ack" `Quick test_import_input_readonly_recovery_and_proof_ack; + Alcotest.test_case "worker sync io fence correlates real child" `Quick test_worker_sync_io_fence_correlates_real_child; + Alcotest.test_case "scope close keeps live staged lease" `Quick test_scope_close_keeps_live_staged_lease; + Alcotest.test_case "host scope settlement preserves live lease" `Quick test_host_scope_settlement_preserves_live_lease; + Alcotest.test_case "observation release waits for initial snapshot" `Quick test_observation_release_waits_for_initial_snapshot; + Alcotest.test_case "host followup survives full actor queue" `Quick test_host_followup_survives_full_actor_queue; + Alcotest.test_case "checked release storage retry" `Quick test_checked_release_storage_retry; + Alcotest.test_case "discarded retain retries real storage cleanup" `Quick test_discarded_retain_retries_real_storage_cleanup; + Alcotest.test_case "asset cleanup snapshot retains physical failure" `Quick test_asset_cleanup_snapshot_retains_physical_failure + ; Alcotest.test_case "worker encoded exception preserves reason" `Quick + (test_worker_schedule_failure_completes + ~failure:(Invalid_argument (Logseq_db_storage.Logseq_sqlite_codec.diagnostic_error + (Logseq_db_storage.Logseq_sqlite_codec.Unsupported_tag "PRIVATE_SYNTHETIC_PAYLOAD"))) + ~expected_source:"unknown" ~expected_reason:"codecUnsupportedTag") + ; Alcotest.test_case "checked release exact physical completion" `Quick test_checked_release_exact_physical_completion + ; Alcotest.test_case "asset cleanup snapshot joins queued release" `Quick test_asset_cleanup_snapshot_joins_queued_release + ; Alcotest.test_case "asset cleanup snapshot joins physical release" `Quick test_asset_cleanup_snapshot_joins_physical_release + ; Alcotest.test_case "asset cleanup stop does not wait for lost io" `Quick test_asset_cleanup_stop_does_not_wait_for_lost_io + ; Alcotest.test_case "recovery prunes original owner staging" `Quick test_recovery_prunes_original_owner_staging + ; Alcotest.test_case "failed staging recovery is not acknowledged" `Quick test_recovery_staging_failure_is_not_acknowledged + ; Alcotest.test_case "owner upload network classification" `Quick test_upload_owner_network_classification + ; Alcotest.test_case "durable upload checkpoint IO" `Quick test_upload_checkpoint_io + ; Alcotest.test_case "plaintext submission encoding" `Quick test_plaintext_submission_encoding + ] ] ;; diff --git a/logseq_db_worker/test/test_lui_service.ml b/logseq_db_worker/test/test_lui_service.ml index bf9f36fb..e4d71693 100644 --- a/logseq_db_worker/test/test_lui_service.ml +++ b/logseq_db_worker/test/test_lui_service.ml @@ -1,41 +1,12 @@ module T = Logseq_db_worker_test_support.Test_support module ID = Journal_worker_ids module Service = Logseq_db_worker_lui.Logseq_db_worker_lui_service -module Runner = Logseq_sync_effect_runner.Effect_runner module Worker = Logseq_db_worker_lui.Journal_worker module Worker_runtime = Logseq_db_worker_lui.Journal_worker_runtime -let crypto = - Runner.crypto - ~encrypt_aes_gcm:(fun ~key:_ ~plaintext:_ -> Error "unavailable") - ~decrypt_aes_gcm:(fun ~key:_ ~iv:_ ~ciphertext:_ -> Error "unavailable") - |> Result.get_ok -;; - -let secrets = - Runner.secrets - ~unlock_private_key: - (fun - ~managed_sync_origin:_ ~user_id:_ ~password:_ ~private_key_package:_ -> - Error "unavailable") - ~unlock_graph_key:(fun ~managed_sync_origin:_ ~user_id:_ ~encrypted_graph_key:_ -> - Error "unavailable") - ~load_wrapped_graph_key:(fun ~managed_sync_origin:_ ~user_id:_ ~graph_id:_ -> - Error (Runner.Wrapped_graph_key_unavailable "unavailable")) - ~verify_and_save_wrapped_graph_key: - (fun - ~managed_sync_origin:_ ~user_id:_ ~graph_id:_ ~encrypted_graph_key:_ -> - Error "unavailable") - ~delete_account_secrets:(fun ~managed_sync_origin:_ ~user_id:_ -> Ok ()) - |> Result.get_ok -;; - let dependencies = Service.dependencies ~overlay:(T.overlay_dependencies ()) - ~tls_authenticator:(Runner.system_tls_authenticator () |> Result.get_ok) - ~secrets - ~crypto ;; let accepted = function @@ -131,8 +102,17 @@ let test_host_handoff_real_service_single_use () = T.with_managed(fun fixture->with_client fixture.config(fun client-> let send request=await_response client(Worker.send client request |> accepted) in ignore(send(Service.Client_command(Restore_local_account{user_id="host-synthetic"}))); - let baseline=match send(Service.Register_coordination 71L)with - |Coordination_registered value->value|_->T.fail "host baseline" in + let deadline = Unix.gettimeofday () +. 2. in + let rec account_baseline () = + match send (Service.Register_coordination 71L) with + | Coordination_registered value when Option.is_some value.baseline.state.sync.account -> value + | Coordination_registered _ when Unix.gettimeofday () < deadline -> + Thread.delay 0.001; + account_baseline () + | Coordination_registered _ -> T.fail "restored account never reached the host baseline" + | _ -> T.fail "host baseline" + in + let baseline = account_baseline () in let owner=Option.get baseline.baseline.state.sync.account in let plan:Logseq_db_worker.host_plan={version=baseline.baseline.version; id=Logseq_db_types.Graph_types.Uuid.of_string "00000000-0000-4000-8000-000000079071" |> Result.get_ok; @@ -186,39 +166,22 @@ let test_managed_client_command_is_accepted () = | _ -> T.fail "managed client command returned the wrong response")) ;; -(* Build opaque unlock evidence through the production reducer, including its - actual key-package and attachment owners. No boolean-to-receipt fixture. *) -let synthetic_unlock_prompt graph_id = - let module C = Logseq_sync_pure_reducer.Core in - let limits = C.limits ~maximum_response_bytes:Logseq_db_types.Limits.maximum_response_bytes - ~maximum_artifact_bytes:1048576 ~submission_batch_size:32 |> Result.get_ok in - let initial = C.config ~managed_sync_origin:(Uri.of_string "https://example.invalid") ~limits - |> Result.get_ok |> C.initial |> Result.get_ok in - let permitted = C.step initial (C.Sign_in_requested - { account_generation = 0; presentation_generation = 0 }) in - let auth = C.step permitted.next (C.Account_authenticated { user_id = Some "synthetic-user" }) in - let graph : C.graph = { graph_id; name = "Private Journal"; encrypted = true - ; schema = { major = 65; minor = 33; exact = true } } in - let catalog = List.find_map (function - | C.Run (C.Request (ticket, C.Fetch_catalog _)) -> Some (C.step auth.next - (C.Runner_completed (C.Completion (ticket, Ok [graph])))) - | _ -> None) auth.effects |> Option.get in - let selected = C.step catalog.next (C.Graph_selected graph_id) in - let mirror = List.find_map (function C.Delegate (C.Inspect_mirror request) -> Some request - | _ -> None) selected.effects |> Option.get in - let local = C.step selected.next (C.Mirror_inspected (C.Mirror_available mirror)) in - let prepare = List.find_map (function - | C.Run (C.Request (ticket, C.Load_and_unlock_graph_key _)) -> Some (C.step local.next - (C.Runner_completed (C.Completion (ticket, Error (C.Effect_failed "synthetic missing key"))))) - | _ -> None) local.effects |> Option.get in - let wrapped = List.find_map (function - | C.Run (C.Request (ticket, C.Fetch_e2ee_graph_key _)) -> Some (C.step prepare.next - (C.Runner_completed (C.Completion (ticket, Ok "synthetic-wrapped")))) - | _ -> None) prepare.effects |> Option.get in - List.find_map (function - | C.Run (C.Request (ticket, C.Fetch_e2ee_user_keys _)) -> Some (C.step wrapped.next - (C.Runner_completed (C.Completion (ticket, Ok "synthetic-package")))) - | _ -> None) wrapped.effects |> Option.get +let synthetic_unlock_prompt graph_id : Logseq_sync_domain.Core.unlock_challenge = + { challenge_scope = + { account = + { managed_sync_origin = Uri.of_string "https://example.invalid" + ; user_id = "synthetic-user" + ; account_generation = 1 + ; presentation_generation = 1 + ; lifecycle_generation = 0L + } + ; graph_id + ; graph_generation = 1 + } + ; challenge_id = 1 + ; encrypted_graph_key = "synthetic-wrapped" + ; private_key_package = "synthetic-package" + } ;; let test_unlock_ignored_logging () = @@ -243,11 +206,9 @@ let test_unlock_ignored_logging () = with_client fixture.config (fun client -> let id = Worker.send client (Service.Client_command (Submit_e2ee_password - (Logseq_sync_pure_reducer.Core.unlock_challenge - (Logseq_sync_pure_reducer.Core.state - (synthetic_unlock_prompt (Logseq_db_types.Graph_types.Uuid.of_string - "00000000-0000-0000-0000-000000000901" |> Result.get_ok)).next).snapshot.unlock - |> Option.get, "LOG_SECRET_PASSWORD"))) |> accepted in + (synthetic_unlock_prompt (Logseq_db_types.Graph_types.Uuid.of_string + "00000000-0000-0000-0000-000000000901" |> Result.get_ok), + "LOG_SECRET_PASSWORD"))) |> accepted in (match await_response client id with | Service.Client_command_completed -> () | _ -> T.fail "password queue acknowledgement changed"); @@ -263,7 +224,7 @@ let test_unlock_ignored_logging () = ;; module Mailbox = Logseq_db_worker_lui.Journal_bounded_mailbox.Coalesced -module Transfer = Logseq_sync_pure_reducer.Asset_transfer +module Transfer = Logseq_sync_asset_transfer.Core let asset_scope generation : Service.asset_scope = { account = @@ -1391,7 +1352,11 @@ let test_real_auth_control ~transition () = Atomic.set pause false; if transition = `Late_restore then ( control_wait "local reset closes network permission" (fun () -> - drain (); !account_generation > initial_account_generation && not !authenticated); + drain (); + !account_generation > initial_account_generation + && Option.exists + (fun startup -> startup.Service.network_phase = Service.Local_only) + !current_startup); permit_online (); reconcile ()); let retry_permitted = ref false in @@ -1433,7 +1398,7 @@ let test_real_auth_control ~transition () = ;; let test_local_copy_presence mode () = - let module Core = Logseq_sync_pure_reducer.Core in + let module Core = Logseq_sync_domain.Core in let module G = Logseq_db_types.Graph_types in T.with_managed (fun fixture -> let root = fixture.config.Logseq_db_worker.Config.application_support_directory in @@ -1553,7 +1518,7 @@ let test_local_copy_presence mode () = reducer already emits Diagnostic. Corrupt only a synthetic mirror, then observe the real service's existing Client_state_changed/Diagnostics push. *) let test_restore_error_reaches_diagnostics () = - let module Core = Logseq_sync_pure_reducer.Core in + let module Core = Logseq_sync_domain.Core in let module G = Logseq_db_types.Graph_types in T.with_managed (fun fixture -> let root = fixture.config.Logseq_db_worker.Config.application_support_directory in @@ -1676,7 +1641,7 @@ let test_authoritative_detail_rows () = (* Integrity text originates below the pure reducer; this public publisher boundary owns validating and displaying its bounded carrier. *) let test_asset_failure_rows () = - let module A=Logseq_sync_pure_reducer.Asset_transfer in + let module A=Logseq_sync_asset_transfer.Core in let module U=Logseq_db_worker_pure_reducer.Asset_upload in let module D=Logseq_db_types.Failure_diagnostic in let uuid=Logseq_db_types.Graph_types.Uuid.of_string "94000000-0000-4000-8000-000000000799" |> Result.get_ok in diff --git a/logseq_db_worker/test/test_managed_sync_e2e.ml b/logseq_db_worker/test/test_managed_sync_e2e.ml index 46e968c1..7b60e119 100644 --- a/logseq_db_worker/test/test_managed_sync_e2e.ml +++ b/logseq_db_worker/test/test_managed_sync_e2e.ml @@ -14,7 +14,7 @@ module Worker_runtime = Logseq_db_worker_lui.Journal_worker_runtime module W = Journal_worker_coordination.Worker_api module Support = Managed_sync_e2e_support module Probe = Managed_sync_protocol_probe -module Sync_protocol = Logseq_sync_pure_reducer.Sync_protocol +module Sync_protocol = Logseq_sync_protocol.Sync_protocol module ID = Journal_worker_ids exception E2e_failure of string @@ -23,6 +23,25 @@ let fail format = Printf.ksprintf (fun message -> raise (E2e_failure message)) f let base_url = "https://api.logseq.io" let report_phase phase = prerr_endline ("managed sync deployed E2E phase: " ^ phase) +let safe_exception_class = function + | Invalid_argument _ -> "invalid-argument" + | Failure _ -> "failure" + | Assert_failure _ -> "assert-failure" + | Match_failure _ -> "match-failure" + | Not_found -> "not-found" + | End_of_file -> "end-of-file" + | Sys_error _ -> "sys-error" + | Unix.Unix_error _ -> "unix-error" + | Fun.Finally_raised _ -> "finally-raised" + | _ -> "unclassified-exception" +;; + +let report_exception exception_ backtrace = + prerr_endline + ("managed sync deployed E2E exception-class=" ^ safe_exception_class exception_); + prerr_string (Printexc.raw_backtrace_to_string backtrace) +;; + let rec remove_tree path = match Unix.lstat path with | { st_kind = Unix.S_DIR; _ } -> @@ -204,7 +223,7 @@ let fail_worker context phase reason = state.snapshot.applied_server_t , startup_failure_stage_name state.snapshot.startup.failure , sync_error_kind state.snapshot.last_error - , Option.fold ~none:"none" ~some:String.escaped state.snapshot.last_error ) + , Option.fold ~none:"none" ~some:(fun _ -> "redacted") state.snapshot.last_error ) in fail "client=%s phase=%s reason=%s sync=%s startup-failure=%s error-kind=%s graph=%s \ @@ -249,10 +268,17 @@ let handle_public_state context state = | Failed, Some During_local_restore when not context.recovery_started -> context.recovery_started <- true; (match snapshot.selected_graph with - | Some graph_id -> send_command context (Begin_online_recovery - {startup={account_generation=snapshot.startup.account_generation; - presentation_generation=snapshot.startup.presentation_generation}; - graph_id; graph_generation=snapshot.startup.graph_generation}) + | Some graph_id -> + send_command + context + (Begin_online_recovery + { startup = + { account_generation = snapshot.startup.account_generation + ; presentation_generation = snapshot.startup.presentation_generation + } + ; graph_id + ; graph_generation = snapshot.startup.graph_generation + }) | None -> fail_worker context "local-restore" "missing-selected-graph") | Failed, _ -> fail_worker context "sync" "failed" | _ -> ()); @@ -266,6 +292,12 @@ let handle_public_state context state = "catalog" (Printf.sprintf "target-unavailable-count-%d" (List.length snapshot.catalog)) | Ok (Some graph) -> + (match Sys.getenv_opt "LOGSEQ_DB_WORKER_E2E_ACTION" with + | Some "guarded-save" -> + if Sys.getenv_opt "LOGSEQ_DB_WORKER_E2E_EXPECTED_GRAPH_UUID" + <> Some (Logseq_db_types.Graph_types.Uuid.to_string graph.graph_id) + then fail_worker context "catalog" "expected-graph-mismatch" + | _ -> ()); context.saw_catalog <- true; (match context.selected_graph with | Some selected @@ -278,27 +310,43 @@ let handle_public_state context state = | None -> ())); if snapshot.startup.awaiting_e2ee_password && not context.password_submitted then ( - context.password_submitted <- true; - send_command context (Submit_e2ee_password - (Logseq_sync_pure_reducer.Core.unlock_challenge snapshot.unlock |> Option.get, - context.credentials.e2ee_password))) + let command = + match Logseq_sync_domain.Core.unlock_challenge snapshot.unlock with + | Some challenge -> + Some (Service.Submit_e2ee_password (challenge, context.credentials.e2ee_password)) + | None -> + List.find_opt + (fun (scope : Service.Unlock.graph_scope) -> + Some scope.graph_id = snapshot.selected_graph + && scope.graph_generation = snapshot.startup.graph_generation + && scope.account.account_generation = snapshot.startup.account_generation) + snapshot.resident_scopes + |> Option.map (fun scope -> + Service.Submit_native_password (scope, context.credentials.e2ee_password)) + in + Option.iter + (fun command -> + context.password_submitted <- true; + send_command context command) + command) ;; let handle_graph_state context state = context.graph_phase <- state.Logseq_db_worker.phase; context.last_event <- "graph-state"; - if state.phase=Logseq_db_worker.Graph_open && not context.coordination_started then ( - let module I=(val context.issuer:Worker_effect_authority.ISSUER)in - context.coordination_started<-true; - ignore(send context(I.bootstrap_request()|>Result.get_ok))) + if state.phase = Logseq_db_worker.Graph_open && not context.coordination_started + then ( + let module I = (val context.issuer : Worker_effect_authority.ISSUER) in + context.coordination_started <- true; + ignore (send context (I.bootstrap_request () |> Result.get_ok))) ;; let accept_baseline context value = - let module I=(val context.issuer:Worker_effect_authority.ISSUER)in - match I.bootstrap(Service.Coordination_registered value)with - |Ok(W.Ready projection)->context.projection<-Some projection - |Ok(W.Awaiting_baseline|W.Bootstrap_not_ready _)->context.projection<-None - |_->fail_worker context "coordination-bootstrap" "invalid-authoritative-baseline" + let module I = (val context.issuer : Worker_effect_authority.ISSUER) in + match I.bootstrap (Service.Coordination_registered value) with + | Ok (W.Ready projection) -> context.projection <- Some projection + | Ok (W.Awaiting_baseline | W.Bootstrap_not_ready _) -> context.projection <- None + | _ -> fail_worker context "coordination-bootstrap" "invalid-authoritative-baseline" ;; let handle_push context = function @@ -317,9 +365,15 @@ let handle_push context = function | Asset_notice _ | Asset_notices _ -> fail_worker context "assets" "unexpected asset notice without demand" | Graph_push _ | Graph_pushes _ -> context.last_event <- "graph-push" - | Graph_states states -> List.iter (fun (_,state)->handle_graph_state context state) states + | Graph_states states -> + List.iter (fun (_, state) -> handle_graph_state context state) states | Session_states _ -> () - | Coordination_updates updates->List.iter(function Service.Coordination_snapshot value->accept_baseline context value|_->())updates + | Coordination_updates updates -> + List.iter + (function + | Service.Coordination_snapshot value -> accept_baseline context value + | _ -> ()) + updates | Coordination_observations _ -> () ;; @@ -333,15 +387,48 @@ let handle_event context = function | Response { outcome = Completed (Service.Local_copy_inspected _); _ } | Response { outcome = Completed (Service.Local_copy_deleted _); _ } -> fail_worker context "local-copy" "unexpected local-copy response" - | Response { outcome = Completed (Service.Coordination_asset_demand_finalized _ | Coordination_observation_finalized _ | Coordination_confirmed _ | Coordination_changes_acknowledged _ | Coordination_released _ | Coordination_observed _ | Coordination_observation_released _ | Coordination_mutations_recovered _ | Coordination_mutation_acknowledged _ | Coordination_asset_demand_accepted _ | Coordination_asset_demand_released _ | Coordination_imported _ | Coordination_writers_quiesced _ | Coordination_host_prepared _ | Coordination_host_applied _ | Coordination_host_confirmed _ | Coordination_import_inputs_recovered _ | Coordination_import_input_confirmed _ | Coordination_import_input_acknowledged _); _ } -> - fail_worker context "coordination" "unexpected unowned coordination response" + | Response + { outcome = + Completed + ( Service.Coordination_asset_demand_finalized _ + | Coordination_observation_finalized _ + | Coordination_confirmed _ + | Coordination_changes_acknowledged _ + | Coordination_released _ + | Coordination_observed _ + | Coordination_observation_released _ + | Coordination_mutations_recovered _ + | Coordination_mutation_acknowledged _ + | Coordination_asset_demand_accepted _ + | Coordination_asset_demand_released _ + | Coordination_imported _ + | Coordination_writers_quiesced _ + | Coordination_host_prepared _ + | Coordination_host_applied _ + | Coordination_host_confirmed _ + | Coordination_import_inputs_recovered _ + | Coordination_import_input_confirmed _ + | Coordination_import_input_acknowledged _ ) + ; _ + } -> fail_worker context "coordination" "unexpected unowned coordination response" | Response { outcome = Completed Service.Client_command_completed; _ } -> None | Response { outcome = Completed (Service.Graph_state state); _ } -> handle_graph_state context state; None - |Response{outcome=Completed(Service.Coordination_registered value);_}->accept_baseline context value;None - |Response{request_id;outcome=Completed((Service.Coordination_queried _|Coordination_query_revoked _|Coordination_mutation_result _)as response);_}-> - context.last_event<-"typed-response";Some(request_id,response) + | Response { outcome = Completed (Service.Coordination_registered value); _ } -> + accept_baseline context value; + None + | Response + { request_id + ; outcome = + Completed + (( Service.Coordination_queried _ + | Coordination_query_revoked _ + | Coordination_mutation_result _ ) as response) + ; _ + } -> + context.last_event <- "typed-response"; + Some (request_id, response) | Response { outcome = Failed _; _ } -> fail_worker context "worker-request" "failed" | Response { outcome = Cancelled; _ } -> fail_worker context "worker-request" "cancelled" @@ -372,26 +459,31 @@ let await_response context ~phase request_id = loop () ;; -let opening_complete context = +let opening_complete ~require_bootstrap context = match context.last_state with | Some state -> state.snapshot.sync_phase = Current && Option.is_some state.snapshot.applied_server_t && context.saw_catalog - && context.saw_bootstrap + && ((not require_bootstrap) || context.saw_bootstrap) && context.saw_opening_pull && context.graph_phase = Graph_open - && Option.exists(fun projection->List.exists(fun resident->resident.W.phase=W.Open)(W.residents projection))context.projection + && Option.exists + (fun projection -> + List.exists + (fun resident -> resident.W.phase = W.Open) + (W.residents projection)) + context.projection | None -> false ;; -let await_opening_current ?(timeout_seconds = 300.) context = +let await_opening_current ?(timeout_seconds = 300.) ?(require_bootstrap = true) context = let deadline = timeout_at timeout_seconds in let rec loop () = if Unix.gettimeofday () >= deadline then fail_worker context "cold-bootstrap" "timeout"; ignore (drain context : (ID.Worker.Request_id.t * Service.response) list); - if opening_complete context + if opening_complete ~require_bootstrap context then ( match context.last_state, context.selected_graph with | Some state, Some graph -> Option.get state.snapshot.applied_server_t, graph @@ -424,39 +516,72 @@ let await_authoritative_current context ~after_server_t phase = ;; let current_session context = - match context.projection,context.selected_graph with - |Some projection,Some graph-> - (match List.find_opt(fun resident->resident.W.phase=W.Open && W.graph_id resident.session=graph.graph_id)(W.residents projection)with - |Some resident->resident.session|None->fail_worker context "typed-request" "graph-session-unavailable") - |_->fail_worker context "typed-request" "bootstrap-not-ready" -;; -let execute :type a. client_context -> W.session -> a W.request -> a = fun context session operation-> - let module I=(val context.issuer:Worker_effect_authority.ISSUER)in - let request_id=W.request_id(fresh_uuid())in - let request=match I.request session ~request_id operation with - |Ok value->value|Error _->fail_worker context "typed-request" "admission-rejected"in - let response=await_response context ~phase:"typed-request" (send context request)in + match context.projection, context.selected_graph with + | Some projection, Some graph -> + (match + List.find_opt + (fun resident -> + resident.W.phase = W.Open && W.graph_id resident.session = graph.graph_id) + (W.residents projection) + with + | Some resident -> resident.session + | None -> fail_worker context "typed-request" "graph-session-unavailable") + | _ -> fail_worker context "typed-request" "bootstrap-not-ready" +;; + +let execute : type a. client_context -> W.session -> a W.request -> a = + fun context session operation -> + let module I = (val context.issuer : Worker_effect_authority.ISSUER) in + let request_id = W.request_id (fresh_uuid ()) in + let request = + match I.request session ~request_id operation with + | Ok value -> value + | Error _ -> fail_worker context "typed-request" "admission-rejected" + in + let response = await_response context ~phase:"typed-request" (send context request) in match I.reply session ~request_id operation response with - |Ok(W.Answered(Ok value))->value - |_->fail_worker context "typed-request" "response-rejected" -;; -let query :type a. ?maximum_items:int -> client_context -> a W.query -> a = - fun ?(maximum_items=200) context query-> - let session=current_session context in - let budget=W.read_budget session ~maximum_items ~maximum_bytes:Worker_pure_data.maximum_response_bytes|>Result.get_ok in - execute context session(W.Query(budget,query)) + | Ok (W.Answered (Ok value)) -> value + | _ -> fail_worker context "typed-request" "response-rejected" +;; + +let query : type a. ?maximum_items:int -> client_context -> a W.query -> a = + fun ?(maximum_items = 200) context query -> + let session = current_session context in + let budget = + W.read_budget + session + ~maximum_items + ~maximum_bytes:Worker_pure_data.maximum_response_bytes + |> Result.get_ok + in + execute context session (W.Query (budget, query)) ;; + let journal_parent context = - let result=query ~maximum_items:50 context(W.Journals{ - from_day=W.journal_day 10101|>Result.get_ok;through_day=W.journal_day 99991231|>Result.get_ok; - revision=None;after=None})in - match List.find_opt(fun(item:Worker_pure_data.journal_item)->not item.page.recycled)result.items with - |Some item->item.page.uuid - |None->fail_worker context "journal-read" "no-active-journal" + let result = + query + ~maximum_items:50 + context + (W.Journals + { from_day = W.journal_day 10101 |> Result.get_ok + ; through_day = W.journal_day 99991231 |> Result.get_ok + ; revision = None + ; after = None + }) + in + match + List.find_opt + (fun (item : Worker_pure_data.journal_item) -> not item.page.recycled) + result.items + with + | Some item -> item.page.uuid + | None -> fail_worker context "journal-read" "no-active-journal" ;; -let read_block context block_uuid=match query context(W.Outline_block{uuid=block_uuid;revision=None})with - |Worker_pure_data.V2_present_block{value;revision}->Ok(value,revision) - |V2_missing_block _->Error() + +let read_block context block_uuid = + match query context (W.Outline_block { uuid = block_uuid; revision = None }) with + | Worker_pure_data.V2_present_block { value; revision } -> Ok (value, revision) + | V2_missing_block _ -> Error () ;; let require_block context ~block_uuid ~parent_uuid ~title = @@ -472,39 +597,76 @@ let require_block context ~block_uuid ~parent_uuid ~title = block, basis ;; -let run_mutation ?(on_applied=fun()->()) context ~after_server_t phase ~mutation_id command = - report_phase(phase^"-request"); - let session=current_session context in - let intent=W.mutation_intent session ~mutation_id command|>Result.get_ok in - (match execute context session(W.Mutate intent)with - |W.Submission_committed receipt-> - let _,before,after=W.receipt_revisions receipt in - if before=after then fail_worker context phase "mutation-not-applied"; - report_phase(phase^"-local-commit");on_applied() - |_->fail_worker context phase "mutation-not-applied"); - let cursor=await_authoritative_current context ~after_server_t phase in - report_phase(phase^"-authoritative");cursor +let run_mutation + ?(on_applied = fun () -> ()) + context + ~after_server_t + phase + ~mutation_id + command + = + report_phase (phase ^ "-request"); + let session = current_session context in + let intent = W.mutation_intent session ~mutation_id command |> Result.get_ok in + (match execute context session (W.Mutate intent) with + | W.Submission_committed receipt -> + let _, before, after = W.receipt_revisions receipt in + if before = after then fail_worker context phase "mutation-not-applied"; + report_phase (phase ^ "-local-commit"); + on_applied () + | _ -> fail_worker context phase "mutation-not-applied"); + let cursor = await_authoritative_current context ~after_server_t phase in + report_phase (phase ^ "-authoritative"); + cursor ;; + let insert_preconditions context parent = - let page_revision=match query context(W.Page{uuid=parent;revision=None})with - |Worker_pure_data.V2_present_page{revision;_}->revision - |_->fail_worker context "insert-precondition" "parent-page-missing"in - let children=query context(W.Child_members{parent;revision=None;after=None})in - Worker_pure_data.{blocks=[];pages=[parent,page_revision]; - scopes=[V2_children_scope parent,children.scope_revision]} + let page_revision = + match query context (W.Page { uuid = parent; revision = None }) with + | Worker_pure_data.V2_present_page { revision; _ } -> revision + | _ -> fail_worker context "insert-precondition" "parent-page-missing" + in + let children = + query context (W.Child_members { parent; revision = None; after = None }) + in + Worker_pure_data. + { blocks = [] + ; pages = [ parent, page_revision ] + ; scopes = [ V2_children_scope parent, children.scope_revision ] + } ;; + let insert_block context ~block_uuid ~parent_uuid ~title ~after_server_t ~on_applied = - run_mutation ~on_applied context ~after_server_t "insert-block" ~mutation_id:(fresh_uuid()) - (W.Insert_blocks{parent=parent_uuid;roots=[{uuid=block_uuid;title;children=[]}]; - preconditions=insert_preconditions context parent_uuid}) + run_mutation + ~on_applied + context + ~after_server_t + "insert-block" + ~mutation_id:(fresh_uuid ()) + (W.Insert_blocks + { parent = parent_uuid + ; roots = [ { uuid = block_uuid; title; children = [] } ] + ; preconditions = insert_preconditions context parent_uuid + }) ;; + let delete_block context ~block_uuid ~after_server_t = report_phase "delete-precondition-start"; - let _,revision=match read_block context block_uuid with - |Ok value->value|Error()->fail_worker context "delete-precondition" "block-missing"in + let _, revision = + match read_block context block_uuid with + | Ok value -> value + | Error () -> fail_worker context "delete-precondition" "block-missing" + in report_phase "delete-precondition-complete"; - run_mutation context ~after_server_t "delete-block" ~mutation_id:(fresh_uuid()) - (W.Delete_blocks{root=block_uuid;preconditions={blocks=[block_uuid,revision];pages=[];scopes=[]}}) + run_mutation + context + ~after_server_t + "delete-block" + ~mutation_id:(fresh_uuid ()) + (W.Delete_blocks + { root = block_uuid + ; preconditions = { blocks = [ block_uuid, revision ]; pages = []; scopes = [] } + }) ;; let raw_transaction ~mutation_id ~operation operations = @@ -622,7 +784,8 @@ let raw_batch ~t_before transactions = let probe_require_ok phase = function | Ok value -> value - | Error message -> fail "deployed protocol phase=%s reason=%s" phase message + | Error message -> + fail "deployed protocol phase=%s error-kind=%s" phase (sync_error_kind (Some message)) ;; let probe_hello clock connection = @@ -631,7 +794,7 @@ let probe_hello clock connection = ~timeout_seconds:30. (function | Sync_protocol.Server.Hello { t; _ } -> Some t - | Error { message } -> fail "deployed protocol hello failed: %s" message + | Error _ -> fail "deployed protocol hello failed: server-error" | _ -> None) connection |> probe_require_ok "hello" @@ -648,7 +811,7 @@ let probe_batch_result clock connection = (function | Sync_protocol.Server.Tx_batch_ok { t; checksum } -> Some (`Accepted (t, checksum)) | Tx_reject rejection -> Some (`Rejected rejection) - | Error { message } -> fail "deployed protocol batch failed: %s" message + | Error _ -> fail "deployed protocol batch failed: server-error" | _ -> None) connection |> probe_require_ok "batch-result" @@ -673,7 +836,10 @@ let probe_expect_accepted clock connection phase = phase (rejection_reason_name rejection.Sync_protocol.reason) (List.length rejection.missing_block_uuids) - (Option.fold ~none:"none" ~some:String.escaped rejection.Sync_protocol.error_detail) + (Option.fold + ~none:"none" + ~some:(fun _ -> "redacted") + rejection.Sync_protocol.error_detail) ;; let probe_pull clock connection ~since = @@ -683,7 +849,7 @@ let probe_pull clock connection ~since = ~timeout_seconds:30. (function | Sync_protocol.Server.Pull_ok { t; checksum; txs } -> Some (t, checksum, txs) - | Error { message } -> fail "deployed protocol pull failed: %s" message + | Error _ -> fail "deployed protocol pull failed: server-error" | _ -> None) connection |> probe_require_ok "pull-result" @@ -752,206 +918,267 @@ let run_deployed_protocol_contract ~source_cursor = report_phase "protocol-contract-start"; - Eio_main.run (fun environment -> - Eio.Switch.run (fun sw -> - let network = Eio.Stdenv.net environment in - let clock = Eio.Stdenv.clock environment in - let first = - Probe.connect ~sw ~network ~clock ~base_url ~graph_id ~token - |> probe_require_ok "connect-first" - in - let second = - Probe.connect ~sw ~network ~clock ~base_url ~graph_id ~token - |> probe_require_ok "connect-second" - in - Fun.protect - ~finally:(fun () -> - Probe.close first; - Probe.close second) - (fun () -> - probe_send - first - (Sync_protocol.Client.Hello { client = "logseq-journal-e2e-first" }) - "hello-first"; - probe_send - second - (Sync_protocol.Client.Hello { client = "logseq-journal-e2e-second" }) - "hello-second"; - let first_t = probe_hello clock first in - let second_t = probe_hello clock second in - let baseline = - if Logseq_db_types.Server_cursor.compare first_t second_t >= 0 - then first_t - else second_t - in - let _, _, source_history = probe_pull clock first ~since:source_cursor in - let protected_title = - match - protected_title_from_history ~block_uuid:source_block_uuid source_history - with - | Some title -> title - | None -> fail "deployed Pull omitted the source protected title" - in - let _, _, _ = probe_pull clock first ~since:baseline in - let _, _, _ = probe_pull clock second ~since:baseline in - let accepted_block = fresh_uuid () in - let second_accepted_block = fresh_uuid () in - let first_id = fresh_uuid () in - let second_id = fresh_uuid () in - let first_transaction = - raw_marker_create - ~mutation_id:first_id - ~block_uuid:accepted_block - ~parent_uuid - ~protected_title - in - let second_transaction = - raw_marker_create - ~mutation_id:second_id - ~block_uuid:second_accepted_block - ~parent_uuid - ~protected_title - in - let group = - raw_batch ~t_before:baseline [ first_transaction; second_transaction ] - in - probe_send first group "group-submit"; - let group_t, _ = probe_expect_accepted clock first "group-submit" in - if - not - (Logseq_db_types.Server_cursor.equal - group_t - (advance_server_cursor baseline 2)) - then fail "deployed group cursors are not contiguous in member order"; - let pulled_t, _, pulled = probe_pull clock second ~since:baseline in - if Logseq_db_types.Server_cursor.compare pulled_t group_t < 0 - then fail "deployed pull did not reach the accepted group"; - let group_members = - pulled - |> List.filter (fun transaction -> - Logseq_db_types.Server_cursor.compare - transaction.Sync_protocol.Server.t - baseline - > 0 - && Logseq_db_types.Server_cursor.compare transaction.t group_t <= 0) - in - if - List.map (fun tx -> tx.Sync_protocol.Server.t) group_members - <> [ advance_server_cursor baseline 1; advance_server_cursor baseline 2 ] - then fail "deployed group interval was interleaved or reordered"; - List.iter2 - (fun (expected_uuid, expected) transaction -> - if not (transaction_adds_block_uuid ~block_uuid:expected_uuid transaction) - then fail "deployed Pull omitted a normalized group member"; - if - transaction.Sync_protocol.Server.outliner_op - <> expected.Sync_protocol.Client.outliner_op - then fail "deployed Pull changed the separate outliner operation") - [ accepted_block, first_transaction - ; second_accepted_block, second_transaction - ] - group_members; - probe_send first group "stale-retry"; - (match probe_batch_result clock first with - | `Rejected { reason = Sync_protocol.Stale; t = Some retry_t; _ } - when Logseq_db_types.Server_cursor.equal retry_t group_t -> () - | `Rejected _ -> fail "deployed retry did not return the current stale cursor" - | `Accepted _ -> fail "deployed retry unexpectedly executed twice"); - let accepted_delete_id = fresh_uuid () in - let accepted_delete = - raw_batch - ~t_before:group_t - [ raw_marker_delete ~mutation_id:accepted_delete_id accepted_block ] - in - probe_send first accepted_delete "accepted-before-remote-delete"; - let delete_t, _ = - probe_expect_accepted clock first "accepted-before-remote-delete" - in - let later_remote_id = fresh_uuid () in - probe_send - second - (raw_batch - ~t_before:delete_t - [ raw_marker_touch - ~mutation_id:later_remote_id - ~block_uuid:second_accepted_block - ~updated_at:1_704_067_200_001 - ]) - "remote-after-delete"; - let later_remote_t, _ = - probe_expect_accepted clock second "remote-after-delete" - in - if Logseq_db_types.Server_cursor.compare delete_t later_remote_t >= 0 - then fail "deployed accepted delete did not precede the remote transaction"; - let stale_block = fresh_uuid () in - let create_stale_id = fresh_uuid () in - probe_send - first - (raw_batch - ~t_before:later_remote_t - [ raw_marker_create - ~mutation_id:create_stale_id - ~block_uuid:stale_block - ~parent_uuid - ~protected_title - ]) - "stale-marker-create"; - let stale_baseline, _ = - probe_expect_accepted clock first "stale-marker-create" - in - let intervening_id = fresh_uuid () in - probe_send - second - (raw_batch - ~t_before:stale_baseline - [ raw_marker_touch - ~mutation_id:intervening_id - ~block_uuid:second_accepted_block - ~updated_at:1_704_067_200_002 - ]) - "intervening-remote"; - let intervening_t, _ = - probe_expect_accepted clock second "intervening-remote" - in - let stale_delete_id = fresh_uuid () in - probe_send - first - (raw_batch - ~t_before:stale_baseline - [ raw_marker_delete ~mutation_id:stale_delete_id stale_block ]) - "stale-delete"; - (match probe_batch_result clock first with - | `Rejected - { reason = Sync_protocol.Stale - ; t = Some rejection_t - ; success_tx_ids = [] - ; failed_tx_id = None - ; _ - } - when Logseq_db_types.Server_cursor.compare rejection_t intervening_t >= 0 -> - () - | `Rejected _ -> - fail "deployed stale delete lacked batch-level non-execution evidence" - | `Accepted _ -> fail "deployed stale t_before delete executed"); - probe_send - first - (raw_batch - ~t_before:intervening_t - [ raw_marker_delete ~mutation_id:(fresh_uuid ()) stale_block ]) - "stale-marker-cleanup"; - let stale_cleanup_t, _ = - probe_expect_accepted clock first "stale-marker-cleanup" - in - probe_send - first - (raw_batch - ~t_before:stale_cleanup_t - [ raw_marker_delete ~mutation_id:(fresh_uuid ()) second_accepted_block ]) - "second-group-marker-cleanup"; - ignore - (probe_expect_accepted clock first "second-group-marker-cleanup" - : Sync_protocol.cursor * string option)))); - report_phase "protocol-contract-complete" + let markers = + Eio_main.run (fun environment -> + Eio.Switch.run (fun sw -> + let network = Eio.Stdenv.net environment in + let clock = Eio.Stdenv.clock environment in + let first = + Probe.connect ~sw ~network ~clock ~base_url ~graph_id ~token + |> probe_require_ok "connect-first" + in + let second = + Probe.connect ~sw ~network ~clock ~base_url ~graph_id ~token + |> probe_require_ok "connect-second" + in + let pending_markers = ref [] in + let marker_ids = ref [] in + let baseline_cursor = ref None in + let remember markers = + pending_markers := markers @ !pending_markers; + marker_ids := markers @ !marker_ids + in + let forget uuid = + pending_markers + := List.filter + (fun marker -> not (Logseq_db_types.Graph_types.Uuid.equal marker uuid)) + !pending_markers + in + Fun.protect + ~finally:(fun () -> + Fun.protect + ~finally:(fun () -> + Probe.close first; + Probe.close second) + (fun () -> + match !pending_markers, !baseline_cursor with + | [], _ -> () + | markers, Some baseline -> + List.iter + (fun uuid -> + try + let cursor, _, _ = probe_pull clock first ~since:baseline in + probe_send + first + (raw_batch + ~t_before:cursor + [ raw_marker_delete ~mutation_id:(fresh_uuid ()) uuid ]) + "protocol-finally-cleanup"; + ignore + (probe_expect_accepted clock first "protocol-finally-cleanup" + : Sync_protocol.cursor * string option); + forget uuid + with + | exception_ -> + let backtrace = Printexc.get_raw_backtrace () in + report_phase + ("protocol-cleanup-failed-marker-" + ^ Logseq_db_types.Graph_types.Uuid.to_string uuid); + report_exception exception_ backtrace) + markers + | markers, None -> + List.iter + (fun uuid -> + report_phase + ("protocol-cleanup-unavailable-marker-" + ^ Logseq_db_types.Graph_types.Uuid.to_string uuid)) + markers)) + (fun () -> + probe_send + first + (Sync_protocol.Client.Hello { client = "logseq-journal-e2e-first" }) + "hello-first"; + probe_send + second + (Sync_protocol.Client.Hello { client = "logseq-journal-e2e-second" }) + "hello-second"; + let first_t = probe_hello clock first in + let second_t = probe_hello clock second in + let baseline = + if Logseq_db_types.Server_cursor.compare first_t second_t >= 0 + then first_t + else second_t + in + baseline_cursor := Some baseline; + let _, _, source_history = probe_pull clock first ~since:source_cursor in + let protected_title = + match + protected_title_from_history ~block_uuid:source_block_uuid source_history + with + | Some title -> title + | None -> fail "deployed Pull omitted the source protected title" + in + let _, _, _ = probe_pull clock first ~since:baseline in + let _, _, _ = probe_pull clock second ~since:baseline in + let accepted_block = fresh_uuid () in + let second_accepted_block = fresh_uuid () in + let first_id = fresh_uuid () in + let second_id = fresh_uuid () in + let first_transaction = + raw_marker_create + ~mutation_id:first_id + ~block_uuid:accepted_block + ~parent_uuid + ~protected_title + in + let second_transaction = + raw_marker_create + ~mutation_id:second_id + ~block_uuid:second_accepted_block + ~parent_uuid + ~protected_title + in + let group = + raw_batch ~t_before:baseline [ first_transaction; second_transaction ] + in + remember [ accepted_block; second_accepted_block ]; + probe_send first group "group-submit"; + let group_t, _ = probe_expect_accepted clock first "group-submit" in + if + not + (Logseq_db_types.Server_cursor.equal + group_t + (advance_server_cursor baseline 2)) + then fail "deployed group cursors are not contiguous in member order"; + let pulled_t, _, pulled = probe_pull clock second ~since:baseline in + if Logseq_db_types.Server_cursor.compare pulled_t group_t < 0 + then fail "deployed pull did not reach the accepted group"; + let group_members = + pulled + |> List.filter (fun transaction -> + Logseq_db_types.Server_cursor.compare + transaction.Sync_protocol.Server.t + baseline + > 0 + && Logseq_db_types.Server_cursor.compare transaction.t group_t <= 0) + in + if + List.map (fun tx -> tx.Sync_protocol.Server.t) group_members + <> [ advance_server_cursor baseline 1; advance_server_cursor baseline 2 ] + then fail "deployed group interval was interleaved or reordered"; + List.iter2 + (fun (expected_uuid, expected) transaction -> + if + not + (transaction_adds_block_uuid ~block_uuid:expected_uuid transaction) + then fail "deployed Pull omitted a normalized group member"; + if + transaction.Sync_protocol.Server.outliner_op + <> expected.Sync_protocol.Client.outliner_op + then fail "deployed Pull changed the separate outliner operation") + [ accepted_block, first_transaction + ; second_accepted_block, second_transaction + ] + group_members; + probe_send first group "stale-retry"; + (match probe_batch_result clock first with + | `Rejected { reason = Sync_protocol.Stale; t = Some retry_t; _ } + when Logseq_db_types.Server_cursor.equal retry_t group_t -> () + | `Rejected _ -> + fail "deployed retry did not return the current stale cursor" + | `Accepted _ -> fail "deployed retry unexpectedly executed twice"); + let accepted_delete_id = fresh_uuid () in + let accepted_delete = + raw_batch + ~t_before:group_t + [ raw_marker_delete ~mutation_id:accepted_delete_id accepted_block ] + in + probe_send first accepted_delete "accepted-before-remote-delete"; + let delete_t, _ = + probe_expect_accepted clock first "accepted-before-remote-delete" + in + forget accepted_block; + let later_remote_id = fresh_uuid () in + probe_send + second + (raw_batch + ~t_before:delete_t + [ raw_marker_touch + ~mutation_id:later_remote_id + ~block_uuid:second_accepted_block + ~updated_at:1_704_067_200_001 + ]) + "remote-after-delete"; + let later_remote_t, _ = + probe_expect_accepted clock second "remote-after-delete" + in + if Logseq_db_types.Server_cursor.compare delete_t later_remote_t >= 0 + then fail "deployed accepted delete did not precede the remote transaction"; + let stale_block = fresh_uuid () in + let create_stale_id = fresh_uuid () in + remember [ stale_block ]; + probe_send + first + (raw_batch + ~t_before:later_remote_t + [ raw_marker_create + ~mutation_id:create_stale_id + ~block_uuid:stale_block + ~parent_uuid + ~protected_title + ]) + "stale-marker-create"; + let stale_baseline, _ = + probe_expect_accepted clock first "stale-marker-create" + in + let intervening_id = fresh_uuid () in + probe_send + second + (raw_batch + ~t_before:stale_baseline + [ raw_marker_touch + ~mutation_id:intervening_id + ~block_uuid:second_accepted_block + ~updated_at:1_704_067_200_002 + ]) + "intervening-remote"; + let intervening_t, _ = + probe_expect_accepted clock second "intervening-remote" + in + let stale_delete_id = fresh_uuid () in + probe_send + first + (raw_batch + ~t_before:stale_baseline + [ raw_marker_delete ~mutation_id:stale_delete_id stale_block ]) + "stale-delete"; + (match probe_batch_result clock first with + | `Rejected + { reason = Sync_protocol.Stale + ; t = Some rejection_t + ; success_tx_ids = [] + ; failed_tx_id = None + ; _ + } + when Logseq_db_types.Server_cursor.compare rejection_t intervening_t >= 0 + -> () + | `Rejected _ -> + fail "deployed stale delete lacked batch-level non-execution evidence" + | `Accepted _ -> fail "deployed stale t_before delete executed"); + probe_send + first + (raw_batch + ~t_before:intervening_t + [ raw_marker_delete ~mutation_id:(fresh_uuid ()) stale_block ]) + "stale-marker-cleanup"; + let stale_cleanup_t, _ = + probe_expect_accepted clock first "stale-marker-cleanup" + in + forget stale_block; + probe_send + first + (raw_batch + ~t_before:stale_cleanup_t + [ raw_marker_delete ~mutation_id:(fresh_uuid ()) second_accepted_block ]) + "second-group-marker-cleanup"; + ignore + (probe_expect_accepted clock first "second-group-marker-cleanup" + : Sync_protocol.cursor * string option); + forget second_accepted_block; + !marker_ids))) + in + report_phase "protocol-contract-complete"; + markers ;; let config support = @@ -994,14 +1221,20 @@ let with_client ~label ~support ~credentials ~cognito f = client | Error _ -> fail "client=%s phase=start reason=failed" label in - let issuer=Worker_effect_authority.issuer ~managed_origin:base_url ~user_id:cognito.Support.user_id - ~runtime_epoch:(Worker.runtime_epoch client) ~worker_generation:(Worker.worker_generation client)|>Result.get_ok in + let issuer = + Worker_effect_authority.issuer + ~managed_origin:base_url + ~user_id:cognito.Support.user_id + ~runtime_epoch:(Worker.runtime_epoch client) + ~worker_generation:(Worker.worker_generation client) + |> Result.get_ok + in let context = { label ; client ; issuer - ; projection=None - ; coordination_started=false + ; projection = None + ; coordination_started = false ; credentials ; cognito ; token_requests = Hashtbl.create 8 @@ -1022,7 +1255,9 @@ let with_client ~label ~support ~credentials ~cognito f = if not (Worker.For_testing.is_stopping client) then Worker_runtime.stop client; await_runtime_idle label) (fun () -> - send_command context (Request_sign_in {account_generation=0; presentation_generation=0}); + send_command + context + (Request_sign_in { account_generation = 0; presentation_generation = 0 }); send_command context (Reconcile_authenticated_user { user_id = Some cognito.user_id }); @@ -1076,7 +1311,13 @@ let require_durable_advance facts initial_cursor label = then fail "%s durable cursor did not advance" label; if String.equal facts.checksum "0000000000000000" then fail "%s durable checksum did not advance" label; - if not facts.outbox_empty then fail "%s durable outbox is not empty" label + if not facts.outbox_empty then fail "%s durable outbox is not empty" label; + report_phase + (Printf.sprintf + "%s-durable-checkpoint-%Ld-to-%Ld-outbox-empty" + label + (Logseq_db_types.Server_cursor.to_int64 initial_cursor) + (Logseq_db_types.Server_cursor.to_int64 facts.cursor)) ;; let cleanup_block context ~block_uuid ~parent_uuid ~title ~opening_cursor = @@ -1112,7 +1353,12 @@ let best_effort_cleanup : Logseq_db_types.Server_cursor.t)); report_phase "best-effort-cleanup-complete" with - | _ -> report_phase "best-effort-cleanup-failed" + | exception_ -> + let backtrace = Printexc.get_raw_backtrace () in + report_phase + ("best-effort-cleanup-failed-marker-" + ^ Logseq_db_types.Graph_types.Uuid.to_string block_uuid); + report_exception exception_ backtrace ;; let require_macos () = @@ -1338,28 +1584,171 @@ let run () = receiver_final_cursor < 0 then fail "receiver durable cursor is behind public state"; - run_deployed_protocol_contract - ~token:cognito.id_token - ~graph_id:receiver_graph.graph_id - ~parent_uuid - ~source_block_uuid:block_uuid - ~source_cursor:sender_opening_cursor))) + let protocol_markers = + run_deployed_protocol_contract + ~token:cognito.id_token + ~graph_id:receiver_graph.graph_id + ~parent_uuid + ~source_block_uuid:block_uuid + ~source_cursor:sender_opening_cursor + in + with_client + ~label:"durable-recovery" + ~support:receiver_support + ~credentials + ~cognito + (fun recovery -> + let cursor, graph = + await_opening_current + ~timeout_seconds:60. + ~require_bootstrap:false + recovery + in + if + not + (Logseq_db_types.Graph_types.Uuid.equal + graph.graph_id + receiver_graph.graph_id) + then fail_worker recovery "durable-recovery" "graph-mismatch"; + if Logseq_db_types.Server_cursor.compare cursor receiver_facts.cursor < 0 + then fail_worker recovery "durable-recovery" "checkpoint-regressed"; + List.iter + (fun uuid -> + match read_block recovery uuid with + | Error () -> () + | Ok _ -> + fail + "remote cleanup verification failed marker=%s" + (Logseq_db_types.Graph_types.Uuid.to_string uuid)) + (block_uuid :: protocol_markers); + report_phase "durable-recovery-all-four-owned-markers-absent"); + let recovered_facts = durable_facts receiver_support receiver_graph.graph_id in + require_durable_advance + recovered_facts + receiver_opening_cursor + "recovered-receiver"))) +;; + +let run_guarded_save () = + require_macos (); + let required name = + match Sys.getenv_opt ("LOGSEQ_DB_WORKER_E2E_" ^ name) with + | Some value when String.length value > 0 && String.length value <= 4096 + && not (String.contains value '\000') -> value + | _ -> fail "missing or invalid guarded input: %s" name + in + let uuid name = + match Logseq_db_types.Graph_types.Uuid.of_string (required name) with + | Ok value -> value + | Error _ -> fail "invalid guarded UUID: %s" name + in + let graph_id = uuid "EXPECTED_GRAPH_UUID" in + let block_uuid = uuid "OWNED_BLOCK_UUID" in + let parent_uuid = uuid "EXPECTED_PARENT_UUID" in + let expected = required "EXPECTED_BODY" in + let prefix = required "SELF_PREFIX" in + let title = required "NEW_BODY" in + if String.length prefix < 16 + || not (String.starts_with ~prefix expected) + || not (String.starts_with ~prefix:expected title) + || String.length title <= String.length expected + || String.length title > 512 + || String.length title - String.length expected > 64 + then fail "guarded body bounds or ownership prefix rejected"; + let credentials = + match Support.credentials_from_environment Sys.getenv_opt with + | Ok value -> value + | Error message -> fail "%s" message + in + let cognito = + match Support.authenticate credentials with + | Ok value -> value + | Error message -> fail "%s" message + in + let verify context body = + let block, revision = require_block context ~block_uuid ~parent_uuid ~title:body in + let status = + match block.task_status with + | Some value -> value + | None -> fail "guarded block is not a task" + in + (match query context (W.Task_status_definitions [ status ]) with + | [ Some definition ] when String.equal definition.title "Done" -> () + | _ -> fail "guarded task is not Done"); + let children = + query ~maximum_items:1 context + (W.Child_members { parent = block_uuid; revision = None; after = None }) + in + if children.items <> [] then fail "guarded owned block has children"; + status, revision + in + with_temp_directory "logseq-guarded-save-" (fun support -> + let initial, acknowledged, status = + with_client ~label:"guarded-save" ~support ~credentials ~cognito (fun context -> + let initial, graph = await_opening_current context in + if not (Logseq_db_types.Graph_types.Uuid.equal graph.graph_id graph_id) + then fail "guarded graph mismatch"; + let status, revision = verify context expected in + report_phase "guarded-save-guards-passed-Done-children-empty"; + let acknowledged = + run_mutation context ~after_server_t:initial "guarded-save" + ~mutation_id:(fresh_uuid ()) + (W.Save_block + { block = block_uuid; title + ; preconditions = + { blocks = [ block_uuid, revision ]; pages = []; scopes = [] } + }) + in + let after_status, _ = verify context title in + if after_status <> status then fail "guarded task status changed"; + initial, acknowledged, status) + in + let facts = durable_facts support graph_id in + require_durable_advance facts initial "guarded-save"; + if Logseq_db_types.Server_cursor.compare facts.cursor acknowledged < 0 + then fail "guarded durable checkpoint behind acknowledgement"; + report_phase + (Printf.sprintf "guarded-save-checkpointAck-%Ld-Done-preserved" + (Logseq_db_types.Server_cursor.to_int64 acknowledged)); + with_temp_directory "logseq-guarded-incoming-" (fun receiver_support -> + with_client ~label:"guarded-incoming" ~support:receiver_support + ~credentials ~cognito (fun context -> + let cursor, _ = await_opening_current context in + if Logseq_db_types.Server_cursor.compare cursor acknowledged < 0 + then fail "incoming checkpoint behind acknowledgement"; + let incoming_status, _ = verify context title in + if incoming_status <> status then fail "incoming task status changed"; + report_phase "guarded-incoming-exact-body-Done-preserved"); + let incoming = durable_facts receiver_support graph_id in + if not incoming.outbox_empty then fail "incoming durable outbox not empty"; + report_phase "guarded-incoming-durable-outbox-empty")) +;; + +let selected_run () = + match Sys.getenv_opt "LOGSEQ_DB_WORKER_E2E_ACTION" with + | None -> run () + | Some "guarded-save" -> run_guarded_save () + | Some _ -> fail "unknown managed sync E2E action" ;; let () = + Printexc.record_backtrace true; let status = try if Sys.getenv_opt platform_crypto_provider_marker = Some "1" then ( - with_execution_lock run; + with_execution_lock selected_run; 0) else run_with_platform_crypto_provider () with | E2e_failure message -> + let backtrace = Printexc.get_raw_backtrace () in prerr_endline ("managed sync deployed E2E failed: " ^ message); + prerr_string (Printexc.raw_backtrace_to_string backtrace); 2 - | _ -> - prerr_endline "managed sync deployed E2E failed: unexpected failure"; + | exception_ -> + let backtrace = Printexc.get_raw_backtrace () in + report_exception exception_ backtrace; 2 in if !runtime_started then Worker_runtime.For_testing.final_shutdown (); diff --git a/logseq_db_worker/test/test_pure_reducer.ml b/logseq_db_worker/test/test_pure_reducer.ml index 1d245392..c90fc96e 100644 --- a/logseq_db_worker/test/test_pure_reducer.ml +++ b/logseq_db_worker/test/test_pure_reducer.ml @@ -1,18 +1,18 @@ module Core = Logseq_db_worker_pure_reducer.Core -module Sync = Logseq_sync_pure_reducer.Core +module Sync = Logseq_sync_domain.Core module Config = Logseq_db_worker.Config let uuid value = Logseq_db_types.Graph_types.Uuid.of_string value |> Result.get_ok module Read=Worker_pure_read -let observation_read (scope:Logseq_sync_pure_reducer.Core.graph_scope) registration query maximum_bytes = +let observation_read (scope:Logseq_sync_domain.Core.graph_scope) registration query maximum_bytes = let module F=Read.For_worker in let limits:Worker_pure_data.capability_limits={response_budget_bytes=Worker_pure_data.maximum_response_bytes; outbox_max_records=10000;outbox_max_bytes=10485760;change_max_items=256;change_max_bytes=1048576;dispatcher_capacity=256;wire_batch_max_bytes=1048576}in let binding=F.binding ~serial:registration ~incarnation:(Read.worker_incarnation ~runtime_epoch:(Journal_worker_ids.Runtime.Epoch.of_int64 1L) ~worker_generation:(Journal_worker_ids.Worker.Generation.of_int64 1L)) ~origin:(Uri.to_string scope.account.managed_sync_origin) ~user:scope.account.user_id in - let owner:Logseq_sync_pure_reducer.Core.timeline_scope={startup={account_generation=scope.account.account_generation; + let owner:Logseq_sync_domain.Core.timeline_scope={startup={account_generation=scope.account.account_generation; presentation_generation=scope.account.presentation_generation};graph_id=scope.graph_id;graph_generation=scope.graph_generation}in let session=F.session ~binding ~graph:scope.graph_id ~account:scope.account.account_generation ~lifecycle:scope.account.lifecycle_generation ~generation:scope.graph_generation ~scope:owner ~asset_scope:scope ~limits in @@ -59,6 +59,36 @@ let initial () = let s=raw_initial () in (Core.step s (Sync_event (Sync.Sign_in_requested (startup_scope s)))).next ;; +let test_worker_does_not_coordinate_accounts () = + let state = raw_initial () in + let transition = Core.step state (Sync_event (Sync.Account_authenticated {user_id=Some "app-owned"})) in + Alcotest.check Alcotest.bool "account projection belongs to App" true + ((Core.view transition.next).sync = (Core.view state).sync); + Alcotest.check Alcotest.bool "intent is forwarded, not executed" true + (List.map Core.instruction_diagnostic transition.effects = ["forward-lifecycle"]) +;; + +let test_production_lifecycle_projection () = + let state = raw_initial () in + let scope : Sync.graph_scope = + { account = { managed_sync_origin=Uri.of_string "https://api.logseq.io"; user_id="production"; + account_generation=7; presentation_generation=9; lifecycle_generation=1L } + ; graph_id=uuid "10000000-0000-4000-8000-000000000001"; graph_generation=11 } in + let sync = (Core.view state).sync in + let snapshot = { sync.snapshot with selected_graph=Some scope.graph_id; resident_graphs=[scope.graph_id]; + startup={sync.snapshot.startup with authenticated=true;account_generation=7;graph_generation=11;presentation_generation=9} } in + let coordination : Sync.coordination_state = + { account=Some scope.account; selected=Some {startup={account_generation=7;presentation_generation=9};graph_id=scope.graph_id;graph_generation=11}; residents=[scope, {sync with snapshot}]; desired_residents=[scope.graph_id]; deletion=None } in + let state = Core.install_lifecycle state ~sync:{sync with snapshot} ~coordination + ~attachments:[scope,Core.database_opened ~database_id:"borrowed" ~graph_id:scope.graph_id] in + Alcotest.check Alcotest.bool "external authority is visible" true + ((Core.coordination_state state).sync.account=Some scope.account); + Alcotest.check Alcotest.bool "external graph is open" true + (Option.map (fun graph -> graph.Core.phase) (Core.graph_state_for_scope state scope)=Some Core.Graph_open); + let retired = Core.install_lifecycle state ~sync ~coordination:{coordination with selected=None;residents=[]} ~attachments:[] in + Alcotest.check Alcotest.bool "retired scope is unavailable" true (Core.graph_state_for_scope retired scope=None) +;; + let test_managed_worker_starts_without_local_open () = let transition = Core.step (initial ()) Core.Start in Alcotest.check Alcotest.int "no local open effects" 0 (List.length transition.effects); @@ -80,36 +110,20 @@ let test_closed_graph_request_replies_once () = List.filter (function | Core.Publish (Core.Reply (actual, _)) -> Core.equal_request_id actual id - | Run_worker _ - | Run_sync _ - | Run_asset _ - | Run_upload _ - | Read_uploads _ - | Close_asset_scope _ - | Deliver_asset_result _ - | Deliver_protected_result _ - | Sync_io_fence_reached _ - | Asset_cleanup_barrier_reached _ - | Publish _ -> false) + | _ -> false) transition.effects in Alcotest.check Alcotest.int "one terminal reply" 1 (List.length replies) ;; let test_sync_instructions_are_translated_in_order () = - let transition = - Core.step - (initial ()) - (Sync_event (Sync.Restore_local_account { user_id = "user-1" })) - in - match List.map Core.instruction_diagnostic transition.effects with - | [ "publish:sync-output"; run ] -> - Alcotest.check - Alcotest.bool - "load catalog follows state publication" - true - (String.starts_with ~prefix:"run-sync:request:" run) - | values -> Alcotest.failf "unexpected child order: %s" (String.concat ", " values) + let event = Sync.Restore_local_account { user_id = "user-1" } in + let state = initial () in + let transition = Core.step state (Sync_event event) in + Alcotest.(check bool) "exact lifecycle intent is forwarded once" true + (transition.effects = [Core.Publish (Lifecycle_intent (Sync_intent event))]); + Alcotest.(check bool) "Worker does not run account IO" true + (Core.equal_view (Core.view state) (Core.view transition.next)) ;; let test_replay_is_deterministic () = @@ -133,90 +147,27 @@ let test_replay_is_deterministic () = let worker_open_graph_transition () = let graph : Sync.graph = - { graph_id = uuid "11111111-1111-4111-8111-111111111111" - ; name = "Journal" - ; schema = { major = 1; minor = 0; exact = true } - ; encrypted = false - } - in - let auth = - Core.step - (initial ()) - (Sync_event (Sync.Account_authenticated { user_id = Some "user" })) - in - let fetched = - List.find_map - (function - | Core.Run_sync (Sync.Request (ticket, Sync.Fetch_catalog _)) -> - Some - (Core.Sync_event - (Sync.Runner_completed (Sync.Completion (ticket, Ok [ graph ])))) - | _ -> None) - auth.effects - |> Option.get - in - let catalog = Core.step auth.next fetched in - let selected = - Core.step catalog.next (Sync_event (Sync.Graph_selected graph.graph_id)) - in - let inspected = - List.find_map - (function - | Core.Run_worker - (Core.Request - (ticket, Core.Handle_sync_worker_effect (Sync.Inspect_mirror request))) -> - Some - (Core.Runner_completed - (Core.Sync_worker_effect_completed - ( ticket - , Ok - { Core.event = - Some (Sync.Mirror_inspected (Sync.Mirror_available request)) - ; lifecycle = Core.Lifecycle_unchanged - } ))) - | _ -> None) - selected.effects - |> Option.get - in - let attaching = Core.step selected.next inspected in - let attached, scope = - List.find_map - (function - | Core.Run_worker - (Core.Request - (ticket, Core.Handle_sync_worker_effect (Sync.Attach_graph request))) -> - let token = - Logseq_overlay_db.Types.sync_token_of_string "sync-token:v1:worker-delete" - |> Result.get_ok - in - let checkpoint = - Logseq_overlay_db.Types.Server_cursor.of_int64 0L - |> Result.map_error (fun `Negative_cursor -> "negative cursor") - |> Result.get_ok - in - let sync = - Logseq_overlay_db.Types.sync_view ~token ~checkpoint ~submissions:[] - in - let opened = - Core.database_opened ~database_id:"db-delete" ~graph_id:graph.graph_id - in - Some - ( Core.Runner_completed - (Core.Sync_worker_effect_completed - ( ticket - , Ok - { Core.event = - Some (Sync.Graph_attached { scope = request.scope; sync }) - ; lifecycle = - Core.Lifecycle_opened (opened, request.scope.graph_generation) - } )) - , request.scope ) - | _ -> None) - attaching.effects - |> Option.get - in - let ready = Core.step attaching.next attached in - ready, scope + {graph_id=uuid "11111111-1111-4111-8111-111111111111";name="Journal"; + schema={major=1;minor=0;exact=true};encrypted=false} in + let state=raw_initial () in + let scope : Sync.graph_scope = + {account={managed_sync_origin=Uri.of_string "https://api.logseq.io";user_id="user"; + account_generation=1;presentation_generation=1;lifecycle_generation=0L}; + graph_id=graph.graph_id;graph_generation=1} in + let initial=(Core.view state).sync in + let sync : Sync.state = + {initial with snapshot={initial.snapshot with sync_phase=Sync.Current;catalog=[graph]; + selected_graph=Some graph.graph_id;resident_graphs=[graph.graph_id]; + startup={initial.snapshot.startup with authenticated=true;local_access=Sync.Local_available; + network_phase=Sync.Online_by_user;account_generation=1;graph_generation=1;presentation_generation=1}}} in + let coordination : Sync.coordination_state = + {account=Some scope.account;selected=Some {startup={account_generation=1;presentation_generation=1}; + graph_id=scope.graph_id;graph_generation=scope.graph_generation}; + residents=[scope,sync];desired_residents=[graph.graph_id];deletion=None} in + let projected=Core.step state (Lifecycle_snapshot(sync,coordination, + [scope,Core.database_opened ~database_id:"db-delete" ~graph_id:graph.graph_id])) in + let started = Core.step projected.next Start in + {started with effects=projected.effects @ started.effects},scope ;; let worker_open_graph () = @@ -224,154 +175,47 @@ let worker_open_graph () = ready.Core.next, scope ;; +let retire_projected_graph state = + let sync=(Core.view state).sync in + let sync={sync with snapshot={sync.snapshot with selected_graph=None;resident_graphs=[]}} in + let coordination=(Core.coordination_state state).sync in + Core.step state (Lifecycle_snapshot(sync,{coordination with selected=None;residents=[];desired_residents=[]},[])) +;; + let test_current_deletion_preserves_work_until_retirement () = let ready, scope = worker_open_graph () in - let read=observation_read scope 71L Read.Graph_info 16384 in - let executing = - Core.step ready (Core.Graph_request { id = Core.request_id_of_int64 11L; read }) - in - let completion = - List.find_map - (function - | Core.Run_worker instruction -> - Core.complete_execute - instruction - (Error - (Logseq_db_worker.Error.create - ~code:Closed_session - ~message:"Request finished." - ~details:[] - |> Result.get_ok)) - | _ -> None) - executing.effects - |> Option.get - in - let inspecting = Core.step executing.next (Sync_event Sync.Local_outbox_changed) in - let inspection_completion = - List.find_map - (function - | Core.Run_worker - (Core.Request (ticket, Core.Handle_sync_worker_effect (Sync.Inspect_sync _))) - -> - Some - (Core.Runner_completed - (Core.Sync_worker_effect_completed - ( ticket - , Error - (Logseq_db_worker.Error.create - ~code:Closed_session - ~message:"Inspection finished." - ~details:[] - |> Result.get_ok) ))) - | _ -> None) - inspecting.effects - |> Option.get - in - let protected = - Core.step - inspecting.next - (Sync_event (Sync.Local_cache_deletion_requested scope.graph_id)) - in - Alcotest.(check bool) - "current-copy deletion preserves the open graph and executing work" - true - ((Core.view protected.next).graph.phase = Core.Graph_open - && (Core.view protected.next).sync.snapshot.local_deletion = None - && protected.effects = []); - let retiring = - Core.step protected.next (Sync_event Sync.Graph_picker_requested) - in - Alcotest.(check bool) - "actual graph retirement closes admission immediately" - true - ((Core.view retiring.next).graph.phase = Core.Graph_closing); - let has_close effects = - List.exists - (function - | Core.Run_worker - (Core.Request (_, Core.Handle_sync_worker_effect (Sync.Detach_graph _))) -> - true - | _ -> false) - effects - in - Alcotest.(check bool) - "close waits for executing work" - false - (has_close retiring.effects); - let rejected = - Core.step - retiring.next - (Core.Graph_request { id = Core.request_id_of_int64 12L; read }) - in - Alcotest.(check bool) - "new reading receives a terminal reply and no execution" - true - (List.exists - (function - | Core.Publish (Core.Reply _) -> true - | _ -> false) - rejected.effects - && not - (List.exists - (function - | Core.Run_worker _ -> true - | _ -> false) - rejected.effects)); - let one_finished = Core.step rejected.next completion in - Alcotest.(check bool) - "close still waits for sync DB operation" - false - (has_close one_finished.effects); - let drained = Core.step one_finished.next inspection_completion in - Alcotest.(check bool) - "all DB completions release one close" - true - (has_close drained.effects); - let duplicate = Core.step drained.next inspection_completion in - Alcotest.(check bool) - "duplicate completion cannot release another close" - true - (duplicate.effects = []); - let close_failure = - List.find_map - (function - | Core.Run_worker - (Core.Request (ticket, Core.Handle_sync_worker_effect (Sync.Detach_graph _))) - -> - Some - (Core.Runner_completed - (Core.Sync_worker_effect_completed - ( ticket - , Error - (Logseq_db_worker.Error.create - ~code:Closed_session - ~message:"private close details" - ~details:[] - |> Result.get_ok) ))) - | _ -> None) - drained.effects - |> Option.get - in - let failed = Core.step drained.next close_failure in - Alcotest.(check bool) - "ordinary retirement failure cannot authorize local deletion" - true - ((Core.view failed.next).sync.snapshot.local_deletion - = None); - Alcotest.(check bool) - "close failure never deletes" - false - (List.exists - (function - | Core.Run_worker - (Core.Request (_, Core.Handle_sync_worker_effect (Sync.Delete_mirror _))) -> - true - | _ -> false) - failed.effects) + let read = observation_read scope 71L Read.Graph_info 16384 in + let executing = Core.step ready (Graph_request_for_scope + {scope; id=Core.request_id_of_int64 11L; read}) in + let execution = List.find_map (function Core.Run_worker request -> Some request | _ -> None) + executing.effects |> Option.get in + let event = Sync.Local_cache_deletion_requested scope.graph_id in + let protected = Core.step executing.next (Sync_event event) in + Alcotest.(check bool) "delete intent preserves admitted work" true + ((Core.view protected.next).graph.phase=Core.Graph_open + && (Core.view protected.next).pending_requests=1 + && protected.effects=[Core.Publish (Lifecycle_intent (Sync_intent event))]); + let retiring = retire_projected_graph protected.next in + let rejected = Core.step retiring.next (Graph_request_for_scope + {scope; id=Core.request_id_of_int64 12L; read}) in + Alcotest.(check bool) "withdrawal closes new admission with one reply" true + (match rejected.effects with [Core.Publish (Reply (id,_))] -> + Core.request_id_to_int64 id=12L | _ -> false); + let error = Logseq_db_worker.Error.create ~code:Closed_session + ~message:"Request finished." ~details:[] |> Result.get_ok in + let completion = Core.complete_execute execution (Error error) |> Option.get in + let drained = Core.step rejected.next completion in + Alcotest.(check bool) "withdrawal retains the original terminal reply" true + (match drained.effects with [Core.Publish (Reply (id,_))] -> + Core.request_id_to_int64 id=11L | _ -> false); + Alcotest.(check int) "duplicate completion is inert" 0 + (List.length (Core.step drained.next completion).effects); + Alcotest.(check bool) "Worker never owns physical graph deletion" true + (List.for_all (function Core.Run_worker _ -> false | _ -> true) retiring.effects) ;; module Asset = Logseq_db_types.Asset_descriptor -module Transfer = Logseq_sync_pure_reducer.Asset_transfer +module Transfer = Logseq_sync_asset_transfer.Core let asset_demand (scope : Sync.graph_scope) = let version = @@ -426,7 +270,7 @@ let test_asset_demand_lifecycle () = -> true | _ -> false) completed.effects); - let switched = Core.step completed.next (Sync_event Sync.Graph_picker_requested) in + let switched = retire_projected_graph completed.next in Alcotest.(check bool) "scope retired" false @@ -591,116 +435,28 @@ let test_asset_manual_retry_is_unlimited () = let test_asset_execution_completion_is_delivered () = let state, scope = worker_open_graph () in let request : Sync.asset_request = - { scope - ; operation = "worker-bridge:stage" - ; action = - Sync.Stage_asset_file - { operation = uuid "77000000-0000-4000-8000-000000000001" - ; file_type = "png" - ; source_file = "picker-reference" - } - } - in - let requested = Core.step state (Core.Sync_event (Sync.Asset_requested request)) in - let ticket = - List.find_map - (function - | Core.Run_sync (Sync.Asset_io (ticket, io)) -> - Alcotest.(check bool) - "worker preserves asset request" - true - (io.context.scope = request.scope && io.action = request.action); - Some ticket - | _ -> None) - requested.effects - |> Option.get - in - Alcotest.(check bool) - "request does not resolve its own waiter" - false - (List.exists - (function - | Core.Deliver_asset_result _ -> true - | _ -> false) - requested.effects); - let result = - Ok - (Sync.Asset_staged { file = "staging-reference"; checksum = "checksum"; size = 4L }) - in - let completed = - Core.step - requested.next - (Core.Sync_event (Sync.Runner_completed (Sync.Asset_completion (ticket, result)))) - in - Alcotest.(check bool) - "accepted completion resolves worker waiter" - true - (match completed.effects with - | [ Core.Deliver_asset_result output ] -> - output.request = request && output.result = result - | _ -> false) + {scope; operation="worker-stage"; action=Stage_asset_file + {operation=uuid "77000000-0000-4000-8000-000000000001"; + file_type="png"; source_file="picker-reference"}} in + let requested = Core.step state (Sync_event (Sync.Asset_requested request)) in + Alcotest.(check bool) "asset service receives the exact lifecycle request" true + (requested.effects=[Core.Publish (Lifecycle_intent (Sync_intent (Sync.Asset_requested request)))]); + Alcotest.(check int) "Worker allocates no legacy asset ticket" 0 + (Core.view requested.next).pending_effects ;; let test_worker_shutdown_forwards_late_resource_cleanup () = let state, scope = worker_open_graph () in + let stopped = Core.step state Shutdown in + Alcotest.(check bool) "shutdown forwards lifecycle stop once" true + (List.filter (function Core.Publish (Lifecycle_intent Stop_intent) -> true | _ -> false) + stopped.effects = [Core.Publish (Lifecycle_intent Stop_intent)]); let request : Sync.asset_request = - { scope - ; operation = "worker-shutdown:stage" - ; action = - Sync.Stage_asset_file - { operation = uuid "77000000-0000-4000-8000-000000000007" - ; file_type = "bin" - ; source_file = "picker-reference" - } - } - in - let requested = Core.step state (Core.Sync_event (Sync.Asset_requested request)) in - let ticket = - List.find_map - (function - | Core.Run_sync (Sync.Asset_io (ticket, _)) -> Some ticket - | _ -> None) - requested.effects - |> Option.get - in - let stopped = Core.step requested.next Core.Shutdown in - Alcotest.(check bool) - "worker shutdown cancels pending requester" - true - (List.exists - (function - | Core.Deliver_asset_result output -> - output.request = request && output.result = Error Sync.Asset_cancelled - | _ -> false) - stopped.effects); - let late = - Core.step - stopped.next - (Core.Sync_event - (Sync.Runner_completed - (Sync.Asset_completion - ( ticket - , Ok - (Sync.Asset_staged - { file = "late-staging-reference" - ; checksum = "checksum" - ; size = 4L - }) )))) - in - Alcotest.(check bool) - "stopped worker keeps accepting cleanup completions" - true - (Core.view late.next).shutdown; - Alcotest.(check bool) - "late resource cleanup is forwarded to sync runner" - true - (List.exists - (function - | Core.Run_sync (Sync.Asset_io (_, io)) -> - io.context.scope = scope - && io.action = Sync.Release_staged_file "late-staging-reference" - | _ -> false) - late.effects) + {scope; operation="late-release"; action=Release_asset_file "late-staging-reference"} in + let late = Core.step stopped.next (Sync_event (Sync.Asset_requested request)) in + Alcotest.(check bool) "shutdown retains cleanup intent delivery without reopening" true + ((Core.view late.next).shutdown + && late.effects=[Core.Publish (Lifecycle_intent (Sync_intent (Sync.Asset_requested request)))]) ;; let test_upload_session () = @@ -790,7 +546,7 @@ let test_upload_session () = | Core.Publish (Asset_notice (_, Upload_status _)) -> true | _ -> false) duplicate.effects)); - let switched = Core.step completed.next (Sync_event Sync.Graph_picker_requested) in + let switched = retire_projected_graph completed.next in Alcotest.(check bool) "graph switch cancels upload" true @@ -829,7 +585,7 @@ let test_upload_recovery () = "duplicate recovery completion ignored" true (duplicate.effects = []); - let switched = Core.step opened.next (Sync_event Sync.Graph_picker_requested) in + let switched = retire_projected_graph opened.next in let stale = Core.step switched.next (Uploads_loaded (ticket, Error "obsolete")) in Alcotest.(check bool) "old graph recovery ignored" true (stale.effects = []) ;; @@ -858,6 +614,43 @@ let recovery_intent (scope : Sync.graph_scope) n = |> Result.get_ok ;; +let test_projection_withdrawal_cancels_upload () = + let module U = Logseq_db_worker_pure_reducer.Asset_upload in + let state, scope = worker_open_graph () in + let intent = recovery_intent scope 1 in + let started = Core.step state (Upload_requested_for_scope (scope, intent.operation_id, U.Start intent)) in + let ticket = List.find_map (function + | Core.Run_upload (_, U.Persist (ticket, _, _)) -> Some ticket + | _ -> None) started.effects |> Option.get in + let retired = retire_projected_graph started.next in + Alcotest.(check bool) "withdrawal cancels the original upload ticket" true + (List.exists (function + | Core.Run_upload (context, U.Cancel_operation actual) -> context.scope = scope && actual = ticket + | _ -> false) retired.effects); + Alcotest.(check bool) "withdrawal closes the original upload scope" true + (List.mem (Core.Close_asset_scope scope) retired.effects); + Alcotest.(check bool) "withdrawn upload cannot complete" false + (Core.upload_ticket_current retired.next ticket) +;; + +let test_projection_withdrawal_cancels_asset_lookup () = + let state, scope = worker_open_graph () in + let requested = Core.step state (asset_demand scope) in + let ticket = lookup requested.effects in + let retired = retire_projected_graph requested.next in + Alcotest.(check bool) "withdrawal cancels the original lookup ticket" true + (List.exists (function + | Core.Run_asset (context, Transfer.Cancel (Transfer.Lookup actual)) -> context.scope = scope && actual = ticket + | _ -> false) retired.effects); + Alcotest.(check bool) "withdrawal closes the original lookup scope" true + (List.mem (Core.Close_asset_scope scope) retired.effects); + Alcotest.(check bool) "withdrawn lookup cannot complete" false + (Core.asset_ticket_current retired.next (Transfer.Lookup ticket)); + let repeated = retire_projected_graph retired.next in + Alcotest.(check bool) "withdrawal cleanup is emitted only once" false + (List.mem (Core.Close_asset_scope scope) repeated.effects) +;; + let test_upload_recovery_backpressure () = let opened, scope = worker_open_graph_transition () in let first = Option.get (recovery_instruction opened) in @@ -1006,225 +799,153 @@ let test_upload_recovery_failure () = (Core.upload_recovery_current stale.next fresh) ;; -(* The worker owns translating its runner's terminal result into Sync state. - Drive a complete cold bootstrap using only public events and issued tickets. *) -let snapshot_activation () = - let graph : Sync.graph = - { graph_id = uuid "71111111-1111-4111-8111-111111111111" - ; name = "Synthetic schema fixture" - ; schema = { major = 65; minor = 29; exact = true } - ; encrypted = false - } - in - let auth = Core.step (initial ()) - (Sync_event (Sync.Account_authenticated { user_id = Some "synthetic" })) in - let catalog = List.find_map (function - | Core.Run_sync (Sync.Request (ticket, Sync.Fetch_catalog _)) -> - Some (Core.step auth.next (Sync_event - (Sync.Runner_completed (Sync.Completion (ticket, Ok [graph]))))) - | _ -> None) auth.effects |> Option.get in - let selected = Core.step catalog.next (Sync_event (Sync.Graph_selected graph.graph_id)) in - let missing, _scope = List.find_map (function - | Core.Run_worker (Core.Request (ticket, - Core.Handle_sync_worker_effect (Sync.Inspect_mirror request))) -> - Some (Core.step selected.next (Runner_completed - (Sync_worker_effect_completed (ticket, Ok - { event = Some (Sync.Mirror_inspected (Mirror_absent request.scope)) - ; lifecycle = Lifecycle_unchanged }))), request.scope) - | _ -> None) selected.effects |> Option.get in - let download = List.find_map (function - | Core.Run_sync (Sync.Request (ticket, Sync.Fetch_snapshot_metadata _)) -> - Some (Core.step missing.next (Sync_event (Sync.Runner_completed - (Sync.Completion (ticket, Ok {|{"ok":true,"t":42,"url":"https://example.invalid/snapshot"}|}))))) - | _ -> None) missing.effects |> Option.get in - let activation = List.find_map (function - | Core.Run_sync (Sync.Request (ticket, Sync.Download_snapshot download_request)) -> - let artifact = Sync.stage_snapshot_download download_request ~path:"/tmp/synthetic-schema.snapshot" ~expected_rows:1 |> Result.get_ok in - Some (Core.step download.next (Sync_event (Sync.Runner_completed - (Sync.Completion (ticket, Ok artifact))))) - | _ -> None) download.effects |> Option.get in - let error = Logseq_db_worker.Error.create ~code:Closed_session - ~message:"Graph schema 65.29 is not supported. Journal requires schema 65.33 or later." - ~details:[] |> Result.get_ok in - let failure, success = List.find_map (function - | Core.Run_worker (Core.Request (ticket, - Core.Handle_sync_worker_effect (Sync.Activate_snapshot request))) -> - Some (Core.Runner_completed (Sync_worker_effect_completed (ticket, Error error)), - Core.Runner_completed (Sync_worker_effect_completed (ticket, Ok - { event = Some (Sync.Snapshot_activated (Sync.snapshot_activation_completed request)) - ; lifecycle = Lifecycle_unchanged }))) - | _ -> None) activation.effects |> Option.get in - activation, failure, success, Logseq_db_worker.Error.message error -;; - let test_snapshot_failure_is_terminal_and_retryable () = - let activation, failure, _, message = snapshot_activation () in - let failed = Core.step activation.next failure in - let snapshot = (Core.view failed.next).sync.snapshot in - Alcotest.(check bool) "failure terminates bootstrap" true - (snapshot.startup.failure = Some Sync.During_bootstrap && snapshot.sync_phase = Sync.Failed); - Alcotest.(check (option string)) "specific reason retained" (Some message) snapshot.last_error; - Alcotest.(check bool) "failure published to UI" true + let ready, scope = worker_open_graph () in + let sync = (Core.view ready).sync in + let message = "Graph schema 65.29 is not supported." in + let failed = {sync with snapshot={sync.snapshot with sync_phase=Sync.Failed; + last_error=Some message; startup={sync.snapshot.startup with failure=Some Sync.During_bootstrap}}} in + let coordination = (Core.coordination_state ready).sync in + let project state sync = Core.step state (Lifecycle_snapshot (sync, + {coordination with residents=[scope,sync]}, + [scope,Core.database_opened ~database_id:"db-delete" ~graph_id:scope.graph_id])) in + let failure = project ready failed in + Alcotest.(check bool) "App failure projection reaches UI unchanged" true (List.exists (function Core.Publish (Sync_output (Sync.State_changed state)) -> - state.snapshot.last_error = Some message | _ -> false) failed.effects); - let duplicate = Core.step failed.next failure in - Alcotest.(check bool) "duplicate completion inert" true - (Core.equal_view (Core.view failed.next) (Core.view duplicate.next) && duplicate.effects = []); - let retry = Core.step failed.next (Sync_event (Sync.Online_recovery_requested (timeline_scope failed.next))) in - Alcotest.(check (option string)) "explicit retry clears failure" None - (Core.view retry.next).sync.snapshot.last_error; - let late = Core.step retry.next failure in - Alcotest.(check bool) "old failure cannot poison retry" true - (Core.equal_view (Core.view retry.next) (Core.view late.next) && late.effects = []) + state=failed | _ -> false) failure.effects); + let intent = Sync.Online_recovery_requested (timeline_scope failure.next) in + let retry = Core.step failure.next (Sync_event intent) in + Alcotest.(check bool) "only App may clear bootstrap failure" true + ((Core.view retry.next).sync=failed + && List.exists ((=) (Core.Publish (Lifecycle_intent (Sync_intent intent)))) retry.effects); + let recovered = project retry.next sync in + Alcotest.(check (option string)) "App recovery clears projected failure" None + (Core.view recovered.next).sync.snapshot.last_error ;; let test_snapshot_success_and_late_switch_completions () = - let activation, failure, success, _ = snapshot_activation () in - let succeeded = Core.step activation.next success in - let attaching = List.find_map (function - | Core.Run_worker (Core.Request (ticket, - Core.Handle_sync_worker_effect (Sync.Inspect_mirror request))) -> - Some (Core.step succeeded.next (Runner_completed (Sync_worker_effect_completed - (ticket, Ok { event = Some (Sync.Mirror_inspected (Mirror_available request)) - ; lifecycle = Lifecycle_unchanged })))) - | _ -> None) succeeded.effects |> Option.get in - Alcotest.(check bool) "success proceeds to attachment" true - (List.exists (function Core.Run_worker (Core.Request (_, - Core.Handle_sync_worker_effect (Sync.Attach_graph _))) -> true | _ -> false) attaching.effects); - Alcotest.(check (option string)) "success has no failure" None - (Core.view succeeded.next).sync.snapshot.last_error; - let switched = Core.step activation.next (Sync_event Sync.Graph_picker_requested) in - List.iter (fun completion -> - let late = Core.step switched.next completion in - Alcotest.(check bool) "retired graph keeps selected state" true - ((Core.view switched.next).sync = (Core.view late.next).sync); - Alcotest.(check bool) "retired graph keeps lifecycle" true - ((Core.view switched.next).graph = (Core.view late.next).graph); - let work = List.filter (function - | Core.Run_worker (Core.Request (_, Core.Handle_sync_worker_effect (Sync.Detach_graph _))) -> false - | _ -> true) late.effects in - Alcotest.(check (list string)) "retired graph emits only pending close cleanup" [] - (List.map Core.instruction_diagnostic work)) - [failure; success] + let ready, scope = worker_open_graph () in + let requested = Core.step ready (Graph_request_for_scope + {scope; id=Core.request_id_of_int64 81L; read=observation_read scope 71L Read.Graph_info 16384}) in + let ticket = List.find_map (function Core.Run_worker (Core.Request (ticket,Execute_request _)) -> + Some ticket | _ -> None) requested.effects |> Option.get in + let completion = Core.Runner_completed (Sync_worker_effect_completed + (ticket,Ok {event=None;lifecycle=Lifecycle_closed scope.graph_generation})) in + let late = Core.step requested.next completion in + Alcotest.(check bool) "unissued lifecycle completion cannot retire an open graph" true + (Core.equal_view (Core.view requested.next) (Core.view late.next) && late.effects=[]) ;; let test_local_restore_worker_failures_are_terminal () = - let error = Logseq_db_worker_contract.Error.create - ~code:Logseq_db_worker_contract.Error.Closed_session ~message:"synthetic local IO failure" ~details:[] |> Result.get_ok in - let make () = - let restored = Core.step (initial ()) (Sync_event (Sync.Restore_local_account {user_id="local-user"})) in - let graph : Sync.graph = {graph_id=uuid "11111111-1111-4111-8111-111111111111"; - name="Synthetic";schema={major=1;minor=0;exact=true};encrypted=false} in - let cache = Sync.catalog_cache ~user_id:"local-user" ~graphs:[graph] ~selected_graph:(Some graph.graph_id) in - let event = List.find_map (function - | Core.Run_sync (Sync.Request (ticket, Sync.Load_catalog _)) -> - Some (Core.Sync_event (Sync.Runner_completed (Sync.Completion(ticket,Ok (Some cache))))) - | _ -> None) restored.effects |> Option.get in - Core.step restored.next event in - List.iter (fun attach -> - let selected = make () in - let ticket,request = List.find_map (function - | Core.Run_worker (Core.Request(ticket,Core.Handle_sync_worker_effect(Sync.Inspect_mirror request))) ->Some(ticket,request) - | _->None) selected.effects |> Option.get in - let flight,ticket = if not attach then selected,ticket else - let attaching = Core.step selected.next (Runner_completed (Sync_worker_effect_completed(ticket, - Ok {Core.event=Some(Sync.Mirror_inspected(Mirror_available request));lifecycle=Lifecycle_unchanged}))) in - let ticket = List.find_map (function - | Core.Run_worker (Core.Request(ticket,Core.Handle_sync_worker_effect(Sync.Attach_graph _)))->Some ticket - | _->None) attaching.effects |> Option.get in attaching,ticket in - let failed = Core.step flight.next (Runner_completed (Sync_worker_effect_completed(ticket,Error error))) in - let f = (Core.view failed.next).sync.snapshot.startup in - Alcotest.(check bool) "worker IO failure releases startup loading" true - (not f.restoring_local && f.failure=Some Sync.During_local_restore); - let duplicate = Core.step failed.next (Runner_completed (Sync_worker_effect_completed(ticket,Error error))) in - Alcotest.(check int) "duplicate failure is inert" 0 (List.length duplicate.effects); - let retired=Core.step flight.next (Sync_event Sync.Graph_picker_requested) in - let late=Core.step retired.next (Runner_completed(Sync_worker_effect_completed(ticket,Error error))) in - Alcotest.(check bool) "retired local restore keeps sync state" true - ((Core.view late.next).sync=(Core.view retired.next).sync); - Alcotest.(check bool) "retired local restore cannot report into another graph" false - (List.exists(function Core.Publish(Diagnostic _)->true | _->false)late.effects) - ) [false;true] -;; - -(* This tests downstream propagation only: the runner owns producer classification. - Closed/version preparation races have no deterministic public runner hook. *) + let ready, scope = worker_open_graph () in + let requested = Core.step ready (Graph_request_for_scope + {scope; id=Core.request_id_of_int64 82L; read=observation_read scope 71L Read.Graph_info 16384}) in + let ticket = List.find_map (function Core.Run_worker (Core.Request (ticket,Execute_request _)) -> + Some ticket | _ -> None) requested.effects |> Option.get in + let error = Logseq_db_worker.Error.create ~code:Closed_session + ~message:"unissued local restore failure" ~details:[] |> Result.get_ok in + let failed = Core.step requested.next (Runner_completed (Sync_worker_effect_completed (ticket,Error error))) in + Alcotest.(check bool) "foreign completion cannot consume an executing read" true + (Core.equal_view (Core.view requested.next) (Core.view failed.next) && failed.effects=[]) +;; + let test_authoritative_failure_categories_reach_diagnostics () = let module Error = Logseq_db_worker.Error in - List.iter (fun (code, category, message) -> - let ready, graph = worker_open_graph_transition () in - let connection = List.find_map (function - | Core.Run_sync (Sync.Start_websocket request) -> Some request.scope - | _ -> None) ready.effects |> Option.get in - let opened = Core.step ready.next (Sync_event (Sync.Websocket_opened connection)) in - let cursor = Logseq_db_types.Server_cursor.of_int64 1L |> Result.get_ok in - let tx : Logseq_sync_pure_reducer.Sync_protocol.Server.pull_transaction = - {t=cursor;tx="[[\"~:db/retractEntity\",99999]]";outliner_op=None} in - let pending = Core.step opened.next (Sync_event (Sync.Websocket_message - (connection, Pull_ok {t=cursor;checksum=None;txs=[tx]}))) in - let ticket = List.find_map (function - | Core.Run_worker (Core.Request (ticket, Handle_sync_worker_effect (Sync.Apply_authoritative_batch _))) -> Some ticket - | _ -> None) pending.effects |> Option.get in - let origin = Error.create_cause ~component:Managed_sync ~operation:"applyAuthoritative" + List.iter (fun (code,category,message) -> + let ready,scope=worker_open_graph () in + let requested=Core.step ready (Graph_request_for_scope + {scope;id=Core.request_id_of_int64 83L;read=observation_read scope 71L Read.Graph_info 16384}) in + let execution=List.find_map(function Core.Run_worker request->Some request|_->None) + requested.effects |> Option.get in + let origin=Error.create_cause ~component:Managed_sync ~operation:"applyAuthoritative" ~code:(Some category) ~message |> Result.get_ok in - let error = Error.create_with_origin ~code ~message ~details:[] ~origin |> Result.get_ok in - let failure = Core.Runner_completed (Sync_worker_effect_completed (ticket, Error error)) in - let failed = Core.step pending.next failure in - Alcotest.(check (option string)) "Last error retains category" (Some message) - (Core.view failed.next).sync.snapshot.last_error; - Alcotest.(check bool) "Worker diagnostics retain category" true - (List.exists (function Core.Publish (Diagnostic signal) -> - signal="worker.applyAuthoritative." ^ category | _ -> false) failed.effects); - Alcotest.(check bool) "local graph remains open" true - ((Core.view failed.next).graph.phase=Graph_open && graph.graph_generation=(Core.view failed.next).graph.generation); + let error=Error.create_with_origin ~code ~message ~details:[] ~origin |> Result.get_ok in + let completion=Core.complete_execute execution (Error error) |> Option.get in + let failed=Core.step requested.next completion in + Alcotest.(check bool) "typed read retains the original error category" true + (List.exists(function Core.Publish(Reply(_,Read.Query_answer(_,_,Error(Read.Db_error actual)))) -> + actual=error|_->false) failed.effects); + Alcotest.(check bool) "read failure cannot change App sync ownership" true + ((Core.view failed.next).sync=(Core.view requested.next).sync + && (Core.view failed.next).graph.phase=Core.Graph_open); Alcotest.(check int) "duplicate completion is inert" 0 - (List.length (Core.step failed.next failure).effects); - let retired=Core.step pending.next (Sync_event Sync.Graph_picker_requested) in - let late=Core.step retired.next failure in - Alcotest.(check bool) "retired authoritative completion keeps sync state" true - ((Core.view late.next).sync=(Core.view retired.next).sync); - Alcotest.(check bool) "retired authoritative completion cannot report into another graph" false - (List.exists(function Core.Publish(Diagnostic _)->true | _->false)late.effects)) - [ Error.Closed_session, "authoritativeDatabaseClosed", "Authoritative database is closed." - ; Conflict, "authoritativeSyncTokenConflict", "Authoritative sync version changed." - ; Ambiguous_sync_state, "authoritativeCursorDiscontinuous", "Authoritative transaction cursors are not continuous." - ; Unsupported_semantics, "authoritativeDecodeFailed", "Authoritative transaction format is unsupported." ] + (List.length(Core.step failed.next completion).effects)) + [Error.Closed_session,"authoritativeDatabaseClosed","Authoritative database is closed." + ;Conflict,"authoritativeSyncTokenConflict","Authoritative sync version changed." + ;Ambiguous_sync_state,"authoritativeCursorDiscontinuous","Authoritative transaction cursors are not continuous." + ;Unsupported_semantics,"authoritativeDecodeFailed","Authoritative transaction format is unsupported."] ;; +let project_residents state (sync:Sync.state) (coordination:Sync.coordination_state) = + let attachments=List.map (fun ((scope:Sync.graph_scope),_) -> + let database_id=if scope.graph_id=uuid "11111111-1111-4111-8111-111111111111" + then "db-delete" else "resident-" ^ Logseq_db_types.Graph_types.Uuid.to_string scope.graph_id in + scope,Core.database_opened ~database_id ~graph_id:scope.graph_id) coordination.residents in + Core.step state (Lifecycle_snapshot(sync,coordination,attachments)) +;; + +let test_lifecycle_snapshot_restores_uploads () = + let module U = Logseq_db_worker_pure_reducer.Asset_upload in + let module I = Logseq_db_types.Asset_upload_intent in + let ready, scope = worker_open_graph () in + let sync = (Core.view ready).sync in + let coordination = (Core.coordination_state ready).sync in + let background = {scope with graph_id=uuid "bbbbbbbb-bbbb-4bbb-8bbb-bbbbbbbbbbbb"; graph_generation=2} in + let coordination = {coordination with residents=[scope,sync;background,sync]; + desired_residents=[scope.graph_id;background.graph_id]} in + let restored = project_residents (raw_initial ()) sync coordination in + let reads = List.filter_map (function Core.Read_uploads ticket -> Some ticket | _ -> None) restored.effects in + Alcotest.(check int) "snapshot alone starts every resident's bounded recovery" 2 (List.length reads); + let repeated = project_residents restored.next sync coordination in + Alcotest.(check bool) "repeated snapshot does not duplicate pending reads" true + (List.for_all (function Core.Read_uploads _ -> false | _ -> true) repeated.effects); + let ticket = List.find (fun ticket -> ticket.Core.scope=background) reads in + let prepared = recovery_intent background 1 in + let pending = List.fold_left (fun intent phase -> I.advance intent phase |> Result.get_ok) + prepared [I.Local_committed;Uploading;Remote_stored;Metadata_pending] in + let loaded = Core.step repeated.next (Uploads_loaded (ticket, Ok [pending])) in + let inspect = List.find_map (function + | Core.Run_upload (context,U.Inspect (ticket,intent)) when context.scope=background && intent=pending -> Some ticket + | _ -> None) loaded.effects |> Option.get in + let observed = Core.step loaded.next (Upload_completed (inspect,U.Inspected U.Publication_acknowledged)) in + let save = List.find_map (function + | Core.Run_upload (_,U.Persist (ticket,intent,Some 4)) when intent.phase=I.Complete -> Some ticket + | _ -> None) observed.effects |> Option.get in + let completed = Core.step observed.next (Upload_completed (save,U.Persisted)) in + Alcotest.(check bool) "exact original ACK releases staging after durable Complete" true + (List.exists (function Core.Run_upload (_,U.Release_staging intent) -> intent.operation_id=pending.operation_id | _ -> false) completed.effects); + Alcotest.(check bool) "replay never stages, inserts, uploads or republishes metadata" true + (List.for_all (function + | Core.Run_upload (_, (U.Apply_local _ | Put _ | Apply_metadata _)) -> false + | _ -> true) (loaded.effects @ observed.effects @ completed.effects)) +;; + +let project_selection state graph_id = + let coordination=(Core.coordination_state state).sync in + let scope,_=List.find(fun((scope:Sync.graph_scope),_)->scope.graph_id=graph_id)coordination.residents in + let selected : Sync.timeline_scope = + {startup={account_generation=scope.account.account_generation; + presentation_generation=scope.account.presentation_generation}; + graph_id;graph_generation=scope.graph_generation} in + let sync=(Core.view state).sync in + project_residents state {sync with snapshot={sync.snapshot with selected_graph=Some graph_id}} + {coordination with selected=Some selected} +;; let resident_worker_fixture () = - let ready, first_scope = worker_open_graph () in - let first_id=first_scope.Sync.graph_id in + let ready,first_scope=worker_open_graph () in let second_id=uuid "bbbbbbbb-bbbb-4bbb-8bbb-bbbbbbbbbbbb" in - let snapshot=(Core.view ready).sync.snapshot in - let second : Sync.graph = {(List.hd snapshot.catalog) with graph_id=second_id;name="Second synthetic graph"} in - let requested=Core.step ready (Sync_event (Sync.Catalog_refresh_requested (startup_scope ready))) in - let fetched=List.find_map (function - | Core.Run_sync (Sync.Request (ticket,Sync.Fetch_catalog _)) -> - Some (Core.Sync_event (Sync.Runner_completed (Sync.Completion (ticket,Ok (snapshot.catalog @ [second]))))) - | _ -> None) requested.effects |> Option.get in - let catalog=Core.step requested.next fetched in - let configured=Core.step catalog.next (Sync_event (Sync.Resident_graphs_changed [first_id;second_id])) in - let rec open_requests core = function - | [] -> core - | Core.Run_worker (Core.Request (ticket,Core.Handle_sync_worker_effect (Sync.Inspect_mirror request))) :: rest -> - let next=Core.step core (Runner_completed (Sync_worker_effect_completed (ticket, - Ok {event=Some (Sync.Mirror_inspected (Mirror_available request));lifecycle=Lifecycle_unchanged}))) in - open_requests next.next (next.effects @ rest) - | Core.Run_worker (Core.Request (ticket,Core.Handle_sync_worker_effect (Sync.Attach_graph request))) :: rest -> - let database_id="resident-" ^ Logseq_db_types.Graph_types.Uuid.to_string request.scope.graph_id in - let database=Core.database_opened ~database_id ~graph_id:request.scope.graph_id in - let module O=Logseq_overlay_db.Types in - let sync=O.sync_view ~token:(O.sync_token_of_string "sync-token:v1:resident" |> Result.get_ok) - ~checkpoint:(O.Server_cursor.of_int64 0L |> Result.get_ok) ~submissions:[] in - let next=Core.step core (Runner_completed (Sync_worker_effect_completed (ticket, - Ok {event=Some (Sync.Graph_attached {scope=request.scope;sync}); - lifecycle=Lifecycle_opened (database,request.scope.graph_generation)}))) in - open_requests next.next rest - | _ :: rest -> open_requests core rest in - let opened=open_requests configured.next configured.effects in - opened,first_scope,second_id + let sync=(Core.view ready).sync in + let second : Sync.graph = {(List.hd sync.snapshot.catalog) with graph_id=second_id;name="Second synthetic graph"} in + let second_scope={first_scope with graph_id=second_id;graph_generation=2} in + let sync={sync with snapshot={sync.snapshot with catalog=sync.snapshot.catalog @ [second]; + resident_graphs=[first_scope.graph_id;second_id]}} in + let coordination=(Core.coordination_state ready).sync in + let opened=project_residents ready sync {coordination with residents=[first_scope,sync;second_scope,sync]; + desired_residents=[first_scope.graph_id;second_id]} in + opened.next,first_scope,second_id ;; + let test_resident_worker_routes_by_scope () = let opened,first_scope,second_id=resident_worker_fixture () in Alcotest.(check int) "two independent DB scopes remain open" 2 (List.length (Core.graph_scopes opened)); @@ -1232,7 +953,7 @@ let test_resident_worker_routes_by_scope () = let read scope=observation_read scope 71L Read.Graph_info 16384 in let started=Core.step opened (Graph_request_for_scope {scope=first_scope;id=Core.request_id_of_int64 101L;read=read first_scope}) in let execution=List.find_map (function Core.Run_worker ((Core.Request (_,Execute_request _)) as instruction)->Some instruction | _->None) started.effects |> Option.get in - let focused=Core.step started.next (Sync_event (Sync.Visible_graph_changed second_id)) in + let focused=project_selection started.next second_id in let hidden=Core.step focused.next (Graph_request_for_scope {scope=first_scope;id=Core.request_id_of_int64 102L;read=read first_scope}) in Alcotest.(check bool) "hidden request still targets first database" true (List.exists (function Core.Run_worker (Core.Request (_,Execute_request {database;_}))-> @@ -1252,63 +973,93 @@ let test_resident_worker_routes_by_scope () = ;; let test_resident_worker_closes_every_scope () = - List.iter (fun event -> + List.iter (fun intent -> let opened,_,_=resident_worker_fixture () in - let closing=Core.step opened event in - let detached=List.filter_map (function - | Core.Run_worker (Core.Request (ticket,Handle_sync_worker_effect (Sync.Detach_graph scope))) -> Some(ticket,scope) - | _->None) closing.effects in - Alcotest.(check int) "every removed DB independently drains and detaches" 2 (List.length detached); - let terminal=List.fold_left (fun state (ticket,scope)-> - (Core.step state (Runner_completed (Sync_worker_effect_completed (ticket, - Ok {event=Some(Sync.Graph_detached(scope,Ok()));lifecycle=Lifecycle_closed scope.graph_generation})))).next) - closing.next detached in - Alcotest.(check int) "no removed graph remains admitted" 0 (List.length(Core.graph_scopes terminal))) - [Core.Sync_event(Sync.Resident_graphs_changed []);Core.Sync_event(Sync.Account_authenticated{user_id=None})] + let forwarded=Core.step opened (Sync_event intent) in + Alcotest.(check int) "intent alone cannot revoke App residents" 2 + (List.length(Core.graph_scopes forwarded.next)); + Alcotest.(check bool) "retirement request reaches App exactly once" true + (forwarded.effects=[Core.Publish(Lifecycle_intent(Sync_intent intent))]); + let retired=retire_projected_graph forwarded.next in + Alcotest.(check int) "App withdrawal removes every scope" 0 + (List.length(Core.graph_scopes retired.next))) + [Sync.Resident_graphs_changed [];Sync.Account_authenticated{user_id=None}] +;; + let test_resident_worker_shutdown_closes_all_databases () = let opened,_,_=resident_worker_fixture () in let stopped=Core.step opened Shutdown in - let databases=List.filter_map (function - | Core.Run_worker(Core.Request(_,Close_database database))->Some(Core.database_handle_id database) - | _->None) stopped.effects in - Alcotest.(check int) "shutdown closes both resident databases" 2 (List.length(List.sort_uniq String.compare databases)) + Alcotest.(check bool) "Worker borrows rather than closes App databases" false + (List.exists(function Core.Run_worker(Core.Request(_,Close_database _))->true|_->false)stopped.effects); + Alcotest.(check int) "stop intent is account-wide, not once per graph" 1 + (List.length(List.filter(function Core.Publish(Lifecycle_intent Stop_intent)->true|_->false)stopped.effects)); + List.iter(fun scope-> + let rejected=Core.step stopped.next(Graph_request_for_scope{scope;id=Core.request_id_of_int64 904L; + read=observation_read scope 71L Read.Graph_info 16384})in + Alcotest.(check bool) "shutdown replies once without executing" true + (match rejected.effects with [Core.Publish(Reply _)]->true|_->false)) (Core.graph_scopes opened) ;; + let test_resident_worker_cold_restore_keeps_open_tickets () = - List.iter(fun count-> - let graphs=List.init count(fun n->({graph_id=uuid(Printf.sprintf "65000000-0000-4000-8000-%012d" (n+1));name="Cached synthetic"; - schema={major=65;minor=33;exact=true};encrypted=false}:Sync.graph)) in - let ids=List.map(fun(g:Sync.graph)->g.graph_id) graphs in - let selected=List.nth ids (count-1) in - let cache=Sync.catalog_cache ~user_id:"cached-resident" ~graphs ~selected_graph:(Some selected) - |> Sync.encode_catalog_cache |> Yojson.Safe.from_string |> Yojson.Safe.Util.to_assoc in - let cache=`Assoc(("cache_resident_graphs",`List(List.map(fun id->`String(Logseq_db_types.Graph_types.Uuid.to_string id)) ids))::cache) - |> Yojson.Safe.to_string |> Sync.decode_catalog_cache |> Result.get_ok in - let restoring=Core.step (raw_initial ()) (Sync_event(Sync.Restore_local_account{user_id="cached-resident"})) in - let completion=List.find_map(function Core.Run_sync(Sync.Request(ticket,Sync.Load_catalog _))-> - Some(Core.Sync_event(Sync.Runner_completed(Sync.Completion(ticket,Ok(Some cache)))))|_->None) restoring.effects |> Option.get in - let restored=Core.step restoring.next completion in - let inspections=List.filter_map(function Core.Run_worker(Core.Request(ticket,Handle_sync_worker_effect(Sync.Inspect_mirror request)))->Some(ticket,request)|_->None) restored.effects in - Alcotest.(check int) "one cold local inspection per resident" count (List.length inspections); - let opened=List.fold_left(fun state (ticket,(request:Sync.mirror_request))-> - let attaching=Core.step state (Runner_completed(Sync_worker_effect_completed(ticket, - Ok{event=Some(Sync.Mirror_inspected(Mirror_available request));lifecycle=Lifecycle_unchanged}))) in - let attachment=List.find_map(function Core.Run_worker(Core.Request(ticket,Handle_sync_worker_effect(Sync.Attach_graph _)))->Some ticket|_->None) attaching.effects in - Alcotest.(check bool) "cached inspection ticket survives session restore" true (Option.is_some attachment); - let database=Core.database_opened ~database_id:("cached-" ^ Logseq_db_types.Graph_types.Uuid.to_string request.scope.graph_id) ~graph_id:request.scope.graph_id in - let module O=Logseq_overlay_db.Types in - let sync=O.sync_view ~token:(O.sync_token_of_string "sync-token:v1:cached" |> Result.get_ok) - ~checkpoint:(O.Server_cursor.of_int64 0L |> Result.get_ok) ~submissions:[] in - (Core.step attaching.next (Runner_completed(Sync_worker_effect_completed(Option.get attachment, - Ok{event=Some(Sync.Graph_attached{scope=request.scope;sync});lifecycle=Lifecycle_opened(database,request.scope.graph_generation)})))).next) - restored.next (List.rev inspections) in - Alcotest.(check bool) "cold restore preserves last focused graph" true ((Core.view opened).sync.snapshot.selected_graph=Some selected); - List.iter(fun scope-> - Alcotest.(check bool) "each restored graph really opens" true - (Option.fold ~none:false ~some:(fun(g:Core.graph_state)->g.phase=Core.Graph_open) (Core.graph_state_for_scope opened scope)); - let read=Core.step opened (Graph_request_for_scope{scope;id=Core.request_id_of_int64 901L;read=observation_read scope 71L Read.Graph_info 16384}) in - Alcotest.(check bool) "restored scope executes against its own DB" true - (List.exists(function Core.Run_worker(Core.Request(_,Execute_request{database;_}))-> - Core.database_handle_id database="cached-" ^ Logseq_db_types.Graph_types.Uuid.to_string scope.graph_id|_->false) read.effects)) (Core.graph_scopes opened)) [1;2;3] + List.iter(fun count-> + let ready,first=worker_open_graph () in + let scopes=List.init count(fun index->{first with graph_id=uuid(Printf.sprintf + "65000000-0000-4000-8000-%012d" (index+1));graph_generation=index+1})in + let selected=List.nth scopes(count-1)in + let sync=(Core.view ready).sync in + let catalog=List.map(fun(scope:Sync.graph_scope)-> + ({graph_id=scope.graph_id;name="Cached synthetic";schema={major=65;minor=33;exact=true};encrypted=false}:Sync.graph))scopes in + let sync={sync with snapshot={sync.snapshot with catalog;selected_graph=Some selected.graph_id; + resident_graphs=List.map(fun(scope:Sync.graph_scope)->scope.graph_id)scopes; + startup={sync.snapshot.startup with graph_generation=selected.graph_generation}}}in + let coordination=(Core.coordination_state ready).sync in + let opened=project_residents ready sync {coordination with residents=List.map(fun scope->scope,sync)scopes; + desired_residents=List.map(fun(scope:Sync.graph_scope)->scope.graph_id)scopes; + selected=Some{startup=startup_scope ready;graph_id=selected.graph_id;graph_generation=selected.graph_generation}}in + Alcotest.(check bool) "restored focus follows App projection" true + ((Core.view opened.next).sync.snapshot.selected_graph=Some selected.graph_id); + List.iter(fun scope-> + let read=Core.step opened.next(Graph_request_for_scope{scope;id=Core.request_id_of_int64 901L; + read=observation_read scope 71L Read.Graph_info 16384})in + Alcotest.(check bool) "each projected resident executes its own database" true + (List.exists(function Core.Run_worker(Core.Request(_,Execute_request{database;_}))-> + Core.database_handle_id database="resident-" ^ Logseq_db_types.Graph_types.Uuid.to_string scope.graph_id|_->false)read.effects))scopes) + [1;2;3] +;; + +let test_hidden_read_survives_withdrawal () = + let opened,first,second=resident_worker_fixture () in + let executing=Core.step opened(Graph_request_for_scope{scope=first; + id=Core.request_id_of_int64 910L;read=observation_read first 71L Read.Graph_info 16384})in + let execution=List.find_map(function Core.Run_worker request->Some request|_->None)executing.effects |> Option.get in + let focused=project_selection executing.next second in + let retired=retire_projected_graph focused.next in + let error=Logseq_db_worker.Error.create ~code:Closed_session ~message:"hidden read finished" ~details:[] |> Result.get_ok in + let completion=Core.complete_execute execution(Error error) |> Option.get in + let completed=Core.step retired.next completion in + Alcotest.(check bool) "withdrawn hidden read keeps its one terminal reply" true + (match completed.effects with [Core.Publish(Reply(id,_))]->Core.request_id_to_int64 id=910L|_->false); + Alcotest.(check int) "duplicate hidden completion cannot reply twice" 0 + (List.length(Core.step completed.next completion).effects) +;; + +let test_read_cannot_borrow_another_scope () = + let opened,first,second=resident_worker_fixture ()in + let other=List.find(fun(scope:Sync.graph_scope)->scope.graph_id=second)(Core.graph_scopes opened)in + List.iter(fun event-> + let transition=Core.step opened event in + Alcotest.(check bool) "foreign read budget receives a terminal error without execution" true + (match transition.effects with [Core.Publish(Reply(_,Read.Query_answer(_,_,Error _)))]->true|_->false)) + [Core.Graph_request_for_scope{scope=first;id=Core.request_id_of_int64 911L; + read=observation_read other 71L Read.Graph_info 16384}; + Core.Graph_request{id=Core.request_id_of_int64 912L;read=observation_read other 71L Read.Graph_info 16384}] +;; + +let test_foreground_intent_is_account_wide () = + let opened,_,_=resident_worker_fixture ()in + let transition=Core.step opened(Set_foreground false)in + Alcotest.(check int) "foreground intent is forwarded once across all residents" 1 + (List.length(List.filter(function Core.Publish(Lifecycle_intent(Foreground_intent false))->true|_->false)transition.effects)) ;; let test_coordination_owner_snapshot () = @@ -1319,8 +1070,8 @@ let test_coordination_owner_snapshot () = List.iter (fun ((scope:Sync.graph_scope),state) -> Alcotest.(check bool) "exact open lifetime" true (state.Core.phase=Graph_open && state.generation=scope.Sync.graph_generation)) before.residents; - let switched=(Core.step opened (Sync_event (Sync.Visible_graph_changed second))).next in - let back=(Core.step switched (Sync_event (Sync.Visible_graph_changed first.graph_id))).next in + let switched=(project_selection opened second).next in + let back=(project_selection switched first.graph_id).next in let after=Core.coordination_state back in Alcotest.(check bool) "focus preserves execution scopes" true (List.map fst before.residents=List.map fst after.residents); @@ -1329,10 +1080,16 @@ let test_coordination_owner_snapshot () = (Core.coordination_state switched).sync.selected=Some second && Option.map (fun (owner:Sync.timeline_scope)->owner.graph_id) after.sync.selected=Some first.graph_id); - let removed=(Core.step back (Sync_event (Sync.Resident_graphs_changed [first.graph_id]))).next in - let deleting=Core.step removed (Sync_event (Sync.Local_cache_deletion_requested second)) in + let sync=(Core.view back).sync in + let coordination=(Core.coordination_state back).sync in + let removed=(project_residents back sync {coordination with + residents=List.filter(fun((scope:Sync.graph_scope),_)->scope=first)coordination.residents; + desired_residents=[first.graph_id]}).next in + let deletion_scope=List.find(fun(scope:Sync.graph_scope)->scope.graph_id=second)(Core.graph_scopes back) in + let coordination=(Core.coordination_state removed).sync in + let deleting=project_residents removed sync {coordination with deletion=Some(deletion_scope,Sync.Deletion_in_progress Sync.Deleting_mirror)} in let current=Core.coordination_state deleting.next in - Alcotest.(check bool) "exact deletion target is visible" true + Alcotest.(check bool) "App deletion target is visible" true (Option.fold ~none:false ~some:(fun ((scope:Sync.graph_scope),_) -> scope.graph_id=second) current.sync.deletion); let empty=Core.coordination_state (raw_initial ()) in @@ -1679,7 +1436,7 @@ let test_changefeed_owner_properties () = let test_unallocated_observation_fence () = let module O = Core.Observation_registry in let _, scope = worker_open_graph () in - let owner : Logseq_sync_pure_reducer.Core.timeline_scope = + let owner : Logseq_sync_domain.Core.timeline_scope = { startup = { account_generation = scope.account.account_generation ; presentation_generation = 0 @@ -1885,7 +1642,7 @@ let test_demand_registry_retirement_requires_physical_settlement () = let test_resource_registry_finalization () = let module O=Core.Observation_registry in let module D=Core.Demand_registry in let _,scope=worker_open_graph() in - let timeline:Logseq_sync_pure_reducer.Core.timeline_scope={ + let timeline:Logseq_sync_domain.Core.timeline_scope={ startup={account_generation=scope.account.account_generation;presentation_generation=scope.account.presentation_generation}; graph_id=scope.graph_id;graph_generation=scope.graph_generation} in let observation=ref O.empty and demand=ref D.empty in @@ -2042,7 +1799,30 @@ let test_writer_fence_properties () = done ;; +let test_lifecycle_host_gate () = + let opened,first,second=resident_worker_fixture() in + let sync=(Core.view opened).sync and coordination=(Core.coordination_state opened).sync in + let external_state=(Core.step opened (Core.Lifecycle_snapshot(sync,coordination,[]))).next in + let id=uuid "10000000-0000-4000-8000-000000079081" in + let owner=Option.get coordination.account in + let fence:Sync.host_fence={id;owner;target=One_graph first.graph_id;captured_scopes=Rrbvec.singleton first;previous=None} in + let fenced=(Core.step external_state (Sync_event(Sync.Host_fence_started fence))).next in + Alcotest.(check bool) "external host gate withdraws target" false + (List.mem first (Core.graph_scopes fenced)); + Alcotest.(check bool) "external host gate keeps unrelated graph" true + (List.exists(fun(scope:Sync.graph_scope)->scope.graph_id=second)(Core.graph_scopes fenced)); + let republished=(Core.step fenced (Core.Lifecycle_snapshot(sync,coordination,[]))).next in + Alcotest.(check bool) "stale projection cannot reopen fenced target" false + (List.mem first (Core.graph_scopes republished)); + let foreign=(Core.step republished (Sync_event(Sync.Host_fence_released(uuid "10000000-0000-4000-8000-000000079082")))).next in + Alcotest.(check bool) "external fence requires exact release" true (Core.host_fence_blocks foreign first.graph_id); + let released=(Core.step foreign (Sync_event(Sync.Host_fence_released id))).next in + Alcotest.(check bool) "exact release clears external fence" false (Core.host_fence_blocks released first.graph_id) +;; + let test_host_gate_keeps_other_graph_live () = + test_lifecycle_host_gate (); + let opened,first,second=resident_worker_fixture() in let id=uuid "10000000-0000-4000-8000-000000079001" in let owner=Option.get(Core.coordination_state opened).sync.account in @@ -2060,54 +1840,67 @@ let test_host_gate_keeps_other_graph_live () = let wrong=Core.step configured.next(Sync_event(Sync.Host_fence_released(uuid "10000000-0000-4000-8000-000000079099"))) in Alcotest.(check bool) "foreign release cannot open target" true(Core.host_fence_blocks wrong.next first.graph_id); let released=Core.step wrong.next(Sync_event(Sync.Host_fence_released id)) in - let reopened=Core.step released.next(Sync_event(Sync.Resident_graphs_changed[first.graph_id;second])) in + let sync=(Core.view opened).sync and coordination=(Core.coordination_state opened).sync in + let fresh={first with graph_generation=first.graph_generation+1}in + let reopened=project_residents released.next sync {coordination with + residents=List.map(fun(scope,state)->(if scope=first then fresh else scope),state)coordination.residents}in Alcotest.(check bool) "fresh execution only after exact fence release" true (List.exists(fun(scope:Sync.graph_scope)->scope.graph_id=first.graph_id && scope.graph_generation>first.graph_generation) (Core.graph_scopes reopened.next)) ;; let test_revoked_account_keeps_hidden_cleanup_owner () = - let opened,_,second=resident_worker_fixture() in + let opened,_,second=resident_worker_fixture () in let hidden=List.find(fun(scope:Sync.graph_scope)->scope.graph_id=second)(Core.graph_scopes opened) in - let revoked=Core.step opened(Sync_event(Sync.Account_authenticated{user_id=None})) in - let release scope operation=Core.step revoked.next(Sync_event(Sync.Asset_requested - {scope;operation;action=Release_asset_file "original-hidden-preview"})) in - let released=release hidden "revoked-hidden-release" in - Alcotest.(check bool) "old hidden owner retains cleanup-only IO authority after auth revocation" true - (List.exists(function Core.Run_sync(Sync.Asset_io(_,{context;action=Release_asset_file "original-hidden-preview";_}))-> - context.scope=hidden|_->false)released.effects); - let forged=release {hidden with graph_generation=hidden.graph_generation+100} "foreign-hidden-release" in - Alcotest.(check bool) "revocation never creates implicit authority for an unknown execution" false - (List.exists(function Core.Run_sync(Sync.Asset_io _)->true|_->false)forged.effects); - let acquire=Core.step revoked.next(Sync_event(Sync.Asset_requested - {scope=hidden;operation="revoked-hidden-acquire";action=Retain_asset_file "original-hidden-preview"})) in - Alcotest.(check bool) "retaining cleanup ownership does not restore old data/acquisition authority" false - (List.exists(function Core.Run_sync(Sync.Asset_io _)->true|_->false)acquire.effects) + let revoked=retire_projected_graph opened in + List.iter(fun scope-> + let request : Sync.asset_request = {scope;operation="revoked-hidden-release"; + action=Release_asset_file "original-hidden-preview"}in + let released=Core.step revoked.next(Sync_event(Sync.Asset_requested request))in + Alcotest.(check bool) "Worker forwards cleanup identity without granting authority" true + (released.effects=[Core.Publish(Lifecycle_intent(Sync_intent(Sync.Asset_requested request)))]); + Alcotest.(check int) "forwarding cannot create a graph owner" 0 + (List.length(Core.graph_scopes released.next))) + [hidden;{hidden with graph_generation=hidden.graph_generation+100}] ;; let test_host_deletion_private_resource_owner () = - let opened,first,_=resident_worker_fixture() in + let opened,first,_=resident_worker_fixture () in let owner=Option.get(Core.coordination_state opened).sync.account in let id=uuid "10000000-0000-4000-8000-000000079003" in - let fenced=Core.step opened(Sync_event(Sync.Host_fence_started{id;owner;target=One_graph first.graph_id;captured_scopes=Rrbvec.singleton first;previous=None})) in - let deleting=Core.step fenced.next(Sync_event(Sync.Local_cache_deletion_requested first.graph_id)) in - let request=List.find_map(function - |Core.Run_worker(Core.Request(_,Core.Handle_sync_worker_effect(Sync.Delete_mirror request)))->Some request|_->None)deleting.effects - |> Option.get in - let scope:Sync.graph_scope={account=request.account;graph_id=request.graph_id;graph_generation=0} in + let fenced=Core.step opened(Sync_event(Sync.Host_fence_started{id;owner;target=One_graph first.graph_id; + captured_scopes=Rrbvec.singleton first;previous=None})) in let retirement=Sync.retirement_id_of_string "10000000-0000-4000-8000-000000079004" |> Result.get_ok in - let started=Core.step deleting.next(Sync_event(Sync.Asset_requested - {scope;operation="host-private-delete";action=Prepare_graph_asset_deletion retirement})) in - let ticket=List.find_map(function - |Core.Run_sync(Sync.Asset_io(ticket,{context;action=Prepare_graph_asset_deletion actual;_})) - when context.scope=scope && actual=retirement->Some ticket|_->None)started.effects in - Alcotest.(check bool) "private deletion resource request reaches its actual owner" true(Option.is_some ticket); - let deletion=Sync.asset_deletion_reference ~scope ~retirement in - let completed=Core.step started.next(Sync_event(Sync.Runner_completed(Sync.Asset_completion - (Option.get ticket,Ok(Sync.Asset_deletion_prepared deletion))))) in - Alcotest.(check bool) "private deletion result reaches original Worker waiter" true - (List.exists(function Core.Deliver_asset_result output->output.request.operation="host-private-delete" - && output.result=Ok(Sync.Asset_deletion_prepared deletion)|_->false)completed.effects) + let request : Sync.asset_request = {scope={first with graph_generation=0};operation="host-private-delete"; + action=Prepare_graph_asset_deletion retirement} in + let started=Core.step fenced.next(Sync_event(Sync.Asset_requested request)) in + Alcotest.(check bool) "private resource request preserves identity for App validation" true + (started.effects=[Core.Publish(Lifecycle_intent(Sync_intent(Sync.Asset_requested request)))]); + Alcotest.(check bool) "resource forwarding never reopens fenced execution" false + (List.mem first(Core.graph_scopes started.next)) +;; + +let test_host_gate_defers_physical_close_to_settlement () = + let opened,first,_=resident_worker_fixture() in + let opened=(Core.step opened(asset_demand first)).next in + let closes effects=List.filter_map(function Core.Close_asset_scope scope->Some scope|_->None)effects in + let ordinary=retire_projected_graph opened in + Alcotest.(check bool) "ordinary withdrawal still closes the resource owner" true + (List.mem first (closes ordinary.effects)); + List.iter(fun target-> + let sync=(Core.view opened).sync and coordination=(Core.coordination_state opened).sync in + let owner=Option.get coordination.account in + let fence:Sync.host_fence={id=uuid "10000000-0000-4000-8000-000000079005";owner;target; + captured_scopes=Rrbvec.of_list(Core.graph_scopes opened);previous=None} in + let fenced=Core.step opened(Sync_event(Sync.Host_fence_started fence)) in + Alcotest.(check bool) "host fence withdraws execution before physical settlement" false + (List.mem first(Core.graph_scopes fenced.next)); + Alcotest.(check int) "host settlement alone may close its fenced resource owner" 0 + (List.length(closes fenced.effects)); + let republished=project_residents fenced.next sync coordination in + Alcotest.(check int) "late lifecycle projection cannot start a competing close" 0 + (List.length(closes republished.effects))) + [Sync.One_graph first.graph_id;All_graphs] ;; let test_account_host_gate_blocks_new_catalog_producer () = @@ -2118,8 +1911,10 @@ let test_account_host_gate_blocks_new_catalog_producer () = let startup=(Core.view closed.next).sync.snapshot.startup in let refresh=Core.step closed.next(Sync_event(Sync.Catalog_refresh_requested {account_generation=startup.account_generation;presentation_generation=startup.presentation_generation})) in - Alcotest.(check bool) "account handoff cannot start another catalog producer" false - (List.exists(function Core.Run_sync(Sync.Request(_,Sync.Fetch_catalog _))->true|_->false)refresh.effects) + Alcotest.(check bool) "Worker forwards catalog intent without opening a producer" true + (refresh.effects=[Core.Publish(Lifecycle_intent(Sync_intent(Sync.Catalog_refresh_requested + {account_generation=startup.account_generation;presentation_generation=startup.presentation_generation})))] + && Core.graph_scopes refresh.next=[]) ;; let test_host_handoff_single_use_and_reopen () = @@ -2344,6 +2139,47 @@ let test_registration_retirement_properties () = done ;; +let test_external_lifecycle_retirement () = + let module L = Core.Lifecycle in + let scope : Sync.graph_scope = + { account = { managed_sync_origin = Uri.of_string "https://api.logseq.io"; + user_id = "external"; account_generation = 1; presentation_generation = 1; + lifecycle_generation = 1L }; + graph_id = uuid "10000000-0000-4000-8000-000000000001"; graph_generation = 1 } + in + let opened = Core.database_opened ~database_id:"service-owned" ~graph_id:scope.graph_id in + let snapshot : L.snapshot = { revision = 1L; scopes = [scope]; selected = Some scope } in + let state = (L.step L.empty (L.Projection snapshot)).next in + let state = (L.step state (L.Attach (scope, opened))).next in + let read = observation_read scope 71L Read.Graph_info 16384 in + let request = L.step state (L.Read {scope; id = Core.request_id_of_int64 91L; read}) in + let ticket = match request.effects with + | [L.Worker (Core.Request (ticket, Core.Execute_request _))] -> ticket + | _ -> Alcotest.fail "external attachment must execute a read" + in + let foreign = {scope with graph_generation = 2} in + let foreign_read = observation_read foreign 72L Read.Graph_info 16384 in + let rejected = L.step request.next (L.Read {scope; id = Core.request_id_of_int64 93L; read = foreign_read}) in + Alcotest.check Alcotest.bool "foreign read budget never reaches the database" true + (match rejected.effects with [L.Publish (Core.Reply _)] -> true | _ -> false); + let retiring = L.step request.next (L.Retire scope) in + Alcotest.check Alcotest.int "retirement waits for physical reads" 0 (List.length retiring.effects); + let denied = L.step retiring.next (L.Read {scope; id = Core.request_id_of_int64 92L; read}) in + Alcotest.check Alcotest.bool "retiring graph rejects new reads" true + (match denied.effects with [L.Publish (Core.Reply _)] -> true | _ -> false); + let Read.Read_query (budget, query) = read in + let answer = Read.For_worker.answer budget query (Error (Read.Service_failed "test")) in + let completed = L.step denied.next (L.Completed (Core.Execute_request_completed (ticket, Ok answer))) in + Alcotest.check Alcotest.bool "reply precedes retirement ACK" true + (match completed.effects with [L.Publish (Core.Reply _); L.Retired actual] -> actual = scope | _ -> false); + Alcotest.check Alcotest.int "duplicate completion ignored" 0 + (List.length (L.step completed.next (L.Completed (Core.Execute_request_completed (ticket, Ok answer)))).effects); + let revoked = L.step completed.next (L.Projection {revision = 2L; scopes = []; selected = None}) in + let stale = L.step revoked.next (L.Projection snapshot) in + Alcotest.check Alcotest.int "stale snapshot cannot re-admit attachment" 0 + (List.length (L.step stale.next (L.Attach (scope, opened))).effects) +;; + let () = Alcotest.run "logseq db worker pure reducer" @@ -2356,14 +2192,17 @@ let () = Alcotest.test_case "coordination owner snapshot" `Quick test_coordination_owner_snapshot; Alcotest.test_case "cold cached membership opens every DB" `Quick test_resident_worker_cold_restore_keeps_open_tickets; Alcotest.test_case "scope routes and delayed replies survive focus" `Quick test_resident_worker_routes_by_scope; - Alcotest.test_case "all removals and logout detach each graph" `Quick test_resident_worker_closes_every_scope; - Alcotest.test_case "shutdown closes every resident database" `Quick test_resident_worker_shutdown_closes_all_databases]) - ; ( "authoritative diagnostics", [Alcotest.test_case "four categories propagate without closing graph" `Quick test_authoritative_failure_categories_reach_diagnostics]) + Alcotest.test_case "App removal projection withdraws each graph" `Quick test_resident_worker_closes_every_scope; + Alcotest.test_case "shutdown borrows DBs and forwards one stop" `Quick test_resident_worker_shutdown_closes_all_databases]) + ; ( "authoritative diagnostics", [Alcotest.test_case "typed read errors preserve four original causes" `Quick test_authoritative_failure_categories_reach_diagnostics]) ; ( "snapshot failure", [ - Alcotest.test_case "terminal failure and explicit retry" `Quick test_snapshot_failure_is_terminal_and_retryable; - Alcotest.test_case "success and late graph switch completions" `Quick test_snapshot_success_and_late_switch_completions ]) + Alcotest.test_case "App failure and recovery projections" `Quick test_snapshot_failure_is_terminal_and_retryable; + Alcotest.test_case "unissued lifecycle success cannot retire graph" `Quick test_snapshot_success_and_late_switch_completions ]) ; ( "asset session" - , [ Alcotest.test_case "bounded upload recovery" `Quick test_upload_recovery + , [ Alcotest.test_case "projection withdrawal cancels upload" `Quick test_projection_withdrawal_cancels_upload + ; Alcotest.test_case "projection withdrawal closes lookup scope" `Quick test_projection_withdrawal_cancels_asset_lookup + ; Alcotest.test_case "lifecycle snapshot restores acknowledged uploads" `Quick test_lifecycle_snapshot_restores_uploads + ; Alcotest.test_case "bounded upload recovery" `Quick test_upload_recovery ; Alcotest.test_case "upload recovery backpressure" `Quick @@ -2375,7 +2214,7 @@ let () = ; Alcotest.test_case "upload recovery failure" `Quick test_upload_recovery_failure ; Alcotest.test_case "durable upload session" `Quick test_upload_session ; Alcotest.test_case - "asset execution completion delivery" + "asset lifecycle intent forwarding" `Quick test_asset_execution_completion_is_delivered ; Alcotest.test_case @@ -2396,7 +2235,7 @@ let () = `Quick test_asset_manual_retry_is_unlimited ] ) - ; ( "host handoff",[Alcotest.test_case "revoked hidden resource cleanup owner" `Quick test_revoked_account_keeps_hidden_cleanup_owner; Alcotest.test_case "private deletion resource owner" `Quick test_host_deletion_private_resource_owner; Alcotest.test_case "account gate blocks catalog producer" `Quick test_account_host_gate_blocks_new_catalog_producer; Alcotest.test_case "target gate preserves other execution" `Quick test_host_gate_keeps_other_graph_live; Alcotest.test_case "single-use and reopen witness" `Quick test_host_handoff_single_use_and_reopen; Alcotest.test_case "generated owner/late result traces" `Quick test_host_handoff_properties]) + ; ( "host handoff",[Alcotest.test_case "host gate defers physical resource close" `Quick test_host_gate_defers_physical_close_to_settlement; Alcotest.test_case "revoked hidden resource cleanup owner" `Quick test_revoked_account_keeps_hidden_cleanup_owner; Alcotest.test_case "private deletion resource owner" `Quick test_host_deletion_private_resource_owner; Alcotest.test_case "account gate blocks catalog producer" `Quick test_account_host_gate_blocks_new_catalog_producer; Alcotest.test_case "target gate preserves other execution" `Quick test_host_gate_keeps_other_graph_live; Alcotest.test_case "single-use and reopen witness" `Quick test_host_handoff_single_use_and_reopen; Alcotest.test_case "generated owner/late result traces" `Quick test_host_handoff_properties]) ; ( "writer fence",[Alcotest.test_case "only physical completion drains" `Quick test_writer_fence_physical_completion; Alcotest.test_case "generated scoped fence traces" `Quick test_writer_fence_properties]) ; ( "typed demand registry",[Alcotest.test_case "generated owner retirement traces" `Quick test_demand_registry_properties; Alcotest.test_case "retirement and exact settlement" `Quick test_demand_registry_retirement_requires_physical_settlement; Alcotest.test_case "resource history finalization" `Quick test_resource_registry_finalization]) ; ( "orphan resource cleanup",[Alcotest.test_case "failed release and stale attempt ACK" `Quick test_orphan_cleanup_retains_failed_release]) @@ -2405,8 +2244,9 @@ let () = Alcotest.test_case "active ACK and tombstone capacities" `Quick test_observation_registry_capacity] ) ; ( "read budget", [Alcotest.test_case "scoped byte budget" `Quick test_scoped_read_byte_budget; Alcotest.test_case "pure bounded page planning" `Quick test_pure_read_page_bounds]) + ; ( "external lifecycle", [Alcotest.test_case "hidden read withdrawal" `Quick test_hidden_read_survives_withdrawal; Alcotest.test_case "read scope authority" `Quick test_read_cannot_borrow_another_scope; Alcotest.test_case "account-wide foreground" `Quick test_foreground_intent_is_account_wide; Alcotest.test_case "account intent forwarding" `Quick test_worker_does_not_coordinate_accounts; Alcotest.test_case "production projection" `Quick test_production_lifecycle_projection; Alcotest.test_case "physical retirement and stale snapshots" `Quick test_external_lifecycle_retirement]) ; ( "managed lifecycle" - , [ Alcotest.test_case "local restore IO failures terminate loading" `Quick test_local_restore_worker_failures_are_terminal + , [ Alcotest.test_case "unissued lifecycle failure cannot consume read" `Quick test_local_restore_worker_failures_are_terminal ; Alcotest.test_case "start has no local open" `Quick @@ -2416,7 +2256,7 @@ let () = `Quick test_closed_graph_request_replies_once ; Alcotest.test_case - "sync instruction order" + "exact lifecycle forwarding" `Quick test_sync_instructions_are_translated_in_order ; Alcotest.test_case diff --git a/logseq_db_worker/typed_read/dune b/logseq_db_worker/typed_read/dune index 0deb89a4..ad2772b7 100644 --- a/logseq_db_worker/typed_read/dune +++ b/logseq_db_worker/typed_read/dune @@ -4,4 +4,4 @@ (wrapped false) (modules worker_pure_read) (libraries logseq_db_worker.identity logseq_db_types logseq_db_worker.typed_data - logseq_db_worker.contract logseq_sync.pure_reducer uri yojson rrbvec)) + logseq_db_worker.contract logseq_sync.domain logseq_sync.asset_transfer uri yojson rrbvec)) diff --git a/logseq_db_worker/typed_read/worker_pure_read.ml b/logseq_db_worker/typed_read/worker_pure_read.ml index 75cca695..a3b5ff36 100644 --- a/logseq_db_worker/typed_read/worker_pure_read.ml +++ b/logseq_db_worker/typed_read/worker_pure_read.ml @@ -22,8 +22,8 @@ type session = ; account : account_generation ; lifecycle : int64 ; generation : graph_generation - ; scope : Logseq_sync_pure_reducer.Core.timeline_scope - ; asset_scope : Logseq_sync_pure_reducer.Core.graph_scope + ; scope : Logseq_sync_domain.Core.timeline_scope + ; asset_scope : Logseq_sync_domain.Core.graph_scope ; limits : Worker_pure_data.capability_limits } @@ -362,7 +362,7 @@ let compare_session (a:session)(b:session)= Stdlib.compare(key a)(key b) ;; -let matches_asset_scope (session:session) (scope:Logseq_sync_pure_reducer.Core.graph_scope)= +let matches_asset_scope (session:session) (scope:Logseq_sync_domain.Core.graph_scope)= let expected=session.asset_scope in expected.graph_id=scope.graph_id && expected.graph_generation=scope.graph_generation && expected.account.user_id=scope.account.user_id @@ -377,7 +377,7 @@ let same_logical_graph (a : session) (b : session) = && Uuid.equal a.graph b.graph ;; -let matches_scope (session : session) (scope : Logseq_sync_pure_reducer.Core.timeline_scope) = +let matches_scope (session : session) (scope : Logseq_sync_domain.Core.timeline_scope) = session.account = scope.startup.account_generation && session.generation = scope.graph_generation && Uuid.equal session.graph scope.graph_id diff --git a/logseq_db_worker/typed_read/worker_pure_read.mli b/logseq_db_worker/typed_read/worker_pure_read.mli index 7e9c0580..f4304258 100644 --- a/logseq_db_worker/typed_read/worker_pure_read.mli +++ b/logseq_db_worker/typed_read/worker_pure_read.mli @@ -40,8 +40,8 @@ val graph_id : session -> Uuid.t val equal_session : session -> session -> bool val compare_session : session -> session -> int val same_logical_graph : session -> session -> bool -val matches_scope : session -> Logseq_sync_pure_reducer.Core.timeline_scope -> bool -val matches_asset_scope : session -> Logseq_sync_pure_reducer.Core.graph_scope -> bool +val matches_scope : session -> Logseq_sync_domain.Core.timeline_scope -> bool +val matches_asset_scope : session -> Logseq_sync_domain.Core.graph_scope -> bool val worker_incarnation : runtime_epoch:Journal_worker_ids.Runtime.epoch @@ -363,13 +363,13 @@ module For_worker : sig val binding_serial : binding -> int64 val binding_origin : binding -> string val binding_user : binding -> string - val session : binding:binding -> graph:Uuid.t -> account:int -> lifecycle:int64 -> generation:int -> scope:Logseq_sync_pure_reducer.Core.timeline_scope -> asset_scope:Logseq_sync_pure_reducer.Core.graph_scope -> limits:Worker_pure_data.capability_limits -> session + val session : binding:binding -> graph:Uuid.t -> account:int -> lifecycle:int64 -> generation:int -> scope:Logseq_sync_domain.Core.timeline_scope -> asset_scope:Logseq_sync_domain.Core.graph_scope -> limits:Worker_pure_data.capability_limits -> session val session_binding : session -> binding val session_account : session -> int val session_lifecycle : session -> int64 val session_generation : session -> int - val session_scope : session -> Logseq_sync_pure_reducer.Core.timeline_scope - val session_asset_scope : session -> Logseq_sync_pure_reducer.Core.graph_scope + val session_scope : session -> Logseq_sync_domain.Core.timeline_scope + val session_asset_scope : session -> Logseq_sync_domain.Core.graph_scope val session_limits : session -> Worker_pure_data.capability_limits val read_budget : owner:session -> maximum_items:int -> maximum_bytes:int -> read_budget val read_budget_owner : read_budget -> session diff --git a/logseq_overlay_db/lib/authoritative_checksum.ml b/logseq_overlay_db/lib/authoritative_checksum.ml index 9b878d77..edc40328 100644 --- a/logseq_overlay_db/lib/authoritative_checksum.ml +++ b/logseq_overlay_db/lib/authoritative_checksum.ml @@ -49,12 +49,10 @@ let tuple_digest (entity_uuid, attr, value) = hash_code state field_separator |> fun state -> digest_string state value ;; -let datoms db ~e ~a = Datascript.datoms db Datascript.Eavt ~e ~a () |> List.of_seq - let first_value db entity attr = - match datoms db ~e:entity ~a:attr with - | datom :: _ -> Some datom.Datascript.v - | [] -> None + Datascript.datoms db Datascript.Eavt ~e:entity ~a:attr () + |> Seq.uncons + |> Option.map (fun (datom, _) -> datom.Datascript.v) ;; let uuid db entity = @@ -69,14 +67,6 @@ let boolean db entity attr = | _ -> None ;; -let has_value db entity attr = not (List.is_empty (datoms db ~e:entity ~a:attr)) - -let ident db entity = - match first_value db entity "db/ident" with - | Some (Datascript.Keyword value) -> Some value - | _ -> None -;; - let page_tag_idents = [ "logseq.class/Journal" ; "logseq.class/Tag" @@ -85,25 +75,6 @@ let page_tag_idents = ] ;; -let page_entity db entity = - datoms db ~e:entity ~a:"block/tags" - |> List.exists (fun datom -> - match datom.Datascript.v with - | Datascript.Ref target -> - (match ident db target with - | Some value -> List.mem value page_tag_idents - | None -> false) - | _ -> false) -;; - -let eligible db entity = - Option.is_some (uuid db entity) - && boolean db entity "logseq.property/built-in?" <> Some true - && (has_value db entity "block/name" - || has_value db entity "block/page" - || page_entity db entity) -;; - let normalized_value db attr value = match attr, value with | ("block/parent" | "block/page"), Datascript.Ref entity -> @@ -113,70 +84,87 @@ let normalized_value db attr value = | _ -> "" ;; -let tuples db ~e2ee entity = - match uuid db entity with - | None -> Tuple_set.empty - | Some entity_uuid -> - let relevant = - if e2ee - then [ "block/uuid"; "block/parent"; "block/page"; "block/order" ] - else - [ "block/uuid" - ; "block/parent" - ; "block/page" - ; "block/order" - ; "block/title" - ; "block/name" - ] - in +let tuples ?(visit = fun () -> ()) db ~e2ee entity = + let facts = + Datascript.datoms db Datascript.Eavt ~e:entity () + |> Seq.map (fun (datom : Datascript.datom) -> + visit (); + datom.a, datom.v) + |> List.of_seq + in + let first attr = List.assoc_opt attr facts in + let page_tag = function + | "block/tags", Datascript.Ref target -> + (match first_value db target "db/ident" with + | Some (Datascript.Keyword value) -> List.mem value page_tag_idents + | _ -> false) + | _ -> false + in + match first "block/uuid" with + | Some (Datascript.Uuid entity_uuid) + when first "logseq.property/built-in?" <> Some (Datascript.Bool true) + && (Option.is_some (first "block/name") + || Option.is_some (first "block/page") + || List.exists page_tag facts) -> List.fold_left - (fun tuples attr -> - datoms db ~e:entity ~a:attr - |> List.fold_left - (fun tuples datom -> - Tuple_set.add - (entity_uuid, attr, normalized_value db attr datom.Datascript.v) - tuples) - tuples) + (fun tuples (attr, value) -> + if + List.mem + attr + (if e2ee + then [ "block/uuid"; "block/parent"; "block/page"; "block/order" ] + else + [ "block/uuid" + ; "block/parent" + ; "block/page" + ; "block/order" + ; "block/title" + ; "block/name" + ]) + then Tuple_set.add (entity_uuid, attr, normalized_value db attr value) tuples + else tuples) Tuple_set.empty - relevant + facts + | _ -> Tuple_set.empty ;; let entities_with_uuid db = - let _, entities = - Datascript.datoms db Datascript.Aevt ~a:"block/uuid" () - |> Seq.fold_left - (fun (previous, entities) (datom : Datascript.datom) -> - match previous with - | Some entity when Int.equal entity datom.e -> previous, entities - | _ -> Some datom.e, datom.e :: entities) - (None, []) - in - entities + Datascript.datoms db Datascript.Aevt ~a:"block/uuid" () + |> Seq.map (fun (datom : Datascript.datom) -> datom.e) + |> Seq.group Int.equal + |> Seq.filter_map (fun group -> Seq.uncons group |> Option.map fst) ;; let hex32 value = Printf.sprintf "%08lx" value -let recompute ~e2ee db = +type t = + { e2ee : bool + ; fnv : int32 + ; djb : int32 + } + +let to_string state = hex32 state.fnv ^ hex32 state.djb + +let initialize ?(check_current = fun () -> ()) ~e2ee db = let state = entities_with_uuid db - |> List.fold_left + |> Seq.fold_left (fun state entity -> - if not (eligible db entity) - then state - else - Tuple_set.fold - (fun tuple (sum_fnv, sum_djb) -> - let fnv, djb = tuple_digest tuple in - Int32.add sum_fnv fnv, Int32.add sum_djb djb) - (tuples db ~e2ee entity) - state) + check_current (); + Tuple_set.fold + (fun tuple (sum_fnv, sum_djb) -> + let fnv, djb = tuple_digest tuple in + Int32.add sum_fnv fnv, Int32.add sum_djb djb) + (tuples ~visit:check_current db ~e2ee entity) + state) (0l, 0l) in let fnv, djb = state in - hex32 fnv ^ hex32 djb + { e2ee; fnv; djb } ;; +let recompute ?check_current ~e2ee db = initialize ?check_current ~e2ee db |> to_string + let graph_e2ee db = match Datascript.datoms @@ -190,3 +178,78 @@ let graph_e2ee db = | Some (datom, _) -> boolean db datom.Datascript.e "kv/value" = Some true | None -> false ;; + +let advance ~maximum ~before ~after ~changes state = + let exception Limit in + let remaining = ref maximum in + let visit () = + if !remaining <= 0 then raise Limit; + decr remaining + in + let entities = Hashtbl.create 32 in + let add entity = Hashtbl.replace entities entity () in + let dependents db attr entity = + Datascript.datoms db Datascript.Avet ~a:attr ~v:(Datascript.Ref entity) () + |> Seq.iter (fun (datom : Datascript.datom) -> + visit (); + add datom.e) + in + try + if graph_e2ee before <> state.e2ee || graph_e2ee after <> state.e2ee + then Error "checksum mode change requires snapshot" + else ( + List.iter + (List.iter (fun (datom : Datascript.datom) -> + visit (); + if + List.mem + datom.a + [ "block/uuid" + ; "block/parent" + ; "block/page" + ; "block/order" + ; "block/title" + ; "block/name" + ; "block/tags" + ; "logseq.property/built-in?" + ] + then add datom.e; + let attributes = + match datom.a with + | "block/uuid" -> [ "block/parent"; "block/page" ] + | "db/ident" -> [ "block/tags" ] + | _ -> [] + in + List.iter + (fun attr -> + dependents before attr datom.e; + dependents after attr datom.e) + attributes)) + changes; + let contribution db entity = tuples ~visit db ~e2ee:state.e2ee entity in + let fnv, djb = + Hashtbl.fold + (fun entity () (fnv, djb) -> + let subtract = contribution before entity + and add = contribution after entity in + let fnv, djb = + Tuple_set.fold + (fun tuple (fnv, djb) -> + let a, b = tuple_digest tuple in + Int32.sub fnv a, Int32.sub djb b) + subtract + (fnv, djb) + in + Tuple_set.fold + (fun tuple (fnv, djb) -> + let a, b = tuple_digest tuple in + Int32.add fnv a, Int32.add djb b) + add + (fnv, djb)) + entities + (state.fnv, state.djb) + in + Ok { state with fnv; djb }) + with + | Limit -> Error "checksum delta exceeds change budget" +;; diff --git a/logseq_overlay_db/lib/authoritative_checksum.mli b/logseq_overlay_db/lib/authoritative_checksum.mli index f0cef27f..89cf9f4e 100644 --- a/logseq_overlay_db/lib/authoritative_checksum.mli +++ b/logseq_overlay_db/lib/authoritative_checksum.mli @@ -1,2 +1,15 @@ +type t + +val initialize : ?check_current:(unit -> unit) -> e2ee:bool -> Datascript.db -> t +val to_string : t -> string + +val advance + : maximum:int + -> before:Datascript.db + -> after:Datascript.db + -> changes:Datascript.datom list list + -> t + -> (t, string) result + val graph_e2ee : Datascript.db -> bool -val recompute : e2ee:bool -> Datascript.db -> string +val recompute : ?check_current:(unit -> unit) -> e2ee:bool -> Datascript.db -> string diff --git a/logseq_overlay_db/lib/database.ml b/logseq_overlay_db/lib/database.ml index 1c39180f..bd392a0a 100644 --- a/logseq_overlay_db/lib/database.ml +++ b/logseq_overlay_db/lib/database.ml @@ -1,6 +1,10 @@ (* Warning output is secondary to committed operations and resource cleanup. *) let report_warning message = - try output_string stderr message; flush stderr with _ -> () + try + output_string stderr message; + flush stderr + with + | _ -> () ;; module Graph = Logseq_db_types.Graph_types @@ -8,6 +12,40 @@ module Storage = Logseq_db_storage.Logseq_sqlite_storage open Types open Authoritative_store open Overlay_effect + +let inspect_snapshot_storage ~check_current ~scratch_bytes ~encrypted ~path = + match Storage.open_database path with + | Error _ -> Error "Snapshot storage is invalid" + | Ok connection -> + let callbacks = Storage.connection_callbacks connection in + let closed = ref (Ok ()) in + let result = + Fun.protect + ~finally:(fun () -> closed := Storage.close callbacks) + (fun () -> + check_current (); + match + Storage.startup_metadata connection, Storage.restore_database connection + with + | Ok _, Ok database -> + if Authoritative_checksum.graph_e2ee database <> encrypted + then Error "Snapshot encryption metadata mismatch" + else if Int64.of_int (Unix.stat path).st_size > scratch_bytes + then Error "Snapshot scratch bound" + else + Types.Checksum.of_string + ("checksum:v1:" + ^ Authoritative_checksum.recompute + ~check_current + ~e2ee:encrypted + database) + | _ -> Error "Snapshot storage is invalid") + in + (match !closed, result with + | Error _, _ -> Error "Snapshot verification storage close failed" + | Ok (), result -> result) +;; + module Uuid_map = Map.Make (String) type effect_footprint = footprint @@ -29,6 +67,7 @@ type mirror_inspection = } type retirement_id = Graph.Retirement_id.t + let retirement_id_of_string = Graph.Retirement_id.of_string let retirement_id_to_string = Graph.Retirement_id.to_string @@ -68,9 +107,16 @@ type write_precondition = Mutation_pure_intent.precondition = type local_intent = Mutation_pure_intent.t type intent_commit = Mutation_pure_intent.commit -type intent_resolution = Intent_pending | Intent_committed of intent_commit + +type intent_resolution = + | Intent_pending + | Intent_committed of intent_commit | Intent_rejected of Types.local_commit_error -type local_intent_page = { items : local_intent list; next : int64 option } + +type local_intent_page = + { items : local_intent list + ; next : int64 option + } type outbox_record = Persistence_outbox_v16.t = { mutation_id : Graph.Uuid.t @@ -135,7 +181,11 @@ type task_catalog_cache = ; catalog_items : Graph.task_status_definition array } -type timeline_scope = Journal_index | Page_roots of Graph.page_uuid | Parent_children of Graph.block_uuid +type timeline_scope = + | Journal_index + | Page_roots of Graph.page_uuid + | Parent_children of Graph.block_uuid + type timeline_scope_version = string type journal_position = string type member_position = string @@ -144,17 +194,24 @@ let equal_timeline_scope_version = String.equal let timeline_scope_version_to_string value = value let journal_position_to_string value = value let member_position_to_string value = value + let journal_position_of_string value = - if String.length value <= 512 then Ok value + if String.length value <= 512 + then Ok value else Error (Types.Invalid_read_request "oversized timeline journal position") +;; + let member_position_of_string value = - if String.length value <= 512 then Ok value + if String.length value <= 512 + then Ok value else Error (Types.Invalid_read_request "oversized timeline member position") +;; let timeline_scope_key = function | Journal_index -> "journal" | Page_roots uuid -> "roots:" ^ Graph.Uuid.to_string uuid | Parent_children uuid -> "children:" ^ Graph.Uuid.to_string uuid +;; type timeline_history = { versions : int Uuid_map.t @@ -164,6 +221,7 @@ type timeline_history = let empty_timeline_history () = Array.make 64 { versions = Uuid_map.empty; count = 0; retirement = 0 } +;; let timeline_bucket key = Hashtbl.hash key mod 64 @@ -190,6 +248,7 @@ type t = ; mutable timeline_serial : int ; mutable timeline_proof_available : bool ; mutable timeline_authoritative : Datascript.db + ; mutable authoritative_checksum : Authoritative_checksum.t ; authoritative_connection : Datascript.conn ; authoritative_listener : string ; authoritative_reports : Datascript.tx_report list ref @@ -199,6 +258,7 @@ type t = ; mutable queryable_block_effects : outbox_record Rrbvec.t Uuid_map.t ; mutable queryable_page_effects : outbox_record Rrbvec.t Uuid_map.t ; mutable queryable_children_effects : outbox_record list Uuid_map.t + ; sync_changed : Eio.Condition.t ; mutable sync_revision : int ; mutable durable_sync_revision : int ; mutable durable_outbox : Logseq_db_storage.Sync_outbox_store.row Uuid_map.t @@ -268,6 +328,8 @@ type prepared_outbox_transition = type authoritative_preparation = { authoritative_owner : t + ; authoritative_base : Datascript.db + ; authoritative_digest : Authoritative_checksum.t ; authoritative_revision : int ; authoritative_batch : Types.authoritative_batch ; authoritative_wires : string list @@ -306,6 +368,7 @@ and authoritative_candidate = list ; stale_no_change : (Graph.Uuid.t * Types.submission_batch_id) list ; stale_blocked : (Graph.Uuid.t * Types.submission_batch_id) list + ; authoritative_digest_after : Authoritative_checksum.t ; authoritative_db_after : Datascript.db } @@ -343,7 +406,8 @@ let mirror_generation value = let mirror_generation_for_path path = let stat = Unix.stat path in mirror_generation - (Digestif.SHA256.digest_string (Printf.sprintf "%s:%d:%d" path stat.st_dev stat.st_ino) + (Digestif.SHA256.digest_string + (Printf.sprintf "%s:%d:%d" path stat.st_dev stat.st_ino) |> Digestif.SHA256.to_hex) ;; @@ -557,34 +621,49 @@ let snapshot_crypto_id preparation index = index) ;; -let next_snapshot_unprotection_batch preparation = - if preparation.snapshot_canceled +let next_snapshot_unprotection_batch ?maximum_items ?maximum_bytes preparation = + let maximum_items = + min + preparation.snapshot_dependencies.limits.change_max_items + (Option.value maximum_items ~default:max_int) + in + let maximum_bytes = + min + preparation.snapshot_dependencies.limits.wire_batch_max_bytes + (Option.value maximum_bytes ~default:max_int) + in + if maximum_items <= 0 || maximum_bytes <= 0 + then Error Types.Snapshot_limit_exceeded + else if preparation.snapshot_canceled then Error Types.Snapshot_preparation_canceled else if preparation.snapshot_committed || Option.is_some preparation.snapshot_persisted then Error (Types.Snapshot_state_error Types.Snapshot_next_after_persist_or_commit) else if preparation.snapshot_awaiting_crypto then Error (Types.Snapshot_state_error Types.Snapshot_crypto_batch_pending) else ( - let maximum = preparation.snapshot_dependencies.limits.wire_batch_max_bytes in - let rec take bytes reversed sequence = - match sequence () with - | Seq.Nil -> Ok (List.rev reversed, Seq.empty) - | Seq.Cons (datom, rest) -> - let value = - match datom.Datascript.v with - | Datascript.String value -> value - | _ -> assert false - in - let next = bytes + String.length value in - if next > maximum - then - if reversed = [] - then Error Types.Snapshot_limit_exceeded - else Ok (List.rev reversed, fun () -> Seq.Cons (datom, rest)) - else take next (datom :: reversed) rest + let maximum = maximum_bytes in + let rec take count bytes reversed sequence = + if count = maximum_items + then Ok (List.rev reversed, sequence) + else ( + match sequence () with + | Seq.Nil -> Ok (List.rev reversed, Seq.empty) + | Seq.Cons (datom, rest) -> + let value = + match datom.Datascript.v with + | Datascript.String value -> value + | _ -> assert false + in + let next = bytes + String.length value in + if next > maximum + then + if reversed = [] + then Error Types.Snapshot_limit_exceeded + else Ok (List.rev reversed, fun () -> Seq.Cons (datom, rest)) + else take (count + 1) next (datom :: reversed) rest) in Result.bind - (take 0 [] preparation.snapshot_remaining_protected_datoms) + (take 0 0 [] preparation.snapshot_remaining_protected_datoms) (fun (datoms, remaining) -> preparation.snapshot_remaining_protected_datoms <- remaining; match datoms with @@ -699,10 +778,13 @@ let persist_snapshot_activation preparation = ~applied_server_t:preparation.snapshot_cursor ~checksum:digest |> Result.map_error (fun message -> - let reason = match message with + let reason = + match message with | "sync schema version must be non-negative" -> `Schema_negative - | "sync checksum must be 16 lowercase hexadecimal characters" -> `Checksum_invalid - | _ -> `Unknown in + | "sync checksum must be 16 lowercase hexadecimal characters" -> + `Checksum_invalid + | _ -> `Unknown + in Types.Snapshot_state_error (Types.Snapshot_checkpoint_invalid reason)) in Result.bind metadata (fun metadata -> @@ -784,7 +866,8 @@ let sync_deletion_directory directory = let directory_stat path = let stat = Unix.lstat path in - if stat.st_kind <> Unix.S_DIR then failwith "Local copy path is not a physical directory"; + if stat.st_kind <> Unix.S_DIR + then failwith "Local copy path is not a physical directory"; stat ;; @@ -796,47 +879,69 @@ let same_directory path expected = let deletion_ancestors (location : durable_mirror_location) = let mirrors = Filename.dirname (Filename.dirname location.database_path) in let worker = Filename.dirname mirrors in - List.map (fun path -> path, directory_stat path) [Filename.dirname worker; worker; mirrors] + List.map + (fun path -> path, directory_stat path) + [ Filename.dirname worker; worker; mirrors ] ;; let validate_deletion_ancestors ancestors = - List.iter (fun (path, expected) -> - if not (same_directory path expected) then failwith "Local copy parent directory changed") ancestors + List.iter + (fun (path, expected) -> + if not (same_directory path expected) + then failwith "Local copy parent directory changed") + ancestors ;; let validate_deletion_lease directory = - List.iter (fun suffix -> - let path = Filename.concat directory (".logseq-overlay-owner.sqlite" ^ suffix) in - match Unix.lstat path with - | { st_kind = Unix.S_REG; st_nlink = 1; _ } -> () - | _ -> failwith "Local copy lease path is unsafe" - | exception Unix.Unix_error (Unix.ENOENT, _, _) -> ()) [""; "-journal"; "-wal"; "-shm"] + List.iter + (fun suffix -> + let path = Filename.concat directory (".logseq-overlay-owner.sqlite" ^ suffix) in + match Unix.lstat path with + | { st_kind = Unix.S_REG; st_nlink = 1; _ } -> () + | _ -> failwith "Local copy lease path is unsafe" + | exception Unix.Unix_error (Unix.ENOENT, _, _) -> ()) + [ ""; "-journal"; "-wal"; "-shm" ] ;; let directory_generation_as ~identity_path directory = let stat = directory_stat directory in let lease = Unix.lstat (Filename.concat directory ".logseq-overlay-owner.sqlite") in - if lease.st_kind <> Unix.S_REG || lease.st_nlink <> 1 then failwith "Local copy lease path is unsafe"; + if lease.st_kind <> Unix.S_REG || lease.st_nlink <> 1 + then failwith "Local copy lease path is unsafe"; (* The held lease inode cannot be reused while this preparation is alive. Its identity fences directory-inode reuse without a new persisted marker. *) - mirror_generation (Digestif.SHA256.digest_string - (Printf.sprintf "directory:%s:%d:%d:%d:%d" identity_path stat.st_dev stat.st_ino lease.st_dev lease.st_ino) - |> Digestif.SHA256.to_hex) + mirror_generation + (Digestif.SHA256.digest_string + (Printf.sprintf + "directory:%s:%d:%d:%d:%d" + identity_path + stat.st_dev + stat.st_ino + lease.st_dev + lease.st_ino) + |> Digestif.SHA256.to_hex) ;; let mirror_deletion_directory (location : durable_mirror_location) = Filename.concat - (Filename.concat (Filename.dirname (Filename.dirname location.database_path)) ".mirror-deletions") + (Filename.concat + (Filename.dirname (Filename.dirname location.database_path)) + ".mirror-deletions") (Graph.Uuid.to_string location.graph_id) ;; let validate_retirement_directories (location : durable_mirror_location) = let root = mirror_deletion_directory location in - List.iter (fun path -> - match directory_stat path with - | _ -> () - | exception Unix.Unix_error (Unix.ENOENT, _, _) -> ()) - [Filename.dirname root; root; Filename.concat root "states"; Filename.concat root "pending"] + List.iter + (fun path -> + match directory_stat path with + | _ -> () + | exception Unix.Unix_error (Unix.ENOENT, _, _) -> ()) + [ Filename.dirname root + ; root + ; Filename.concat root "states" + ; Filename.concat root "pending" + ] ;; let rec ensure_retirement_directory path = @@ -857,21 +962,32 @@ let mirror_deletion_paths (location : durable_mirror_location) retirement = ;; let write_deletion_receipt path (result : Types.mirror_delete) committed = - let data = Yojson.Safe.to_string (`Assoc - [ "version", `Int 2 - ; "graph", Graph.Uuid.to_yojson (Option.get result.graph_uuid) - ; "retirement", `String (Filename.chop_suffix (Filename.basename path) ".json") + let data = + Yojson.Safe.to_string + (`Assoc + [ "version", `Int 2 + ; "graph", Graph.Uuid.to_yojson (Option.get result.graph_uuid) + ; "retirement", `String (Filename.chop_suffix (Filename.basename path) ".json") ; ( "previous" , `String (Types.Mirror_generation.to_string result.previous_generation) ) - ; "absent", `String (Types.Mirror_generation.to_string result.absent_generation) - ; "committed", `Bool committed ]) in - let temporary, output = Filename.open_temp_file ~temp_dir:(Filename.dirname path) ".retirement-" ".part" in - Fun.protect ~finally:(fun () -> close_out_noerr output; - if Sys.file_exists temporary then Sys.remove temporary) (fun () -> - output_string output data; flush output; - Unix.fsync (Unix.descr_of_out_channel output); close_out output; - Unix.rename temporary path; - sync_deletion_directory (Filename.dirname path)) + ; "absent", `String (Types.Mirror_generation.to_string result.absent_generation) + ; "committed", `Bool committed + ]) + in + let temporary, output = + Filename.open_temp_file ~temp_dir:(Filename.dirname path) ".retirement-" ".part" + in + Fun.protect + ~finally:(fun () -> + close_out_noerr output; + if Sys.file_exists temporary then Sys.remove temporary) + (fun () -> + output_string output data; + flush output; + Unix.fsync (Unix.descr_of_out_channel output); + close_out output; + Unix.rename temporary path; + sync_deletion_directory (Filename.dirname path)) ;; let read_deletion_receipt (location : durable_mirror_location) receipt_path = @@ -882,60 +998,92 @@ let read_deletion_receipt (location : durable_mirror_location) receipt_path = let json = Yojson.Safe.from_file receipt_path in let version = json |> member "version" |> to_int in if version <> 1 && version <> 2 then failwith "unsupported retirement receipt"; - if json |> member "graph" |> to_string <> Graph.Uuid.to_string location.graph_id - || json |> member "retirement" |> to_string <> Filename.chop_suffix (Filename.basename receipt_path) ".json" + if + json |> member "graph" |> to_string <> Graph.Uuid.to_string location.graph_id + || json + |> member "retirement" + |> to_string + <> Filename.chop_suffix (Filename.basename receipt_path) ".json" then failwith "retirement receipt identity mismatch"; - let generation name = json |> member name |> to_string - |> Types.Mirror_generation.of_string |> Result.get_ok in - ({ Types.graph_uuid = Some location.graph_id - ; previous_generation = generation "previous"; absent_generation = generation "absent" }, - json |> member "committed" |> to_bool, version) + let generation name = + json |> member name |> to_string |> Types.Mirror_generation.of_string |> Result.get_ok + in + ( { Types.graph_uuid = Some location.graph_id + ; previous_generation = generation "previous" + ; absent_generation = generation "absent" + } + , json |> member "committed" |> to_bool + , version ) ;; let collect_retired_mirrors (location : durable_mirror_location) = ignore (deletion_ancestors location); validate_retirement_directories location; let root = Filename.concat (mirror_deletion_directory location) "pending" in - if Sys.file_exists root then - Sys.readdir root |> Array.iter (fun name -> + if Sys.file_exists root + then + Sys.readdir root + |> Array.iter (fun name -> if Filename.check_suffix name ".data" then ( try - let id = String.sub name 0 (String.length name - 5) |> retirement_id_of_string |> Result.get_ok in + let id = + String.sub name 0 (String.length name - 5) + |> retirement_id_of_string + |> Result.get_ok + in let _, receipt, data = mirror_deletion_paths location id in let _, committed, _ = read_deletion_receipt location receipt in - if committed && Sys.file_exists data then ( + if committed && Sys.file_exists data + then ( ignore (directory_stat data); validate_deletion_lease data; match Ownership.acquire ~graph_directory:data with | Error Ownership.Already_owned -> () | Error (Ownership.Unavailable _) -> - report_warning "journal: phase=cleanup operation=collectRetiredMirrors code=ownershipUnavailable\n" - | Ok owner -> Fun.protect ~finally:(fun () -> ignore (Ownership.release owner)) - (fun () -> remove_mirror_tree data; sync_deletion_directory root)) + report_warning + "journal: phase=cleanup operation=collectRetiredMirrors \ + code=ownershipUnavailable\n" + | Ok owner -> + Fun.protect + ~finally:(fun () -> ignore (Ownership.release owner)) + (fun () -> + remove_mirror_tree data; + sync_deletion_directory root)) with - | _ -> report_warning "journal: phase=cleanup operation=collectRetiredMirrors code=collectionPending\n")) + | _ -> + report_warning + "journal: phase=cleanup operation=collectRetiredMirrors code=collectionPending\n")) ;; let prepare_mirror_deletion ~application_support_directory ~graph_id ~retirement = let ( let* ) = Result.bind in - let* location = durable_mirror_location ~application_support_directory ~graph_id - |> Result.map_error (fun _ -> Types.Mirror_delete_failed "Invalid local copy directory") in + let* location = + durable_mirror_location ~application_support_directory ~graph_id + |> Result.map_error (fun _ -> + Types.Mirror_delete_failed "Invalid local copy directory") + in try let ancestors = deletion_ancestors location in validate_retirement_directories location; let _root, receipt, data = mirror_deletion_paths location retirement in let graph_directory = Filename.dirname location.database_path in - let exists path = match Unix.lstat path with + let exists path = + match Unix.lstat path with | _ -> true - | exception Unix.Unix_error (Unix.ENOENT, _, _) -> false in - let read_existing () = if exists receipt then Some (read_deletion_receipt location receipt) else None in + | exception Unix.Unix_error (Unix.ENOENT, _, _) -> false + in + let read_existing () = + if exists receipt then Some (read_deletion_receipt location receipt) else None + in let existing = read_existing () in let preparation ?ownership ?committed ~durable generation = ( { deletion_location = location ; deletion_generation = generation ; deletion_ancestors = ancestors - ; deletion_retirement = retirement; deletion_ownership = ownership; deletion_committed = committed + ; deletion_retirement = retirement + ; deletion_ownership = ownership + ; deletion_committed = committed ; deletion_cancelled = false ; deletion_durable = durable } @@ -947,25 +1095,35 @@ let prepare_mirror_deletion ~application_support_directory ~graph_id ~retirement sync_deletion_directory (Filename.dirname data); sync_deletion_directory (Filename.dirname graph_directory); write_deletion_receipt receipt result true; - Ok (preparation ~committed:result ~durable:true result.previous_generation) in - if committed then persist_commit () else ( + Ok (preparation ~committed:result ~durable:true result.previous_generation) + in + if committed + then persist_commit () + else ( let observed = directory_stat data in validate_deletion_lease data; match Ownership.acquire ~graph_directory:data with | Error Ownership.Already_owned -> Error Types.Mirror_delete_busy | Error (Unavailable message) -> Error (Types.Mirror_delete_failed message) - | Ok owner -> Fun.protect ~finally:(fun () -> ignore (Ownership.release owner)) (fun () -> - validate_deletion_ancestors ancestors; - validate_retirement_directories location; + | Ok owner -> + Fun.protect + ~finally:(fun () -> ignore (Ownership.release owner)) + (fun () -> + validate_deletion_ancestors ancestors; + validate_retirement_directories location; if (not (same_directory data observed)) || read_existing () <> existing - then failwith "retired mirror changed while acquiring ownership"; - (* An obsolete transaction can only finish its already-retired namespace; + then failwith "retired mirror changed while acquiring ownership"; + (* An obsolete transaction can only finish its already-retired namespace; it never receives a new canonical rename capability. Its generation is only the existing coordinator's opaque identity. *) - if version = 2 && not (Types.Mirror_generation.equal result.previous_generation - (directory_generation_as ~identity_path:graph_directory data)) - then failwith "retired mirror identity mismatch"; - persist_commit ())) + if + version = 2 + && not + (Types.Mirror_generation.equal + result.previous_generation + (directory_generation_as ~identity_path:graph_directory data)) + then failwith "retired mirror identity mismatch"; + persist_commit ())) | Some (old, false, 1) when not (exists graph_directory) -> (* With neither namespace present there is nothing to commit. Recheck the receipt and destination rather than inferring commit from absence. *) @@ -973,7 +1131,7 @@ let prepare_mirror_deletion ~application_support_directory ~graph_id ~retirement if exists data || read_existing () <> existing then Error Types.Mirror_delete_stale else ( - let canceled, _ = preparation ~durable:false old.previous_generation in + let canceled, _ = preparation ~durable:false old.previous_generation in Ok ({ canceled with deletion_cancelled = true }, old.previous_generation)) | _ -> let observed = directory_stat graph_directory in @@ -987,31 +1145,39 @@ let prepare_mirror_deletion ~application_support_directory ~graph_id ~retirement Error error in (try - validate_deletion_ancestors ancestors; + validate_deletion_ancestors ancestors; if (not (same_directory graph_directory observed)) || exists data || read_existing () <> existing then fail Types.Mirror_delete_stale else ( - let generation = directory_generation_as ~identity_path:graph_directory graph_directory in - let result = { Types.graph_uuid = Some graph_id; previous_generation = generation + let generation = + directory_generation_as ~identity_path:graph_directory graph_directory + in + let result = + { Types.graph_uuid = Some graph_id + ; previous_generation = generation ; absent_generation = absent_mirror_generation location.database_path } in match existing with - | Some (old, false, 1) -> - (* A pre-rename V1 preparation can only be canceled after acquiring + | Some (old, false, 1) -> + (* A pre-rename V1 preparation can only be canceled after acquiring the live lease. Never grant it a fresh rename capability. *) - let canceled, _ = preparation ~ownership:owner ~durable:false generation in - Ok ({ canceled with deletion_cancelled = true }, old.previous_generation) - | Some (old, _, _) when not (Types.Mirror_generation.equal old.previous_generation generation) -> - fail Types.Mirror_delete_stale - | _ -> - ensure_retirement_directory (Filename.dirname receipt); - ensure_retirement_directory (Filename.dirname data); - write_deletion_receipt receipt result false; - Ok (preparation ~ownership:owner ~durable:false generation)) + let canceled, _ = + preparation ~ownership:owner ~durable:false generation + in + Ok ({ canceled with deletion_cancelled = true }, old.previous_generation) + | Some (old, _, _) + when not + (Types.Mirror_generation.equal old.previous_generation generation) + -> fail Types.Mirror_delete_stale + | _ -> + ensure_retirement_directory (Filename.dirname receipt); + ensure_retirement_directory (Filename.dirname data); + write_deletion_receipt receipt result false; + Ok (preparation ~ownership:owner ~durable:false generation)) with | exn -> fail (Types.Mirror_delete_failed (Printexc.to_string exn)))) with @@ -1022,18 +1188,28 @@ let prepare_mirror_deletion ~application_support_directory ~graph_id ~retirement let mirror_deletion_committed preparation = preparation.deletion_durable let cancel_mirror_deletion_checked preparation = - let released=match preparation.deletion_ownership with None->Ok()|Some owner->Ownership.release owner in - Result.map(fun()-> - preparation.deletion_ownership<-None; - if preparation.deletion_committed=None then preparation.deletion_cancelled<-true)released + let released = + match preparation.deletion_ownership with + | None -> Ok () + | Some owner -> Ownership.release owner + in + Result.map + (fun () -> + preparation.deletion_ownership <- None; + if preparation.deletion_committed = None + then preparation.deletion_cancelled <- true) + released ;; -let cancel_mirror_deletion preparation = ignore(cancel_mirror_deletion_checked preparation) +let cancel_mirror_deletion preparation = + ignore (cancel_mirror_deletion_checked preparation) ;; let commit_mirror_deletion preparation = let location = preparation.deletion_location in - let _root, receipt, data = mirror_deletion_paths location preparation.deletion_retirement in + let _root, receipt, data = + mirror_deletion_paths location preparation.deletion_retirement + in let durable = try validate_deletion_ancestors preparation.deletion_ancestors; @@ -1050,55 +1226,68 @@ let commit_mirror_deletion preparation = validate_deletion_ancestors preparation.deletion_ancestors; validate_retirement_directories location; sync_deletion_directory (Filename.dirname data); - sync_deletion_directory (Filename.dirname (Filename.dirname location.database_path)); + sync_deletion_directory + (Filename.dirname (Filename.dirname location.database_path)); write_deletion_receipt receipt result true; preparation.deletion_durable <- true; collect_retired_mirrors location - with _ -> report_warning "journal: phase=retirement operation=confirmMirrorRetirement code=confirmationPending\n"); + with + | _ -> + report_warning + "journal: phase=retirement operation=confirmMirrorRetirement \ + code=confirmationPending\n"); Ok result - | None, None when preparation.deletion_cancelled || preparation.deletion_ownership = None -> Error Types.Mirror_delete_stale + | None, None + when preparation.deletion_cancelled || preparation.deletion_ownership = None -> + Error Types.Mirror_delete_stale | None, None -> let graph_directory = Filename.dirname location.database_path in - let unchanged = try - validate_deletion_ancestors preparation.deletion_ancestors; - validate_retirement_directories location; - validate_deletion_lease graph_directory; - Types.Mirror_generation.equal preparation.deletion_generation - (directory_generation_as ~identity_path:graph_directory graph_directory) + let unchanged = + try + validate_deletion_ancestors preparation.deletion_ancestors; + validate_retirement_directories location; + validate_deletion_lease graph_directory; + Types.Mirror_generation.equal + preparation.deletion_generation + (directory_generation_as ~identity_path:graph_directory graph_directory) with | _ -> false in if not unchanged then Error Types.Mirror_delete_stale else ( - let renamed = - try - if Sys.file_exists data then failwith "retirement destination is occupied"; - let result, _, _ = read_deletion_receipt location receipt in - Unix.rename graph_directory data; - Ok result + let renamed = + try + if Sys.file_exists data then failwith "retirement destination is occupied"; + let result, _, _ = read_deletion_receipt location receipt in + Unix.rename graph_directory data; + Ok result with | exn -> Error (Types.Mirror_delete_failed (Printexc.to_string exn)) in match renamed with - | Error _ as error -> error - | Ok result -> - preparation.deletion_committed <- Some result; + | Error _ as error -> error + | Ok result -> + preparation.deletion_committed <- Some result; (try Option.iter (fun owner -> ignore (Ownership.release owner)) preparation.deletion_ownership with | _ -> ()); - preparation.deletion_ownership <- None; - (try - sync_deletion_directory (Filename.dirname data); - sync_deletion_directory (Filename.dirname graph_directory); - write_deletion_receipt receipt result true; - preparation.deletion_durable <- true; - collect_retired_mirrors location - with _ -> report_warning "journal: phase=retirement operation=confirmMirrorRetirement code=confirmationPending\n"); - Ok result) + preparation.deletion_ownership <- None; + (try + sync_deletion_directory (Filename.dirname data); + sync_deletion_directory (Filename.dirname graph_directory); + write_deletion_receipt receipt result true; + preparation.deletion_durable <- true; + collect_retired_mirrors location + with + | _ -> + report_warning + "journal: phase=retirement operation=confirmMirrorRetirement \ + code=confirmationPending\n"); + Ok result) ;; let validate_receipts sqlite = @@ -1160,7 +1349,11 @@ let collect_garbage inspection = in Fun.protect ~finally:(fun () -> - (if not (Sqlite3.db_close sqlite) then report_warning "journal: phase=cleanup operation=closeSqlite scope=temporaryStorageHandle code=closePending recovery=reopen\n" ); + if not (Sqlite3.db_close sqlite) + then + report_warning + "journal: phase=cleanup operation=closeSqlite \ + scope=temporaryStorageHandle code=closePending recovery=reopen\n"; ignore (Ownership.release ownership)) (fun () -> match @@ -2011,7 +2204,12 @@ let decode_outbox records = let read_outbox path = let sqlite = Sqlite3.db_open ~mode:`NO_CREATE path in Fun.protect - ~finally:(fun () -> (if not (Sqlite3.db_close sqlite) then report_warning "journal: phase=cleanup operation=closeSqlite scope=temporaryStorageHandle code=closePending recovery=reopen\n" )) + ~finally:(fun () -> + if not (Sqlite3.db_close sqlite) + then + report_warning + "journal: phase=cleanup operation=closeSqlite scope=temporaryStorageHandle \ + code=closePending recovery=reopen\n") (fun () -> let exec sql = let rc = Sqlite3.exec sqlite sql in @@ -2020,7 +2218,8 @@ let read_outbox path = Result.bind (exec "BEGIN") (fun () -> let result = try - Result.bind (Logseq_db_storage.Sync_outbox_store.read_database sqlite) + Result.bind + (Logseq_db_storage.Sync_outbox_store.read_database sqlite) (fun records -> Result.map (fun revision -> records, revision) @@ -2029,11 +2228,15 @@ let read_outbox path = | exn -> Error (Printexc.to_string exn) in match result with - | Error _ as error -> ignore (exec "ROLLBACK"); error + | Error _ as error -> + ignore (exec "ROLLBACK"); + error | Ok _ -> (match exec "COMMIT" with | Ok () -> result - | Error _ as error -> ignore (exec "ROLLBACK"); error))) + | Error _ as error -> + ignore (exec "ROLLBACK"); + error))) ;; (* Global revision belongs to metadata. Frozen encodings are independent of it @@ -2042,21 +2245,23 @@ let durable_outbox_rows records = List.fold_left (fun rows (record : outbox_record) -> let mutation_id = Graph.Uuid.to_string record.mutation_id in - Uuid_map.add mutation_id + Uuid_map.add + mutation_id Logseq_db_storage.Sync_outbox_store. { mutation_id ; sequence = record.sequence ; record = Persistence_outbox_v16.encode ~sync_revision:0 record } rows) - Uuid_map.empty records + Uuid_map.empty + records ;; let ordered_durable_rows rows = - Uuid_map.bindings rows |> List.map snd - |> List.sort - (fun (left : Logseq_db_storage.Sync_outbox_store.row) right -> - Int.compare left.sequence right.sequence) + Uuid_map.bindings rows + |> List.map snd + |> List.sort (fun (left : Logseq_db_storage.Sync_outbox_store.row) right -> + Int.compare left.sequence right.sequence) ;; let outbox_write_set database ~revision records = @@ -2067,15 +2272,16 @@ let outbox_write_set database ~revision records = match Uuid_map.find_opt id database.durable_outbox with | Some before when before = row -> changed | _ -> row :: changed) - rows [] - |> List.sort - (fun (left : Logseq_db_storage.Sync_outbox_store.row) right -> - Int.compare left.sequence right.sequence) + rows + [] + |> List.sort (fun (left : Logseq_db_storage.Sync_outbox_store.row) right -> + Int.compare left.sequence right.sequence) in let deletes = Uuid_map.fold (fun id _ deleted -> if Uuid_map.mem id rows then deleted else id :: deleted) - database.durable_outbox [] + database.durable_outbox + [] in ( Logseq_db_storage.Sync_outbox_store. { expected_revision = database.durable_sync_revision; revision; upserts; deletes } @@ -2084,24 +2290,43 @@ let outbox_write_set database ~revision records = let migrate_outbox path ~revision ~expected_records rows = let sqlite = Sqlite3.db_open ~mode:`NO_CREATE path in - Fun.protect ~finally:(fun () -> (if not (Sqlite3.db_close sqlite) then report_warning "journal: phase=cleanup operation=closeSqlite scope=temporaryStorageHandle code=closePending recovery=reopen\n" )) (fun () -> - Logseq_db_storage.Sync_outbox_store.migrate_database sqlite ~revision ~expected_records - (ordered_durable_rows rows)) + Fun.protect + ~finally:(fun () -> + if not (Sqlite3.db_close sqlite) + then + report_warning + "journal: phase=cleanup operation=closeSqlite scope=temporaryStorageHandle \ + code=closePending recovery=reopen\n") + (fun () -> + Logseq_db_storage.Sync_outbox_store.migrate_database + sqlite + ~revision + ~expected_records + (ordered_durable_rows rows)) ;; let initialize_durable_receipts path = let sqlite = Sqlite3.db_open ~mode:`NO_CREATE path in Fun.protect - ~finally:(fun () -> (if not (Sqlite3.db_close sqlite) then report_warning "journal: phase=cleanup operation=closeSqlite scope=temporaryStorageHandle code=closePending recovery=reopen\n" )) - (fun () -> - Logseq_db_storage.Mutation_receipt_store.initialize_database sqlite) + ~finally:(fun () -> + if not (Sqlite3.db_close sqlite) + then + report_warning + "journal: phase=cleanup operation=closeSqlite scope=temporaryStorageHandle \ + code=closePending recovery=reopen\n") + (fun () -> Logseq_db_storage.Mutation_receipt_store.initialize_database sqlite) ;; let with_receipt_database path read = try let sqlite = Sqlite3.db_open ~mode:`NO_CREATE path in Fun.protect - ~finally:(fun () -> (if not (Sqlite3.db_close sqlite) then report_warning "journal: phase=cleanup operation=closeSqlite scope=temporaryStorageHandle code=closePending recovery=reopen\n" )) + ~finally:(fun () -> + if not (Sqlite3.db_close sqlite) + then + report_warning + "journal: phase=cleanup operation=closeSqlite scope=temporaryStorageHandle \ + code=closePending recovery=reopen\n") (fun () -> read sqlite) with | exn -> Error (Printexc.to_string exn) @@ -2110,30 +2335,30 @@ let with_receipt_database path read = let read_durable_receipt path mutation_id = let key = Persistence_receipt_v1.receipt_key mutation_id in with_receipt_database path (fun sqlite -> - Result.bind - (Logseq_db_storage.Mutation_receipt_store.read_mutation sqlite key) - (function - | None -> Ok None - | Some source -> - Result.map - Option.some - (Persistence_receipt_v1.decode_mutation - ~startup_generation - ~projection_revision:(projection_revision 0) - (key, source)))) + Result.bind + (Logseq_db_storage.Mutation_receipt_store.read_mutation sqlite key) + (function + | None -> Ok None + | Some source -> + Result.map + Option.some + (Persistence_receipt_v1.decode_mutation + ~startup_generation + ~projection_revision:(projection_revision 0) + (key, source)))) ;; let read_terminal_batch database batch_id = let key = Persistence_receipt_v1.terminal_batch_key batch_id in with_receipt_database database.database_path (fun sqlite -> - Result.bind - (Logseq_db_storage.Mutation_receipt_store.read_terminal_batch sqlite key) - (function - | None -> Ok None - | Some source -> - Result.map - Option.some - (Persistence_receipt_v1.decode_terminal_batch (key, source)))) + Result.bind + (Logseq_db_storage.Mutation_receipt_store.read_terminal_batch sqlite key) + (function + | None -> Ok None + | Some source -> + Result.map + Option.some + (Persistence_receipt_v1.decode_terminal_batch (key, source)))) ;; let record_is_logically_active (record : outbox_record) = @@ -2161,6 +2386,7 @@ let close database = then false, [], [], [] else ( database.closed <- true; + Eio.Condition.broadcast database.sync_changed; database.timeline_versions <- empty_timeline_history (); database.release_slot := None; let callback_waiters = @@ -2233,14 +2459,21 @@ let close database = match storage_result, ownership_result with | Ok (), Ok () -> Ok () | Error error, _ -> - let reason,source=match error with - | Logseq_db_storage.Storage_session.Closed -> "databaseClosed",None - | Fatal s -> "fatalState",Some s | Stage_failed s -> "storageWriteFailed",Some s - | Persistence_failed s -> "closeFailed",Some s | Already_consumed -> "preparationConsumed",None in - let module D=Logseq_db_types.Failure_diagnostic in - let diagnostic=match source with None->D.create ~operation:"closeDatabase" ~reason () - | Some source->D.of_source ~operation:"closeDatabase" ~reason source in - Error (Types.Close_failed(D.encode diagnostic)) + let reason, source = + match error with + | Logseq_db_storage.Storage_session.Closed -> "databaseClosed", None + | Fatal s -> "fatalState", Some s + | Stage_failed s -> "storageWriteFailed", Some s + | Persistence_failed s -> "closeFailed", Some s + | Already_consumed -> "preparationConsumed", None + in + let module D = Logseq_db_types.Failure_diagnostic in + let diagnostic = + match source with + | None -> D.create ~operation:"closeDatabase" ~reason () + | Some source -> D.of_source ~operation:"closeDatabase" ~reason source + in + Error (Types.Close_failed (D.encode diagnostic)) | Ok (), Error message -> Error (Types.Close_failed message)) else Ok () ;; @@ -2287,7 +2520,7 @@ let frozen_outbox (records : outbox_record list) = records ;; -let page_projection_effects (record:outbox_record) = +let page_projection_effects (record : outbox_record) = match record.mutation with | Types.Append_journal _ when not record.append_create -> [] | _ -> record.effect_footprint.page_uuids @@ -2519,17 +2752,21 @@ let assigned_orders (record : outbox_record) = let assigned_order orders uuid = Uuid_map.find (Graph.Uuid.to_string uuid) orders -let cached_insert_orders cache (record:outbox_record) = - memoized cache.insert_orders (Graph.Uuid.to_string record.mutation_id)(fun()->assigned_orders record) +let cached_insert_orders cache (record : outbox_record) = + memoized cache.insert_orders (Graph.Uuid.to_string record.mutation_id) (fun () -> + assigned_orders record) ;; -let cached_append_node cache (record:outbox_record) root uuid = - let nodes=memoized cache.append_nodes (Graph.Uuid.to_string record.mutation_id)(fun()-> - let rec index parent found (node:Types.append_node) = - let found=Uuid_map.add(Graph.Uuid.to_string node.uuid)(node,parent)found in - Rrbvec.fold_left(index node.uuid)found node.children in - index(List.hd record.effect_footprint.page_uuids)Uuid_map.empty root) in - Uuid_map.find_opt(Graph.Uuid.to_string uuid)nodes +let cached_append_node cache (record : outbox_record) root uuid = + let nodes = + memoized cache.append_nodes (Graph.Uuid.to_string record.mutation_id) (fun () -> + let rec index parent found (node : Types.append_node) = + let found = Uuid_map.add (Graph.Uuid.to_string node.uuid) (node, parent) found in + Rrbvec.fold_left (index node.uuid) found node.children + in + index (List.hd record.effect_footprint.page_uuids) Uuid_map.empty root) + in + Uuid_map.find_opt (Graph.Uuid.to_string uuid) nodes ;; let rec tree_children parent (tree : Types.block_tree) = @@ -2546,13 +2783,17 @@ let rec tree_children_interests (tree : Types.block_tree) = ;; let append_uuids root = - Overlay_planner.append_nodes root |> Rrbvec.map(fun(node:Types.append_node)->node.uuid) + Overlay_planner.append_nodes root + |> Rrbvec.map (fun (node : Types.append_node) -> node.uuid) |> Rrbvec.to_list ;; let append_interests root = - Overlay_planner.append_nodes root |> Rrbvec.filter_map(fun(node:Types.append_node)-> - if Rrbvec.is_empty node.children then None else Some(Types.Children_interest node.uuid)) + Overlay_planner.append_nodes root + |> Rrbvec.filter_map (fun (node : Types.append_node) -> + if Rrbvec.is_empty node.children + then None + else Some (Types.Children_interest node.uuid)) |> Rrbvec.to_list ;; @@ -2674,8 +2915,8 @@ let logical_page_at ?cache ?initial (snapshot : snapshot) uuid = else ( match record.mutation with | Types.Append_journal { title; journal_day; _ } - when record.append_create && record.effect_footprint.page_uuids=[uuid] -> - let page=uuid in + when record.append_create && record.effect_footprint.page_uuids = [ uuid ] -> + let page = uuid in let properties = [] |> set_property_values @@ -2931,23 +3172,54 @@ let logical_block_at ?cache ?initial (snapshot : snapshot) uuid = } }) current - | Append_journal {root;_} -> - let page=List.hd record.effect_footprint.page_uuids in - let node=match cache with - | Some cache->cached_append_node cache record root uuid - | None->Overlay_planner.append_parent ~parent:page ~target:uuid root in + | Append_journal { root; _ } -> + let page = List.hd record.effect_footprint.page_uuids in + let node = + match cache with + | Some cache -> cached_append_node cache record root uuid + | None -> Overlay_planner.append_parent ~parent:page ~target:uuid root + in (match node with - | None->current - | Some(node,parent)-> - let leaf:Types.block_tree={uuid=node.uuid;title=node.title;children=[]} in - let inserted=inserted_block ?cache authoritative ~target:uuid ~parent ~page - ~orders:(match cache with Some cache->cached_insert_orders cache record|None->assigned_orders record) - ~now:record.intent_time_ms leaf |> Option.get in - let properties=match node.task_status with None->inserted.block.properties - |Some status->(match status_entity authoritative (Types.Status_uuid status) with None->inserted.block.properties - |Some entity->set_property_values ?cache authoritative "logseq.property/status" - [Datascript.Ref entity] inserted.block.properties) in - Some{inserted with block={inserted.block with properties};task_status=Option.map(fun id->Types.Status_uuid id)node.task_status}) + | None -> current + | Some (node, parent) -> + let leaf : Types.block_tree = + { uuid = node.uuid; title = node.title; children = [] } + in + let inserted = + inserted_block + ?cache + authoritative + ~target:uuid + ~parent + ~page + ~orders: + (match cache with + | Some cache -> cached_insert_orders cache record + | None -> assigned_orders record) + ~now:record.intent_time_ms + leaf + |> Option.get + in + let properties = + match node.task_status with + | None -> inserted.block.properties + | Some status -> + (match status_entity authoritative (Types.Status_uuid status) with + | None -> inserted.block.properties + | Some entity -> + set_property_values + ?cache + authoritative + "logseq.property/status" + [ Datascript.Ref entity ] + inserted.block.properties) + in + Some + { inserted with + block = { inserted.block with properties } + ; task_status = + Option.map (fun id -> Types.Status_uuid id) node.task_status + }) | Insert_blocks { parent; tree; asset; _ } -> let page = match record.effect_footprint.page_uuids with @@ -3397,11 +3669,18 @@ let local_child_facts ?cache (snapshot : snapshot) parent = else ( let inserted = match record.mutation with - | Types.Append_journal{root;_} -> - let children=if record.effect_footprint.page_uuids=[parent] then Rrbvec.singleton root - else Overlay_planner.append_children root parent in - let orders=assigned_orders record in - Rrbvec.map(fun(child:Types.append_node)->assigned_order orders child.uuid,child.uuid)children |> Rrbvec.to_list + | Types.Append_journal { root; _ } -> + let children = + if record.effect_footprint.page_uuids = [ parent ] + then Rrbvec.singleton root + else Overlay_planner.append_children root parent + in + let orders = assigned_orders record in + Rrbvec.map + (fun (child : Types.append_node) -> + assigned_order orders child.uuid, child.uuid) + children + |> Rrbvec.to_list | Types.Insert_blocks { parent = candidate; tree; _ } -> let children = if Graph.Uuid.equal parent candidate @@ -3546,9 +3825,12 @@ let local_journal_candidates (snapshot : snapshot) = List.fold_left (fun candidates (record : outbox_record) -> match record.mutation with - | Types.Append_journal{journal_day;_} when record_is_logically_active record -> - let page=List.hd record.effect_footprint.page_uuids in - Uuid_map.add(Graph.Uuid.to_string page){day=journal_day;uuid=page;entity=None} candidates + | Types.Append_journal { journal_day; _ } when record_is_logically_active record -> + let page = List.hd record.effect_footprint.page_uuids in + Uuid_map.add + (Graph.Uuid.to_string page) + { day = journal_day; uuid = page; entity = None } + candidates | Types.Create_journal_page { page; journal_day; _ } when record_is_logically_active record -> Uuid_map.add @@ -4040,7 +4322,8 @@ let get_journals snapshot ~from_day ~through_day ~limit ~cursor = retain logical_page_at's existing fallback. *) let value = match initial with - | Some page -> logical_page_at ~cache ~initial:(Some page) snapshot candidate.uuid + | Some page -> + logical_page_at ~cache ~initial:(Some page) snapshot candidate.uuid | None -> logical_page_at ~cache snapshot candidate.uuid in Option.get value @@ -4178,7 +4461,6 @@ let authoritative_tree_candidate snapshot cache uuid = ;; let tree_insert_orders cache record = cached_insert_orders cache.hydration record -;; let logical_tree_candidate snapshot cache uuid = let key = Graph.Uuid.to_string uuid in @@ -4215,10 +4497,15 @@ let logical_tree_candidate snapshot cache uuid = then current else ( match record.mutation with - | Types.Append_journal{root;_} -> + | Types.Append_journal { root; _ } -> (match cached_append_node cache.hydration record root uuid with - | Some(_,parent)->Some{tree_uuid=uuid;tree_parent=parent;tree_order=assigned_order(tree_insert_orders cache record)uuid} - | None->current) + | Some (_, parent) -> + Some + { tree_uuid = uuid + ; tree_parent = parent + ; tree_order = assigned_order (tree_insert_orders cache record) uuid + } + | None -> current) | Types.Insert_blocks { parent; tree; _ } -> let rec find parent (tree : Types.block_tree) = if Graph.Uuid.equal tree.uuid uuid @@ -4277,10 +4564,12 @@ let ordered_tree_children snapshot cache parent = else ( let inserted = match record.mutation with - | Types.Append_journal{root;_} -> - (if record.effect_footprint.page_uuids=[parent] then Rrbvec.singleton root + | Types.Append_journal { root; _ } -> + (if record.effect_footprint.page_uuids = [ parent ] + then Rrbvec.singleton root else Overlay_planner.append_children root parent) - |> Rrbvec.map(fun(tree:Types.append_node)->tree.uuid) |> Rrbvec.to_list + |> Rrbvec.map (fun (tree : Types.append_node) -> tree.uuid) + |> Rrbvec.to_list | Types.Insert_blocks { parent = candidate; tree; _ } -> (if Graph.Uuid.equal parent candidate then [ tree ] @@ -4502,15 +4791,22 @@ let get_structure snapshot request = ;; let timeline_version_at (snapshot : snapshot) scope = - if not snapshot.timeline_proof_available then Error Types.Read_limit_exceeded - else + if not snapshot.timeline_proof_available + then Error Types.Read_limit_exceeded + else ( let key = timeline_scope_key scope in let history = snapshot.timeline_versions.(timeline_bucket key) in - let kind, serial = match Uuid_map.find_opt key history.versions with + let kind, serial = + match Uuid_map.find_opt key history.versions with | Some serial -> "written", serial - | None -> "retained", history.retirement in - Ok (Printf.sprintf "timeline:v2:%s:%s:%d" - (Types.Generation.to_string snapshot.version.generation) kind serial) + | None -> "retained", history.retirement + in + Ok + (Printf.sprintf + "timeline:v2:%s:%s:%d" + (Types.Generation.to_string snapshot.version.generation) + kind + serial)) ;; let timeline_scope_version snapshot scope = @@ -4519,29 +4815,51 @@ let timeline_scope_version snapshot scope = let get_timeline_journals snapshot ~from_day ~through_day ~limit ~after = Result.bind (timeline_scope_version snapshot Journal_index) (fun version -> - let cursor = match after with + let cursor = + match after with | None -> Ok None | Some position -> (match String.split_on_char '|' position with - | ["tj1"; previous; lower; upper; day; uuid] - when int_of_string_opt lower = Some from_day && int_of_string_opt upper = Some through_day -> - if previous <> version then Error Types.Stale_read_cursor - else (match int_of_string_opt day, Graph.Uuid.of_string uuid with + | [ "tj1"; previous; lower; upper; day; uuid ] + when int_of_string_opt lower = Some from_day + && int_of_string_opt upper = Some through_day -> + if previous <> version + then Error Types.Stale_read_cursor + else ( + match int_of_string_opt day, Graph.Uuid.of_string uuid with | Some day, Ok uuid when day >= from_day && day <= through_day -> - Ok (Some (journal_cursor snapshot ~from_day ~through_day {day; uuid; entity=None})) + Ok + (Some + (journal_cursor + snapshot + ~from_day + ~through_day + { day; uuid; entity = None })) | _ -> Error (Types.Invalid_read_request "invalid timeline date position")) - | _ -> Error (Types.Invalid_read_request "timeline date position belongs to another query")) + | _ -> + Error + (Types.Invalid_read_request "timeline date position belongs to another query")) in Result.bind cursor (fun cursor -> - Result.map (fun (result : Types.journal_list_result) -> - let next = match result.next_cursor with - | None -> None - | Some cursor -> - (match journal_cursor_position snapshot ~from_day ~through_day cursor with - | Ok (Some position) -> Some (Printf.sprintf "tj1|%s|%d|%d|%d|%s" - version from_day through_day position.day (Graph.Uuid.to_string position.uuid)) - | _ -> assert false) - in result.items, next) + Result.map + (fun (result : Types.journal_list_result) -> + let next = + match result.next_cursor with + | None -> None + | Some cursor -> + (match journal_cursor_position snapshot ~from_day ~through_day cursor with + | Ok (Some position) -> + Some + (Printf.sprintf + "tj1|%s|%d|%d|%d|%s" + version + from_day + through_day + position.day + (Graph.Uuid.to_string position.uuid)) + | _ -> assert false) + in + result.items, next) (get_journals snapshot ~from_day ~through_day ~limit ~cursor))) ;; @@ -4549,72 +4867,128 @@ let timeline_member_offset scope version = function | None -> Ok 0 | Some position -> (match String.split_on_char '|' position with - | ["tm1"; key; previous; offset] when key = timeline_scope_key scope -> - if previous <> version then Error Types.Stale_read_cursor - else (match int_of_string_opt offset with + | [ "tm1"; key; previous; offset ] when key = timeline_scope_key scope -> + if previous <> version + then Error Types.Stale_read_cursor + else ( + match int_of_string_opt offset with | Some offset when offset >= 0 && offset <= maximum_cursor_offset -> Ok offset | _ -> Error (Types.Invalid_read_request "invalid timeline member offset")) - | _ -> Error (Types.Invalid_read_request "timeline member position belongs to another query")) + | _ -> + Error + (Types.Invalid_read_request "timeline member position belongs to another query")) ;; let timeline_direct_members snapshot scope ~parent ~limit ~after = with_snapshot_read snapshot (fun snapshot -> - if not (valid_page_limit limit) then Error (Types.Invalid_read_request "limit must be between 1 and 200") - else Result.bind (timeline_version_at snapshot scope) (fun version -> - Result.bind (timeline_member_offset scope version after) (fun offset -> - let database = Option.get snapshot.authoritative_database in - let cache = - { hydration = hydration_cache ~index_idents:false database - ; entities = Hashtbl.create 64; fields = Hashtbl.create 64 - ; authoritative_candidates = Hashtbl.create 64; logical_candidates = Hashtbl.create 64 } - in - let identities = Hashtbl.create 64 in - let examined = ref 0 in - let add uuid = - incr examined; - if !examined > maximum_cursor_offset * 2 then raise Exit; - Hashtbl.replace identities (Graph.Uuid.to_string uuid) uuid - in - let candidates = try - (match tree_entity snapshot cache parent with - | None -> () - | Some entity -> - Datascript.datoms database Datascript.Avet ~a:"block/parent" ~v:(Datascript.Ref entity) () - |> Seq.iter (fun (datom : Datascript.datom) -> - incr examined; - if !examined > maximum_cursor_offset then raise Exit; - Option.iter (fun uuid -> Hashtbl.replace identities (Graph.Uuid.to_string uuid) uuid) - (cached_uuid_of_entity database cache.hydration datom.e))); - Uuid_map.find_opt (Graph.Uuid.to_string parent) (Option.get snapshot.children_effects) - |> Option.value ~default:[] - |> List.iter (fun (record : outbox_record) -> - if record_is_logically_active record then List.iter add record.effect_footprint.block_uuids; - if record_has_active_dependency_shadows record then - List.iter (fun shadow -> if Graph.Uuid.equal shadow.shadow_parent parent then add shadow.shadow_block_uuid) - record.dependency_shadows.shadow_blocks); - if Hashtbl.length identities > maximum_cursor_offset then raise Exit; - Some (Hashtbl.to_seq_values identities |> Rrbvec.of_seq - |> Rrbvec.sort Graph.Uuid.compare - |> Rrbvec.filter_map (fun uuid -> match logical_tree_candidate snapshot cache uuid with - | Some candidate when Graph.Uuid.equal candidate.tree_parent parent -> Some candidate - | _ -> None) - |> Rrbvec.sort (fun left right -> String.compare left.tree_order right.tree_order)) - with Exit -> None - in - match candidates with - | None -> Error Types.Read_limit_exceeded - | Some candidates -> - let total = Rrbvec.length candidates in - if offset > total then Error (Types.Invalid_read_request "timeline offset exceeds scope") - else - let selected = Rrbvec.to_seq candidates |> Seq.drop offset |> Seq.take limit |> List.of_seq in - let context = read_context snapshot in - let items = List.filter_map (fun candidate -> - Option.map (fun block -> Types.{block; revision=logical_block_revision candidate.tree_uuid (Some block)}) - (read_block context candidate.tree_uuid)) selected in - let consumed = offset + List.length selected in - let next = if consumed < total then Some (Printf.sprintf "tm1|%s|%s|%d" (timeline_scope_key scope) version consumed) else None in - Ok (items,next)))) + if not (valid_page_limit limit) + then Error (Types.Invalid_read_request "limit must be between 1 and 200") + else + Result.bind (timeline_version_at snapshot scope) (fun version -> + Result.bind (timeline_member_offset scope version after) (fun offset -> + let database = Option.get snapshot.authoritative_database in + let cache = + { hydration = hydration_cache ~index_idents:false database + ; entities = Hashtbl.create 64 + ; fields = Hashtbl.create 64 + ; authoritative_candidates = Hashtbl.create 64 + ; logical_candidates = Hashtbl.create 64 + } + in + let identities = Hashtbl.create 64 in + let examined = ref 0 in + let add uuid = + incr examined; + if !examined > maximum_cursor_offset * 2 then raise Exit; + Hashtbl.replace identities (Graph.Uuid.to_string uuid) uuid + in + let candidates = + try + (match tree_entity snapshot cache parent with + | None -> () + | Some entity -> + Datascript.datoms + database + Datascript.Avet + ~a:"block/parent" + ~v:(Datascript.Ref entity) + () + |> Seq.iter (fun (datom : Datascript.datom) -> + incr examined; + if !examined > maximum_cursor_offset then raise Exit; + Option.iter + (fun uuid -> + Hashtbl.replace identities (Graph.Uuid.to_string uuid) uuid) + (cached_uuid_of_entity database cache.hydration datom.e))); + Uuid_map.find_opt + (Graph.Uuid.to_string parent) + (Option.get snapshot.children_effects) + |> Option.value ~default:[] + |> List.iter (fun (record : outbox_record) -> + if record_is_logically_active record + then List.iter add record.effect_footprint.block_uuids; + if record_has_active_dependency_shadows record + then + List.iter + (fun shadow -> + if Graph.Uuid.equal shadow.shadow_parent parent + then add shadow.shadow_block_uuid) + record.dependency_shadows.shadow_blocks); + if Hashtbl.length identities > maximum_cursor_offset then raise Exit; + Some + (Hashtbl.to_seq_values identities + |> Rrbvec.of_seq + |> Rrbvec.sort Graph.Uuid.compare + |> Rrbvec.filter_map (fun uuid -> + match logical_tree_candidate snapshot cache uuid with + | Some candidate when Graph.Uuid.equal candidate.tree_parent parent -> + Some candidate + | _ -> None) + |> Rrbvec.sort (fun left right -> + String.compare left.tree_order right.tree_order)) + with + | Exit -> None + in + match candidates with + | None -> Error Types.Read_limit_exceeded + | Some candidates -> + let total = Rrbvec.length candidates in + if offset > total + then Error (Types.Invalid_read_request "timeline offset exceeds scope") + else ( + let selected = + Rrbvec.to_seq candidates + |> Seq.drop offset + |> Seq.take limit + |> List.of_seq + in + let context = read_context snapshot in + let items = + List.filter_map + (fun candidate -> + Option.map + (fun block -> + Types. + { block + ; revision = + logical_block_revision candidate.tree_uuid (Some block) + }) + (read_block context candidate.tree_uuid)) + selected + in + let consumed = offset + List.length selected in + let next = + if consumed < total + then + Some + (Printf.sprintf + "tm1|%s|%s|%d" + (timeline_scope_key scope) + version + consumed) + else None + in + Ok (items, next))))) ;; let get_timeline_children snapshot ~parent ~limit ~after = @@ -4622,8 +4996,14 @@ let get_timeline_children snapshot ~parent ~limit ~after = ;; let get_timeline_roots snapshot ~page ~limit ~after = - Result.map (fun (items,next) -> - List.map (fun (item : Types.child_member) -> Types.{block=item.block; revision=item.revision;depth=0;parent=page}) items,next) + Result.map + (fun (items, next) -> + ( List.map + (fun (item : Types.child_member) -> + Types. + { block = item.block; revision = item.revision; depth = 0; parent = page }) + items + , next )) (timeline_direct_members snapshot (Page_roots page) ~parent:page ~limit ~after) ;; @@ -4633,7 +5013,8 @@ let get_timeline_roots snapshot ~page ~limit ~after = exception Timeline_proof_limit let maintain_timeline_scopes database (before : snapshot) (after : snapshot) = - if database.timeline_proof_available then ( + if database.timeline_proof_available + then ( let old_db = Option.get before.authoritative_database in let new_db = Option.get after.authoritative_database in let visited = Hashtbl.create 64 in @@ -4641,55 +5022,95 @@ let maintain_timeline_scopes database (before : snapshot) (after : snapshot) = let pending = ref [] in let enqueue sequence = pending := sequence :: !pending in let entity_jobs db datoms forced = - Seq.filter_map (fun (datom : Datascript.datom) -> - Option.map (fun uuid -> uuid, forced) (uuid_of_entity db datom.e)) datoms + Seq.filter_map + (fun (datom : Datascript.datom) -> + Option.map (fun uuid -> uuid, forced) (uuid_of_entity db datom.e)) + datoms in let changed_datoms = - if old_db == new_db then Seq.empty - else !(database.authoritative_reports) |> List.to_seq - |> Seq.filter (fun (report : Datascript.tx_report) -> report.db_after.max_tx > old_db.max_tx) - |> Seq.flat_map (fun (report : Datascript.tx_report) -> List.to_seq report.tx_data) + if old_db == new_db + then Seq.empty + else + !(database.authoritative_reports) + |> List.to_seq + |> Seq.filter (fun (report : Datascript.tx_report) -> + report.db_after.max_tx > old_db.max_tx) + |> Seq.flat_map (fun (report : Datascript.tx_report) -> + List.to_seq report.tx_data) in let revoked_identities = Hashtbl.create 16 in - Seq.iter (fun (datom : Datascript.datom) -> - if datom.a = "block/uuid" && not datom.added then - Option.iter (fun uuid -> Hashtbl.replace revoked_identities (Graph.Uuid.to_string uuid) ()) - (uuid_of_entity old_db datom.e)) changed_datoms; + Seq.iter + (fun (datom : Datascript.datom) -> + if datom.a = "block/uuid" && not datom.added + then + Option.iter + (fun uuid -> + Hashtbl.replace revoked_identities (Graph.Uuid.to_string uuid) ()) + (uuid_of_entity old_db datom.e)) + changed_datoms; enqueue (entity_jobs old_db changed_datoms false); enqueue (entity_jobs new_db changed_datoms false); - let index records = List.fold_left (fun map (record : outbox_record) -> - Uuid_map.add (Graph.Uuid.to_string record.mutation_id) record map) Uuid_map.empty records in - let old_records = index (Option.get before.outbox) and new_records = index (Option.get after.outbox) in + let index records = + List.fold_left + (fun map (record : outbox_record) -> + Uuid_map.add (Graph.Uuid.to_string record.mutation_id) record map) + Uuid_map.empty + records + in + let old_records = index (Option.get before.outbox) + and new_records = index (Option.get after.outbox) in let record_jobs (record : outbox_record) = - Seq.map (fun uuid -> uuid, false) - (Seq.append (List.to_seq record.effect_footprint.block_uuids) + Seq.map + (fun uuid -> uuid, false) + (Seq.append + (List.to_seq record.effect_footprint.block_uuids) (List.to_seq record.effect_footprint.page_uuids)) in - Uuid_map.merge (fun _ old current -> - if old <> current then ( - Option.iter (fun record -> enqueue (record_jobs record)) old; - Option.iter (fun record -> enqueue (record_jobs record)) current); - None) old_records new_records |> ignore; - if old_db != new_db then ( + Uuid_map.merge + (fun _ old current -> + if old <> current + then ( + Option.iter (fun record -> enqueue (record_jobs record)) old; + Option.iter (fun record -> enqueue (record_jobs record)) current); + None) + old_records + new_records + |> ignore; + if old_db != new_db + then ( let add_logical_dependents _ (record : outbox_record) = if record_is_logically_active record then enqueue (record_jobs record); - if record_has_active_dependency_shadows record then ( - enqueue (List.to_seq record.dependency_shadows.shadow_blocks - |> Seq.map (fun shadow -> shadow.shadow_block_uuid, false)); - enqueue (List.to_seq record.dependency_shadows.shadow_pages - |> Seq.map (fun shadow -> shadow.shadow_page_uuid, false))) + if record_has_active_dependency_shadows record + then ( + enqueue + (List.to_seq record.dependency_shadows.shadow_blocks + |> Seq.map (fun shadow -> shadow.shadow_block_uuid, false)); + enqueue + (List.to_seq record.dependency_shadows.shadow_pages + |> Seq.map (fun shadow -> shadow.shadow_page_uuid, false))) in Uuid_map.iter add_logical_dependents old_records; Uuid_map.iter add_logical_dependents new_records); - let reference_attributes db = Datascript.schema db |> List.filter_map (fun (attribute,(schema : Datascript.schema_attr)) -> - if schema.value_type = Some Datascript.RefType && attribute <> "block/parent" && attribute <> "block/page" - then Some attribute else None) in - let databases = [old_db, reference_attributes old_db; new_db, reference_attributes new_db] in - let old_context = ref (read_context before) and new_context = ref (read_context after) in + let reference_attributes db = + Datascript.schema db + |> List.filter_map (fun (attribute, (schema : Datascript.schema_attr)) -> + if + schema.value_type = Some Datascript.RefType + && attribute <> "block/parent" + && attribute <> "block/page" + then Some attribute + else None) + in + let databases = + [ old_db, reference_attributes old_db; new_db, reference_attributes new_db ] + in + let old_context = ref (read_context before) + and new_context = ref (read_context after) in let batch_size = 128 in let in_batch = ref 0 in let next_contexts () = - if !in_batch = batch_size then ( + if !in_batch = batch_size + then ( old_context := read_context before; new_context := read_context after; in_batch := 0); @@ -4704,33 +5125,60 @@ let maintain_timeline_scopes database (before : snapshot) (after : snapshot) = let key = timeline_scope_key scope in let bucket = timeline_bucket key in let history = versions.(bucket) in - if Uuid_map.find_opt key history.versions <> Some serial then ( + if Uuid_map.find_opt key history.versions <> Some serial + then ( let present = Uuid_map.mem key history.versions in let history = - if not present && history.count = 64 then - { versions = Uuid_map.empty; count = 0; retirement = serial } - else history in - versions.(bucket) <- - { history with - versions = Uuid_map.add key serial history.versions - ; count = history.count + if present then 0 else 1 }) + if (not present) && history.count = 64 + then { versions = Uuid_map.empty; count = 0; retirement = serial } + else history + in + versions.(bucket) + <- { history with + versions = Uuid_map.add key serial history.versions + ; count = (history.count + if present then 0 else 1) + }) + in + let mark_parent parent = + mark (Parent_children parent); + mark (Page_roots parent) + in + let parent_of = function + | None -> None + | Some (record : Types.block_record) -> Some record.block.parent + in + let tag_titles context tags = + List.filter_map + (fun uuid -> + Option.map + (fun (page : Types.page_record) -> uuid, page.page.title) + (read_page context uuid)) + tags + in + let block_dto context value = + Option.map + (fun (record : Types.block_record) -> + record, tag_titles context record.block.tags) + value in - let mark_parent parent = mark (Parent_children parent); mark (Page_roots parent) in - let parent_of = function None -> None | Some (record : Types.block_record) -> Some record.block.parent in - let tag_titles context tags = List.filter_map (fun uuid -> - Option.map (fun (page : Types.page_record) -> uuid,page.page.title) (read_page context uuid)) tags in - let block_dto context value = Option.map (fun (record : Types.block_record) -> record,tag_titles context record.block.tags) value in let journal_dto context = function | Some (page : Types.page_record) when not page.page.recycled -> - (match page.page.kind with Graph.Journal_page _ -> Some (page,tag_titles context page.page.tags) | _ -> None) - | _ -> None in + (match page.page.kind with + | Graph.Journal_page _ -> Some (page, tag_titles context page.page.tags) + | _ -> None) + | _ -> None + in let rec take () = match !pending with | [] -> None | sequence :: rest -> (match sequence () with - | Seq.Nil -> pending := rest; take () - | Seq.Cons (job, tail) -> pending := tail :: rest; Some job) + | Seq.Nil -> + pending := rest; + take () + | Seq.Cons (job, tail) -> + pending := tail :: rest; + Some job) in let rec drain () = match take () with @@ -4739,61 +5187,134 @@ let maintain_timeline_scopes database (before : snapshot) (after : snapshot) = let key = Graph.Uuid.to_string uuid in let fresh = not (Hashtbl.mem visited key) in let fresh_ancestry = forced && not (Hashtbl.mem ancestry_visited key) in - if fresh || fresh_ancestry then ( + if fresh || fresh_ancestry + then ( let old_context, new_context = next_contexts () in - let old_block = read_block old_context uuid and new_block = read_block new_context uuid in - let old_page = read_page old_context uuid and new_page = read_page new_context uuid in - if fresh then ( + let old_block = read_block old_context uuid + and new_block = read_block new_context uuid in + let old_page = read_page old_context uuid + and new_page = read_page new_context uuid in + if fresh + then ( Hashtbl.add visited key (); let revoked = Hashtbl.mem revoked_identities key in - if revoked || block_dto old_context old_block <> block_dto new_context new_block then - List.iter mark_parent (List.filter_map Fun.id [parent_of old_block;parent_of new_block]); + if + revoked + || block_dto old_context old_block <> block_dto new_context new_block + then + List.iter + mark_parent + (List.filter_map Fun.id [ parent_of old_block; parent_of new_block ]); let old_journal = journal_dto old_context old_page and new_journal = journal_dto new_context new_page in - if old_journal <> new_journal - || (revoked && (Option.is_some old_journal || Option.is_some new_journal)) + if + old_journal <> new_journal + || (revoked && (Option.is_some old_journal || Option.is_some new_journal)) then mark Journal_index); - let rendered_title = function None -> None | Some (page : Types.page_record) -> Some page.page.title in + let rendered_title = function + | None -> None + | Some (page : Types.page_record) -> Some page.page.title + in let title_changed = rendered_title old_page <> rendered_title new_page in - let deleted db = Option.bind (entity_of_uuid db uuid) (fun entity -> one db entity "logseq.property/deleted-at") in - let ancestry block = Option.map (fun (record : Types.block_record) -> record.block.parent,record.block.page) block in - let ancestor_changed = deleted old_db <> deleted new_db || ancestry old_block <> ancestry new_block in - let expand_ancestry = (forced || ancestor_changed) && not (Hashtbl.mem ancestry_visited key) in + let deleted db = + Option.bind (entity_of_uuid db uuid) (fun entity -> + one db entity "logseq.property/deleted-at") + in + let ancestry block = + Option.map + (fun (record : Types.block_record) -> + record.block.parent, record.block.page) + block + in + let ancestor_changed = + deleted old_db <> deleted new_db || ancestry old_block <> ancestry new_block + in + let expand_ancestry = + (forced || ancestor_changed) && not (Hashtbl.mem ancestry_visited key) + in if expand_ancestry then Hashtbl.add ancestry_visited key (); - let entity_payload db = Option.map (entity_datoms db) (entity_of_uuid db uuid) in - let expand_references = fresh && (old_block <> new_block || old_page <> new_page - || (Option.is_none old_block && Option.is_none new_block - && Option.is_none old_page && Option.is_none new_page) - || Hashtbl.mem revoked_identities key || entity_payload old_db <> entity_payload new_db) in - List.iter (fun (db, attributes) -> - match entity_of_uuid db uuid with None -> () | Some entity -> - if expand_references then ( - let inspect_refs attribute = enqueue (entity_jobs db - (Datascript.datoms db Datascript.Avet ~a:attribute ~v:(Datascript.Ref entity) ()) false) in - List.iter inspect_refs attributes; - if title_changed then inspect_refs "block/page"; - Option.iter (fun ident -> enqueue (entity_jobs db - (Datascript.datoms db Datascript.Aevt ~a:ident ()) false)) - (Option.bind (one db entity "db/ident") ident_of_value)); - if expand_ancestry then ( - let children = Datascript.datoms db Datascript.Avet ~a:"block/parent" ~v:(Datascript.Ref entity) () in - (match children () with - | Seq.Nil -> () - | Seq.Cons _ -> - mark_parent uuid; - List.iter mark_parent (List.filter_map Fun.id [parent_of old_block;parent_of new_block])); - enqueue (entity_jobs db children true))) databases); + let entity_payload db = + Option.map (entity_datoms db) (entity_of_uuid db uuid) + in + let expand_references = + fresh + && (old_block <> new_block + || old_page <> new_page + || (Option.is_none old_block + && Option.is_none new_block + && Option.is_none old_page + && Option.is_none new_page) + || Hashtbl.mem revoked_identities key + || entity_payload old_db <> entity_payload new_db) + in + List.iter + (fun (db, attributes) -> + match entity_of_uuid db uuid with + | None -> () + | Some entity -> + if expand_references + then ( + let inspect_refs attribute = + enqueue + (entity_jobs + db + (Datascript.datoms + db + Datascript.Avet + ~a:attribute + ~v:(Datascript.Ref entity) + ()) + false) + in + List.iter inspect_refs attributes; + if title_changed then inspect_refs "block/page"; + Option.iter + (fun ident -> + enqueue + (entity_jobs + db + (Datascript.datoms db Datascript.Aevt ~a:ident ()) + false)) + (Option.bind (one db entity "db/ident") ident_of_value)); + if expand_ancestry + then ( + let children = + Datascript.datoms + db + Datascript.Avet + ~a:"block/parent" + ~v:(Datascript.Ref entity) + () + in + (match children () with + | Seq.Nil -> () + | Seq.Cons _ -> + mark_parent uuid; + List.iter + mark_parent + (List.filter_map + Fun.id + [ parent_of old_block; parent_of new_block ])); + enqueue (entity_jobs db children true))) + databases); drain () in drain (); database.timeline_versions <- versions; database.timeline_serial <- serial - with Timeline_proof_limit -> database.timeline_proof_available <- false) + with + | Timeline_proof_limit -> database.timeline_proof_available <- false) ;; let refresh_queryable_outbox database = - let before = { (snapshot_of_database database) with authoritative_database = Some database.timeline_authoritative } in - let outbox, block_effects, page_effects, children_effects = queryable_outbox_root database.outbox in + let before = + { (snapshot_of_database database) with + authoritative_database = Some database.timeline_authoritative + } + in + let outbox, block_effects, page_effects, children_effects = + queryable_outbox_root database.outbox + in database.queryable_outbox <- outbox; database.queryable_block_effects <- block_effects; database.queryable_page_effects <- page_effects; @@ -4932,10 +5453,11 @@ let write_precondition ~blocks ~pages ~scopes = | None -> Ok { blocks; pages; scopes })) ;; -let mutation_introduced_uuids (record:outbox_record) = match record.mutation with - | Types.Append_journal{root;_}-> +let mutation_introduced_uuids (record : outbox_record) = + match record.mutation with + | Types.Append_journal { root; _ } -> (if record.append_create then record.effect_footprint.page_uuids else []) - @ (append_uuids root) + @ append_uuids root | Types.Insert_blocks { tree; _ } -> Outliner.Tree.uuids tree | Create_journal_page { page; _ } -> [ page ] | Save_block _ @@ -4946,38 +5468,63 @@ let mutation_introduced_uuids (record:outbox_record) = match record.mutation wit | Clear_task_status _ -> [] ;; -let append_identity_present (snapshot:snapshot) uuid = +let append_identity_present (snapshot : snapshot) uuid = let introduced index = - Uuid_map.find_opt(Graph.Uuid.to_string uuid)index |> Option.value ~default:Rrbvec.empty - |> Rrbvec.exists(fun(record:outbox_record)->record_is_logically_active record - && List.exists(Graph.Uuid.equal uuid)(mutation_introduced_uuids record)) in - entity_of_uuid (Option.get snapshot.authoritative_database)uuid<>None - || introduced(Option.get snapshot.block_effects) || introduced(Option.get snapshot.page_effects) + Uuid_map.find_opt (Graph.Uuid.to_string uuid) index + |> Option.value ~default:Rrbvec.empty + |> Rrbvec.exists (fun (record : outbox_record) -> + record_is_logically_active record + && List.exists (Graph.Uuid.equal uuid) (mutation_introduced_uuids record)) + in + entity_of_uuid (Option.get snapshot.authoritative_database) uuid <> None + || introduced (Option.get snapshot.block_effects) + || introduced (Option.get snapshot.page_effects) ;; let authoritative_append_page snapshot day = indexed_journal_candidates snapshot ~from_day:day ~upper_day:day ~locals:Uuid_map.empty - |> Seq.find_map(fun candidate-> - match page_of_database (Option.get snapshot.authoritative_database)candidate.uuid with - | Some page when not page.page.recycled ->Some candidate.uuid - | _->None) + |> Seq.find_map (fun candidate -> + match + page_of_database (Option.get snapshot.authoritative_database) candidate.uuid + with + | Some page when not page.page.recycled -> Some candidate.uuid + | _ -> None) ;; let resolve_append_page snapshot ~candidate ~day = match authoritative_append_page snapshot day with - | Some page->page - | None-> - let chosen=Uuid_map.fold(fun _ effects chosen-> - Rrbvec.fold_left(fun chosen (record:outbox_record)-> - let page=match record.mutation with - | Types.Create_journal_page{page;journal_day;_} when record_is_logically_active record && journal_day=day ->Some page - | Append_journal{journal_day;_} when record_is_logically_active record && journal_day=day -> - Some(List.hd record.effect_footprint.page_uuids) - | _->None in - match page with - | Some page when Option.exists(fun(value:Types.page_record)->not value.page.recycled)(logical_page_at snapshot page)-> - (match chosen with None->Some page|Some old->Some(if Graph.Uuid.compare old page<0 then old else page)) - | _->chosen) chosen effects) (Option.get snapshot.page_effects) None in + | Some page -> page + | None -> + let chosen = + Uuid_map.fold + (fun _ effects chosen -> + Rrbvec.fold_left + (fun chosen (record : outbox_record) -> + let page = + match record.mutation with + | Types.Create_journal_page { page; journal_day; _ } + when record_is_logically_active record && journal_day = day -> + Some page + | Append_journal { journal_day; _ } + when record_is_logically_active record && journal_day = day -> + Some (List.hd record.effect_footprint.page_uuids) + | _ -> None + in + match page with + | Some page + when Option.exists + (fun (value : Types.page_record) -> not value.page.recycled) + (logical_page_at snapshot page) -> + (match chosen with + | None -> Some page + | Some old -> + Some (if Graph.Uuid.compare old page < 0 then old else page)) + | _ -> chosen) + chosen + effects) + (Option.get snapshot.page_effects) + None + in Option.value chosen ~default:candidate ;; @@ -4991,30 +5538,46 @@ let uuid_lookup uuid = let uuid_temp prefix uuid = Datascript.Temp_id (prefix ^ ":" ^ Graph.Uuid.to_string uuid) let tree_orders lower tree = - let rec assign orders order (tree:Types.block_tree) = - let orders=Uuid_map.add(Graph.Uuid.to_string tree.uuid)order orders in - List.fold_left2 assign orders - (Outliner_order.generate_n ~lower:None ~upper:None(List.length tree.children))tree.children in - assign Uuid_map.empty(Outliner_order.generate ~lower ~upper:None)tree + let rec assign orders order (tree : Types.block_tree) = + let orders = Uuid_map.add (Graph.Uuid.to_string tree.uuid) order orders in + List.fold_left2 + assign + orders + (Outliner_order.generate_n ~lower:None ~upper:None (List.length tree.children)) + tree.children + in + assign Uuid_map.empty (Outliner_order.generate ~lower ~upper:None) tree ;; let append_orders lower root = - let rec assign orders order (node:Types.append_node) = - let orders=Uuid_map.add(Graph.Uuid.to_string node.uuid)order orders in - fst(Rrbvec.fold_left(fun(orders,lower)child-> - let order=Outliner_order.generate ~lower ~upper:None in assign orders order child,Some order) - (orders,None)node.children) in - assign Uuid_map.empty(Outliner_order.generate ~lower ~upper:None)root + let rec assign orders order (node : Types.append_node) = + let orders = Uuid_map.add (Graph.Uuid.to_string node.uuid) order orders in + fst + (Rrbvec.fold_left + (fun (orders, lower) child -> + let order = Outliner_order.generate ~lower ~upper:None in + assign orders order child, Some order) + (orders, None) + node.children) + in + assign Uuid_map.empty (Outliner_order.generate ~lower ~upper:None) root ;; let insertion_orders ?append_parent context = function - | Types.Append_journal{page;journal_day;root;_}-> - let lower=match append_parent with Some(_,lower)->lower|None-> - let parent=resolve_append_page context.read_snapshot ~candidate:page ~day:journal_day in - snd(read_siblings context parent) in + | Types.Append_journal { page; journal_day; root; _ } -> + let lower = + match append_parent with + | Some (_, lower) -> lower + | None -> + let parent = + resolve_append_page context.read_snapshot ~candidate:page ~day:journal_day + in + snd (read_siblings context parent) + in append_orders lower root - | Types.Insert_blocks{parent;tree;_}->tree_orders(snd(read_siblings context parent))tree - | _->Uuid_map.empty + | Types.Insert_blocks { parent; tree; _ } -> + tree_orders (snd (read_siblings context parent)) tree + | _ -> Uuid_map.empty ;; module Asset_descriptor = Logseq_db_types.Asset_descriptor @@ -5145,14 +5708,14 @@ let published_asset_matches descriptor version = ;; let rec local_operations - ~append_create - ~fingerprint:_ - ~intent_time_ms - ~next_tx - ~orders - ~(effect_footprint : effect_footprint) - ~delete_artifacts - mutation + ~append_create + ~fingerprint:_ + ~intent_time_ms + ~next_tx + ~orders + ~(effect_footprint : effect_footprint) + ~delete_artifacts + mutation = let now = Int64.to_int intent_time_ms in let operations = @@ -5174,34 +5737,69 @@ let rec local_operations ; Add (uuid_lookup block, "block/updated-at", Int64 (Int64.of_int now)) ; Add (uuid_lookup block, "block/tx-id", Int64 (Int64.of_int next_tx)) ] - | Types.Append_journal{mutation_id;page=_;title;journal_day;root}-> - let page=List.hd effect_footprint.page_uuids in - let run mutation=local_operations ~append_create:false ~fingerprint:"" - ~intent_time_ms ~next_tx ~orders ~effect_footprint ~delete_artifacts:None mutation in - let creation=if append_create then run(Types.Create_journal_page{mutation_id;page;title;journal_day}) else [] in - let rec add_tree parent_ref (node:Types.append_node) = - let node_ref=uuid_temp "overlay-insert" node.uuid in - let fields=Datascript.[ - Add(node_ref,"block/uuid",Uuid(Graph.Uuid.to_string node.uuid));Add(node_ref,"block/title",String node.title); - Add(node_ref,"block/parent",Ref_to parent_ref);Add(node_ref,"block/page",Ref_to(uuid_lookup page)); - Add(node_ref,"block/order",String(assigned_order orders node.uuid)); - Add(node_ref,"block/created-at",Int64 intent_time_ms);Add(node_ref,"block/updated-at",Int64 intent_time_ms); - Add(node_ref,"block/tx-id",Int64(Int64.of_int next_tx))] in - Seq.append(List.to_seq fields)(Rrbvec.to_seq node.children |> Seq.flat_map(add_tree node_ref)) in - let insertion=add_tree(uuid_lookup page)root |> List.of_seq in - let statuses=Overlay_planner.append_nodes root |> Rrbvec.to_seq - |> Seq.flat_map(fun(node:Types.append_node)->match node.task_status with - |None->Seq.empty - |Some status->run(Types.Set_task_status{mutation_id;block=node.uuid;status=Types.Status_uuid status}) |> List.to_seq) - |> List.of_seq in - let operations=creation@insertion@statuses in - if not append_create then operations else - let page_ref=uuid_temp "overlay-page" page in - List.map(function - | Datascript.Add(entity,attribute,Ref_to reference) when reference=uuid_lookup page -> - Datascript.Add(entity,attribute,Ref_to page_ref) - | operation->operation)operations - | Types.Save_block { block; title; _ } -> + | Types.Append_journal { mutation_id; page = _; title; journal_day; root } -> + let page = List.hd effect_footprint.page_uuids in + let run mutation = + local_operations + ~append_create:false + ~fingerprint:"" + ~intent_time_ms + ~next_tx + ~orders + ~effect_footprint + ~delete_artifacts:None + mutation + in + let creation = + if append_create + then run (Types.Create_journal_page { mutation_id; page; title; journal_day }) + else [] + in + let rec add_tree parent_ref (node : Types.append_node) = + let node_ref = uuid_temp "overlay-insert" node.uuid in + let fields = + Datascript. + [ Add (node_ref, "block/uuid", Uuid (Graph.Uuid.to_string node.uuid)) + ; Add (node_ref, "block/title", String node.title) + ; Add (node_ref, "block/parent", Ref_to parent_ref) + ; Add (node_ref, "block/page", Ref_to (uuid_lookup page)) + ; Add (node_ref, "block/order", String (assigned_order orders node.uuid)) + ; Add (node_ref, "block/created-at", Int64 intent_time_ms) + ; Add (node_ref, "block/updated-at", Int64 intent_time_ms) + ; Add (node_ref, "block/tx-id", Int64 (Int64.of_int next_tx)) + ] + in + Seq.append + (List.to_seq fields) + (Rrbvec.to_seq node.children |> Seq.flat_map (add_tree node_ref)) + in + let insertion = add_tree (uuid_lookup page) root |> List.of_seq in + let statuses = + Overlay_planner.append_nodes root + |> Rrbvec.to_seq + |> Seq.flat_map (fun (node : Types.append_node) -> + match node.task_status with + | None -> Seq.empty + | Some status -> + run + (Types.Set_task_status + { mutation_id; block = node.uuid; status = Types.Status_uuid status }) + |> List.to_seq) + |> List.of_seq + in + let operations = creation @ insertion @ statuses in + if not append_create + then operations + else ( + let page_ref = uuid_temp "overlay-page" page in + List.map + (function + | Datascript.Add (entity, attribute, Ref_to reference) + when reference = uuid_lookup page -> + Datascript.Add (entity, attribute, Ref_to page_ref) + | operation -> operation) + operations) + | Types.Save_block { block; title; _ } -> let entity = uuid_lookup block in [ Datascript.Add (entity, "block/title", String title) ; Add (entity, "block/updated-at", Int64 (Int64.of_int now)) @@ -5398,10 +5996,16 @@ let dependency_shadows_are_canonical ~effect_footprint mutation shadows = | Types.Save_block { block; _ } | Set_task_status { block; _ } | Clear_task_status { block; _ } -> add_block [] [] block - | Append_journal{journal_day;_} -> - if List.length effect_footprint.page_uuids=1 && List.for_all(fun shadow-> - not shadow.shadow_recycled && shadow.shadow_page_kind=Graph.Journal_page{journal_day})shadows.shadow_pages - then Some([],effect_footprint.page_uuids) else None + | Append_journal { journal_day; _ } -> + if + List.length effect_footprint.page_uuids = 1 + && List.for_all + (fun shadow -> + (not shadow.shadow_recycled) + && shadow.shadow_page_kind = Graph.Journal_page { journal_day }) + shadows.shadow_pages + then Some ([], effect_footprint.page_uuids) + else None | Insert_blocks { parent; _ } -> add_parent [] [] parent | Delete_blocks _ | Create_journal_page _ -> Some ([], []) in @@ -5473,7 +6077,10 @@ let outbox_record_is_canonical (record : outbox_record) = | Submitted _ | Accepted_pending_authoritative _ | Stale_rejected_pending_authoritative _ -> - dependency_shadows_are_canonical ~effect_footprint:record.effect_footprint record.mutation record.dependency_shadows + dependency_shadows_are_canonical + ~effect_footprint:record.effect_footprint + record.mutation + record.dependency_shadows in Result.is_ok (Sync_tx_codec.protected_values record.normalized_transaction) && String.equal record.fingerprint (fingerprint record.mutation) @@ -5950,21 +6557,42 @@ let delete_artifacts database context ~now root = ;; let planned_effect database context ~now = function - | Types.Append_journal{page;root;journal_day;_}-> - let resolved=resolve_append_page context.read_snapshot ~candidate:page ~day:journal_day in - let nodes=Overlay_planner.append_nodes root in - let invalid=Rrbvec.find_map(fun(node:Types.append_node)-> - if append_identity_present context.read_snapshot node.uuid then Some "An append block UUID is already present." - else match node.task_status with - |Some status when task_status_at context.read_snapshot status=None ->Some "Append status is missing or recycled." - |_->None) nodes in - (match invalid,read_page context resolved with - | Some message,_->Error(Types.Invalid_local_mutation message) - | _,Some page when page.page.recycled || page.page.kind<>Graph.Journal_page{journal_day}-> - Error(Types.Invalid_local_mutation "Append candidate belongs to another page.") - | _,None when append_identity_present context.read_snapshot resolved ->Error(Types.Invalid_local_mutation "Append candidate UUID is already occupied.") - | _->Ok({block_uuids=append_uuids root;page_uuids=[resolved];structure_interests= - [Types.Children_interest resolved;Page_tree_interest resolved;Journal_index_interest]@append_interests root},None)) + | Types.Append_journal { page; root; journal_day; _ } -> + let resolved = + resolve_append_page context.read_snapshot ~candidate:page ~day:journal_day + in + let nodes = Overlay_planner.append_nodes root in + let invalid = + Rrbvec.find_map + (fun (node : Types.append_node) -> + if append_identity_present context.read_snapshot node.uuid + then Some "An append block UUID is already present." + else ( + match node.task_status with + | Some status when task_status_at context.read_snapshot status = None -> + Some "Append status is missing or recycled." + | _ -> None)) + nodes + in + (match invalid, read_page context resolved with + | Some message, _ -> Error (Types.Invalid_local_mutation message) + | _, Some page + when page.page.recycled || page.page.kind <> Graph.Journal_page { journal_day } -> + Error (Types.Invalid_local_mutation "Append candidate belongs to another page.") + | _, None when append_identity_present context.read_snapshot resolved -> + Error (Types.Invalid_local_mutation "Append candidate UUID is already occupied.") + | _ -> + Ok + ( { block_uuids = append_uuids root + ; page_uuids = [ resolved ] + ; structure_interests = + [ Types.Children_interest resolved + ; Page_tree_interest resolved + ; Journal_index_interest + ] + @ append_interests root + } + , None )) | Types.Set_asset_reference { block; previous; asset; _ } -> (match read_block context block, asset_descriptor_at context.read_snapshot asset with | Some holder, Ok (Some _) when asset_reference_matches holder previous -> @@ -6121,7 +6749,11 @@ let dependency_shadows_for_snapshot ?append_page snapshot mutation = | Types.Save_block { block; _ } | Set_task_status { block; _ } | Clear_task_status { block; _ } -> add_block block - | Append_journal{page;journal_day;_}->add_page(match append_page with Some page->page|None->resolve_append_page snapshot ~candidate:page ~day:journal_day) + | Append_journal { page; journal_day; _ } -> + add_page + (match append_page with + | Some page -> page + | None -> resolve_append_page snapshot ~candidate:page ~day:journal_day) | Insert_blocks { parent; _ } -> add_block parent | Delete_blocks _ | Create_journal_page _ -> ()); { shadow_blocks = @@ -6137,10 +6769,16 @@ let dependency_shadows_for_snapshot ?append_page snapshot mutation = } ;; -let dependency_shadows_for database (record:outbox_record) = - let append_page=match record.mutation,record.effect_footprint.page_uuids with - | Types.Append_journal _,[page]->Some page|_->None in - dependency_shadows_for_snapshot ?append_page (snapshot_of_database database) record.mutation +let dependency_shadows_for database (record : outbox_record) = + let append_page = + match record.mutation, record.effect_footprint.page_uuids with + | Types.Append_journal _, [ page ] -> Some page + | _ -> None + in + dependency_shadows_for_snapshot + ?append_page + (snapshot_of_database database) + record.mutation ;; let local_candidate_record @@ -6161,9 +6799,11 @@ let local_candidate_record + 1 in let planned_tx = (authoritative_database database).Datascript.max_tx + 1 in - let append_create=match mutation,effect_footprint.page_uuids with - | Types.Append_journal _,[page]->read_page context page=None - | _->false in + let append_create = + match mutation, effect_footprint.page_uuids with + | Types.Append_journal _, [ page ] -> read_page context page = None + | _ -> false + in { mutation_id ; append_create ; fingerprint @@ -6232,7 +6872,8 @@ let bounded_logical_summary database ~block_uuids ~page_uuids ~structure_interes ;; let logical_change_for_effects database ~before ~after effects = - let before_context=read_context before and after_context=read_context after in + let before_context = read_context before + and after_context = read_context after in let block_candidates, page_candidates, structure_interests = List.fold_left (fun (blocks, pages, interests) footprint -> @@ -6252,7 +6893,8 @@ let logical_change_for_effects database ~before ~after effects = let page_uuids = page_candidates |> List.sort_uniq Graph.Uuid.compare - |> List.filter (fun uuid -> read_page before_context uuid <> read_page after_context uuid) + |> List.filter (fun uuid -> + read_page before_context uuid <> read_page after_context uuid) in if block_uuids = [] && page_uuids = [] then Types.No_logical_change @@ -6293,177 +6935,201 @@ let open_owned ~sw dependencies inspection ~graph_name ownership = | Ok (outbox, legacy_revision) -> let sync_revision = Option.value durable_revision ~default:legacy_revision in let durable_outbox = durable_outbox_rows outbox in - (match migrate_outbox path ~revision:sync_revision ~expected_records:encoded durable_outbox with + (match + migrate_outbox + path + ~revision:sync_revision + ~expected_records:encoded + durable_outbox + with | Error message -> Error (Types.Corrupt_outbox message) | Ok () -> - (match Storage.open_database path with - | Error error -> Error (Types.Restore_failed (Storage.diagnostic_error ~operation:"attachGraph" error)) - | Ok connection -> - let callbacks = Storage.connection_callbacks connection in - let transferred = ref false in - let installed_listener = ref None in - Fun.protect - ~finally:(fun () -> - if not !transferred - then ( - Option.iter - (fun (connection, listener) -> - Datascript.unlisten connection listener) - !installed_listener; - ignore (Storage.close callbacks))) - (fun () -> - match Storage.restore_database connection with - | Error error -> - Error (Types.Restore_failed (Storage.diagnostic_error ~operation:"attachGraph" error)) - | Ok authoritative_database -> - let authoritative_connection = - Datascript.Conn.from_db - { empty_db = Datascript.empty_db - ; init_db = Datascript.init_db - ; store = (fun ?storage:_ database -> database) - } - authoritative_database - in - let authoritative_reports = ref [] in - let authoritative_listener = - Datascript.listen - authoritative_connection - "logseq-overlay-authoritative" - (fun report -> - authoritative_reports := report :: !authoritative_reports) - in - installed_listener - := Some (authoritative_connection, authoritative_listener); - let storage_session = - Logseq_db_storage.Storage_session.create - ~tail: - (Datascript.Storage.restore_tail_groups - (Storage.datascript_storage connection)) - ~callbacks - in - let release_slot = ref None in - let checkpoint_metadata = - Storage.sync_metadata connection |> Result.get_ok - in - let checkpoint = - match inspection.presence with - | Types.Available { checkpoint; _ } -> checkpoint - | Absent _ -> Types.Server_cursor.zero - in - let ( queryable_outbox - , queryable_block_effects - , queryable_page_effects - , queryable_children_effects ) - = - queryable_outbox_root outbox - in - let database = - { lock = Eio.Mutex.create () + (match Storage.open_database path with + | Error error -> + Error + (Types.Restore_failed + (Storage.diagnostic_error ~operation:"attachGraph" error)) + | Ok connection -> + let callbacks = Storage.connection_callbacks connection in + let transferred = ref false in + let installed_listener = ref None in + Fun.protect + ~finally:(fun () -> + if not !transferred + then ( + Option.iter + (fun (connection, listener) -> + Datascript.unlisten connection listener) + !installed_listener; + ignore (Storage.close callbacks))) + (fun () -> + match Storage.restore_database connection with + | Error error -> + Error + (Types.Restore_failed + (Storage.diagnostic_error ~operation:"attachGraph" error)) + | Ok authoritative_database -> + let restored_entities = ref 0 in + let authoritative_checksum = + Authoritative_checksum.initialize + ~check_current:(fun () -> + restored_entities := (!restored_entities + 1) mod 256; + if !restored_entities = 0 then Eio.Fiber.yield ()) + ~e2ee:(Authoritative_checksum.graph_e2ee authoritative_database) + authoritative_database + in + let authoritative_connection = + Datascript.Conn.from_db + { empty_db = Datascript.empty_db + ; init_db = Datascript.init_db + ; store = (fun ?storage:_ database -> database) + } + authoritative_database + in + let authoritative_reports = ref [] in + let authoritative_listener = + Datascript.listen + authoritative_connection + "logseq-overlay-authoritative" + (fun report -> + authoritative_reports := report :: !authoritative_reports) + in + installed_listener + := Some (authoritative_connection, authoritative_listener); + let storage_session = + Logseq_db_storage.Storage_session.create + ~tail: + (Datascript.Storage.restore_tail_groups + (Storage.datascript_storage connection)) + ~callbacks + in + let release_slot = ref None in + let checkpoint_metadata = + Storage.sync_metadata connection |> Result.get_ok + in + let checkpoint = + match inspection.presence with + | Types.Available { checkpoint; _ } -> checkpoint + | Absent _ -> Types.Server_cursor.zero + in + let ( queryable_outbox + , queryable_block_effects + , queryable_page_effects + , queryable_children_effects ) + = + queryable_outbox_root outbox + in + let database = + { lock = Eio.Mutex.create () ; task_catalog_cache = Atomic.make None ; notification_lock = Eio.Mutex.create () - ; sw - ; dispatch_stream = - Eio.Stream.create dependencies.limits.dispatcher_capacity - ; dispatch_slots = - Eio.Semaphore.make dependencies.limits.dispatcher_capacity - ; dependencies - ; database_path = path - ; graph_uuid = inspection.location.graph_id - ; graph_name - ; schema = Graph.{ major = 65; minor = 33 } - ; admission_facts = - [ Graph.Remote_flag_true - ; Synced_graph_identity inspection.location.graph_id - ; Lossless_codec - ; Ownership_verified - ] - ; storage_session - ; ownership - ; release_slot - ; closed = false - ; generation = current_generation - ; projection = 0 - ; timeline_versions = empty_timeline_history () - ; timeline_serial = 0 - ; timeline_proof_available = true - ; timeline_authoritative = authoritative_database - ; authoritative_connection - ; authoritative_listener - ; authoritative_reports - ; receipts = [] - ; outbox - ; queryable_outbox - ; queryable_block_effects - ; queryable_page_effects - ; queryable_children_effects - ; sync_revision - ; durable_sync_revision = sync_revision - ; durable_outbox - ; checkpoint - ; checkpoint_metadata - ; subscriptions = [] - ; snapshots = [] - } - in - release_slot := Some database; - Eio.Switch.on_release sw (fun () -> - match !release_slot with - | None -> () - | Some database -> ignore (close database)); - Eio.Fiber.fork_daemon ~sw (fun () -> - let rec dispatch () = - match Eio.Stream.take database.dispatch_stream with - | Stop_dispatch -> () - | Dispatch_change (change, queued) -> - Eio.Semaphore.release database.dispatch_slots; - let subscriptions = - Eio.Mutex.use_rw ~protect:true database.lock (fun () -> - List.filter - (fun (subscription : subscription) -> - subscription.lifecycle = Active) - database.subscriptions) - in - let after_revision = - match change with - | Types.Exact { after_revision; _ } - | Projection_resync_required { after_revision; _ } -> - after_revision - in - List.iter - (fun (subscription : subscription) -> - if - Eio.Stream.length subscription.events - >= dependencies.limits.dispatcher_capacity - then ( - while - Option.is_some - (Eio.Stream.take_nonblocking subscription.events) - do - () - done; - Eio.Stream.add - subscription.events - (Some - (Types.Projection_resync_required - { generation = database.generation - ; after_revision - ; reason = Dispatcher_retention_exceeded - }))) - else Eio.Stream.add subscription.events (Some change)) - subscriptions; - Eio.Promise.resolve queued (); - dispatch () - in - dispatch (); - `Stop_daemon); - transferred := true; - installed_listener := None; - Ok database))))) + ; sw + ; dispatch_stream = + Eio.Stream.create dependencies.limits.dispatcher_capacity + ; dispatch_slots = + Eio.Semaphore.make dependencies.limits.dispatcher_capacity + ; dependencies + ; database_path = path + ; graph_uuid = inspection.location.graph_id + ; graph_name + ; schema = Graph.{ major = 65; minor = 33 } + ; admission_facts = + [ Graph.Remote_flag_true + ; Synced_graph_identity inspection.location.graph_id + ; Lossless_codec + ; Ownership_verified + ] + ; storage_session + ; ownership + ; release_slot + ; closed = false + ; generation = current_generation + ; projection = 0 + ; timeline_versions = empty_timeline_history () + ; timeline_serial = 0 + ; timeline_proof_available = true + ; timeline_authoritative = authoritative_database + ; authoritative_checksum + ; authoritative_connection + ; authoritative_listener + ; authoritative_reports + ; receipts = [] + ; outbox + ; queryable_outbox + ; queryable_block_effects + ; queryable_page_effects + ; queryable_children_effects + ; sync_changed = Eio.Condition.create () + ; sync_revision + ; durable_sync_revision = sync_revision + ; durable_outbox + ; checkpoint + ; checkpoint_metadata + ; subscriptions = [] + ; snapshots = [] + } + in + release_slot := Some database; + Eio.Switch.on_release sw (fun () -> + match !release_slot with + | None -> () + | Some database -> ignore (close database)); + Eio.Fiber.fork_daemon ~sw (fun () -> + let rec dispatch () = + match Eio.Stream.take database.dispatch_stream with + | Stop_dispatch -> () + | Dispatch_change (change, queued) -> + Eio.Semaphore.release database.dispatch_slots; + let subscriptions = + Eio.Mutex.use_rw ~protect:true database.lock (fun () -> + List.filter + (fun (subscription : subscription) -> + subscription.lifecycle = Active) + database.subscriptions) + in + let after_revision = + match change with + | Types.Exact { after_revision; _ } + | Projection_resync_required { after_revision; _ } -> + after_revision + in + List.iter + (fun (subscription : subscription) -> + if + Eio.Stream.length subscription.events + >= dependencies.limits.dispatcher_capacity + then ( + while + Option.is_some + (Eio.Stream.take_nonblocking subscription.events) + do + () + done; + Eio.Stream.add + subscription.events + (Some + (Types.Projection_resync_required + { generation = database.generation + ; after_revision + ; reason = Dispatcher_retention_exceeded + }))) + else Eio.Stream.add subscription.events (Some change)) + subscriptions; + Eio.Promise.resolve queued (); + dispatch () + in + dispatch (); + `Stop_daemon); + transferred := true; + installed_listener := None; + Ok database))))) ;; let open_ ~sw dependencies inspection ~graph_name = - (try collect_retired_mirrors inspection.location with _ -> - report_warning "journal: phase=cleanup operation=collectRetiredMirrors code=collectionPending\n"); + (try collect_retired_mirrors inspection.location with + | _ -> + report_warning + "journal: phase=cleanup operation=collectRetiredMirrors code=collectionPending\n"); if String.trim graph_name = "" then Error Types.Invalid_graph_name else ( @@ -6613,6 +7279,7 @@ let persist_outbox ?(terminal_batches = []) ?intent_commit database = | Ok () -> database.durable_outbox <- durable_outbox; database.durable_sync_revision <- database.sync_revision; + Eio.Condition.broadcast database.sync_changed; Ok () | Error message -> rollback message)))))) ;; @@ -7125,6 +7792,26 @@ let inspect_sync database = ~submissions:(List.map descriptor database.outbox))) ;; +let await_sync_change database ~after = + Eio.Condition.loop_no_mutex database.sync_changed (fun () -> + match inspect_sync database with + | Error _ as error -> Some error + | Ok view when not (Types.sync_token_equal after (Types.sync_view_token view)) -> + Some (Ok view) + | Ok _ -> None) +;; + +let check_checkpoint_checksum database ~through ~expected = + Eio.Mutex.use_rw ~protect:true database.lock (fun () -> + if database.closed + then Error Types.Sync_database_closed + else if not (Types.Server_cursor.equal database.checkpoint through) + then Ok None + else ( + let digest = Authoritative_checksum.to_string database.authoritative_checksum in + Ok (Some (String.equal (checksum_value expected) digest)))) +;; + let find_outbox database id = List.find_opt (fun (record : outbox_record) -> Graph.Uuid.equal record.mutation_id id) @@ -7158,11 +7845,13 @@ let records_for_ids records ids = ids ;; -let mutation_required_uuids (record:outbox_record) = match record.mutation with - | Types.Append_journal{root;_}-> +let mutation_required_uuids (record : outbox_record) = + match record.mutation with + | Types.Append_journal { root; _ } -> (if record.append_create then [] else record.effect_footprint.page_uuids) @ (Overlay_planner.append_nodes root - |> Rrbvec.filter_map(fun(node:Types.append_node)->node.task_status) |> Rrbvec.to_list) + |> Rrbvec.filter_map (fun (node : Types.append_node) -> node.task_status) + |> Rrbvec.to_list) | Types.Set_asset_reference { block; asset; _ } -> [ block; asset ] | Types.Publish_asset { block; _ } | Types.Save_block { block; _ } @@ -7201,8 +7890,7 @@ let ineligible_submission_dependency database selected = (fun (candidate : outbox_record) -> candidate == owner) selected) in - if - List.exists has_unselected_queued_owner (mutation_required_uuids record) + if List.exists has_unselected_queued_owner (mutation_required_uuids record) then Some record.mutation_id else None) selected @@ -7301,17 +7989,30 @@ let authoritative_import_matches ;; let authoritative_satisfies_mutation_in database = function - | Types.Append_journal{root;journal_day;_}-> - let rec matches parent (node:Types.append_node)=match block_of_database database node.uuid with - | Some value->(match parent with None->true|Some parent->value.block.parent=parent) - && value.block.title=node.title && value.task_status=Option.map(fun id->Types.Status_uuid id)node.task_status - && Rrbvec.for_all(matches(Some node.uuid))node.children - | None->false in - matches None root && (match block_of_database database root.uuid with - | Some value->(match page_of_database database value.block.page with - |Some page->not page.page.recycled && page.page.kind=Graph.Journal_page{journal_day} - && value.block.parent=page.page.uuid|None->false) - |None->false) + | Types.Append_journal { root; journal_day; _ } -> + let rec matches parent (node : Types.append_node) = + match block_of_database database node.uuid with + | Some value -> + (match parent with + | None -> true + | Some parent -> value.block.parent = parent) + && value.block.title = node.title + && value.task_status + = Option.map (fun id -> Types.Status_uuid id) node.task_status + && Rrbvec.for_all (matches (Some node.uuid)) node.children + | None -> false + in + matches None root + && + (match block_of_database database root.uuid with + | Some value -> + (match page_of_database database value.block.page with + | Some page -> + (not page.page.recycled) + && page.page.kind = Graph.Journal_page { journal_day } + && value.block.parent = page.page.uuid + | None -> false) + | None -> false) | Types.Set_asset_reference { block; asset; _ } -> block_of_database database block |> option_satisfies (fun holder -> asset_reference_matches holder (Some asset)) @@ -7709,7 +8410,7 @@ let make_batch preparation encrypted = let expected = local_operations ~append_create:record.append_create - ~fingerprint:record.fingerprint + ~fingerprint:record.fingerprint ~intent_time_ms:record.intent_time_ms ~next_tx:record.planned_tx ~orders:(assigned_orders record) @@ -7724,7 +8425,7 @@ let make_batch preparation encrypted = let operations = local_operations ~append_create:record.append_create - ~fingerprint:record.fingerprint + ~fingerprint:record.fingerprint ~intent_time_ms:record.intent_time_ms ~next_tx:record.planned_tx ~orders:(assigned_orders record) @@ -7994,6 +8695,8 @@ let begin_authoritative database ~expected batch = in Ok ( { authoritative_owner = database + ; authoritative_base = authoritative_database database + ; authoritative_digest = database.authoritative_checksum ; authoritative_revision = database.sync_revision ; authoritative_batch = batch ; authoritative_wires = wires @@ -8018,7 +8721,7 @@ let split_prefix count values = let synchronized_record_operations (record : outbox_record) = local_operations ~append_create:record.append_create - ~fingerprint:record.fingerprint + ~fingerprint:record.fingerprint ~intent_time_ms:record.intent_time_ms ~next_tx:record.planned_tx ~orders:(assigned_orders record) @@ -8036,7 +8739,8 @@ let authoritative_exception_kind = function ;; let simulation_failure_context exn = - let cause = match exn with + let cause = + match exn with | Invalid_argument message -> let prefixes = [ "Nothing found for entity id [", "lookup-ref-missing" @@ -8050,19 +8754,25 @@ let simulation_failure_context exn = ; "instant attribute requires instant value:", "value-type-mismatch" ; "tuple attribute element type mismatch:", "value-type-mismatch" ; "tuple attribute requires tuple value:", "value-type-mismatch" - ] in + ] + in (match message with | "unique constraint" -> "unique-constraint" | "Cannot store nil as a value" -> "nil-value" | "cannot modify tuple attributes directly" -> "tuple-write" - | _ -> Option.value ~default:"unknown" - (List.find_map (fun (prefix,cause) -> - if String.starts_with ~prefix message then Some cause else None) prefixes)) - | _ -> "unknown" in + | _ -> + Option.value + ~default:"unknown" + (List.find_map + (fun (prefix, cause) -> + if String.starts_with ~prefix message then Some cause else None) + prefixes)) + | _ -> "unknown" + in [ "exception", authoritative_exception_kind exn; "simulationCause", cause ] ;; -let expected_record_transaction_data database record = +let expected_record_transaction database record = try Ok (Datascript.transact @@ -8070,23 +8780,77 @@ let expected_record_transaction_data database record = database (synchronized_record_operations record) : Datascript.tx_report) - .tx_data with | exn -> Error (simulation_failure_context exn) ;; -let equal_transaction_data left right = List.sort compare left = List.sort compare right +let expected_record_transaction_data database record = + Result.map + (fun (report : Datascript.tx_report) -> report.tx_data) + (expected_record_transaction database record) +;; -let transaction_matches_record_at_cursor record cursor root_before transaction_data = +let canonical_transaction_data root_before root_after transaction_data = + let identities = Hashtbl.create 16 in + let identity entity = + match Hashtbl.find_opt identities entity with + | Some reference -> reference + | None -> + let find database = + match uuid_of_entity database entity with + | Some uuid -> + Some + (Datascript.Lookup_ref + ("block/uuid", Datascript.Uuid (Graph.Uuid.to_string uuid))) + | None -> + (match one database entity "db/ident" with + | Some (Datascript.Keyword ident) -> + Some (Datascript.Lookup_ref ("db/ident", Datascript.Keyword ident)) + | _ -> None) + in + let reference = + match find root_after with + | Some reference -> reference + | None -> Option.value (find root_before) ~default:(Datascript.Entity_id entity) + in + Hashtbl.add identities entity reference; + reference + in + let rec value = function + | Datascript.Ref entity -> Datascript.Ref_to (identity entity) + | List values -> Datascript.List (List.map value values) + | Vector values -> Datascript.Vector (List.map value values) + | Set values -> Datascript.Set (List.map value values) + | Map entries -> + Datascript.Map (List.map (fun (key, item) -> value key, value item) entries) + | Tuple values -> Datascript.Tuple (List.map (Option.map value) values) + | value -> value + in + List.map + (fun (datom : Datascript.datom) -> + identity datom.e, datom.a, value datom.v, datom.tx, datom.added) + transaction_data + |> List.sort compare +;; + +let transaction_matches_record_at_cursor + record + cursor + root_before + root_after + transaction_data + = match expected_origin_cursor record with | Error _ -> false | Ok expected_cursor -> if not (Types.Server_cursor.equal cursor expected_cursor) then false else ( - match expected_record_transaction_data root_before record with + match expected_record_transaction root_before record with | Error _ -> false - | Ok expected -> equal_transaction_data expected transaction_data) + | Ok expected -> + canonical_transaction_data root_before expected.db_after expected.tx_data + = canonical_transaction_data root_before root_after transaction_data) ;; let integrity_record_context (record : outbox_record) = @@ -8138,11 +8902,22 @@ let integrity_batch_source owner batch source = |> Authoritative_diagnostic.encode ;; -let validate_active_transaction_origin owner cursor root_before transaction_data = +let validate_active_transaction_origin + owner + cursor + root_before + root_after + transaction_data + = let matches = List.filter (fun record -> - transaction_matches_record_at_cursor record cursor root_before transaction_data) + transaction_matches_record_at_cursor + record + cursor + root_before + root_after + transaction_data) owner.outbox in match matches with @@ -8162,9 +8937,18 @@ let validate_active_transaction_origin owner cursor root_before transaction_data (* Only schema identifiers in this list may leave the database in evidence. Values, arbitrary attribute names and wire payloads must never be emitted. *) let integrity_evidence_attributes = - [ "block/uuid"; "block/title"; "block/name"; "block/journal-day" - ; "block/created-at"; "block/updated-at"; "block/tags"; "block/parent" - ; "block/page"; "block/order"; "block/refs"; "block/tx-id" + [ "block/uuid" + ; "block/title" + ; "block/name" + ; "block/journal-day" + ; "block/created-at" + ; "block/updated-at" + ; "block/tags" + ; "block/parent" + ; "block/page" + ; "block/order" + ; "block/refs" + ; "block/tx-id" ] ;; @@ -8188,17 +8972,28 @@ let integrity_evidence_value_type = function | TxRef -> "transaction-reference" ;; -let report_integrity_evidence owner batch record cursor root_before expected actual operations = +let report_integrity_evidence + owner + batch + record + cursor + root_before + expected + actual + operations + = (* This best-effort observation must not affect validation or publication. *) try let target, day = match record.mutation with | Types.Create_journal_page { page; journal_day; _ } -> page, Some journal_day - | Append_journal {root;journal_day;_}->root.uuid,Some journal_day + | Append_journal { root; journal_day; _ } -> root.uuid, Some journal_day | Insert_blocks { tree; _ } -> tree.uuid, None | Delete_blocks { root; _ } -> root, None - | Save_block { block; _ } | Set_task_status { block; _ } - | Clear_task_status { block; _ } | Publish_asset { block; _ } + | Save_block { block; _ } + | Set_task_status { block; _ } + | Clear_task_status { block; _ } + | Publish_asset { block; _ } | Set_asset_reference { block; _ } -> block, None in let entity = entity_of_uuid root_before target in @@ -8210,10 +9005,13 @@ let report_integrity_evidence owner batch record cursor root_before expected act | None -> [] | Some entity -> Datascript.datoms root_before Datascript.Eavt ~e:entity ~a:attribute () - |> Seq.take 4 |> List.of_seq + |> Seq.take 4 + |> List.of_seq in let values = List.map (fun (d : Datascript.datom) -> d.v) sample in - let types = List.map integrity_evidence_value_type values |> List.sort_uniq compare in + let types = + List.map integrity_evidence_value_type values |> List.sort_uniq compare + in `Assoc [ "attribute", `String attribute ; "datoms", `Int (min 3 (List.length sample)) @@ -8229,21 +9027,28 @@ let report_integrity_evidence owner batch record cursor root_before expected act let attribute = if List.mem datom.a integrity_evidence_attributes then datom.a else "other" in - let added, retracted = Option.value ~default:(0, 0) (Hashtbl.find_opt delta attribute) in - Hashtbl.replace delta attribute - (added + (if datom.added then 1 else 0), retracted + (if datom.added then 0 else 1))) + let added, retracted = + Option.value ~default:(0, 0) (Hashtbl.find_opt delta attribute) + in + Hashtbl.replace + delta + attribute + ( (added + if datom.added then 1 else 0) + , retracted + if datom.added then 0 else 1 )) actual; let pairs = Hashtbl.create 4 in List.iter (fun operation -> let item = match operation with - | Datascript.Add (entity, attribute, value) -> Some (entity, attribute, value, true) - | Retract (entity, attribute, Some value) -> Some (entity, attribute, value, false) + | Datascript.Add (entity, attribute, value) -> + Some (entity, attribute, value, true) + | Retract (entity, attribute, Some value) -> + Some (entity, attribute, value, false) | _ -> None in match item with - | Some (entity, ("block/title" | "block/name" as attribute), value, added) -> + | Some (entity, (("block/title" | "block/name") as attribute), value, added) -> let key = entity, attribute, value in let a, r = Option.value ~default:(false, false) (Hashtbl.find_opt pairs key) in Hashtbl.replace pairs key (a || added, r || not added) @@ -8253,7 +9058,8 @@ let report_integrity_evidence owner batch record cursor root_before expected act Hashtbl.fold (fun (_, a, _) (added, retracted) count -> if String.equal a attribute && added && retracted then count + 1 else count) - pairs 0 + pairs + 0 in let str s = `String s in let evidence = @@ -8263,20 +9069,37 @@ let report_integrity_evidence owner batch record cursor root_before expected act ; "target", str (Graph.Uuid.to_string target) ; "graph", str (Graph.Uuid.to_string owner.graph_uuid) ; "generation", str (Types.Generation.to_string owner.generation) - ; "projection", str (Types.Projection_revision.to_string (projection_revision owner.projection)) - ; "checkpoint", str (Int64.to_string (Types.Server_cursor.to_int64 owner.checkpoint)) - ; "through", str (Int64.to_string (Types.Server_cursor.to_int64 (Types.authoritative_batch_through batch))) + ; ( "projection" + , str + (Types.Projection_revision.to_string + (projection_revision owner.projection)) ) + ; ( "checkpoint" + , str (Int64.to_string (Types.Server_cursor.to_int64 owner.checkpoint)) ) + ; ( "through" + , str + (Int64.to_string + (Types.Server_cursor.to_int64 (Types.authoritative_batch_through batch))) + ) ; "expectedCursor", str (Int64.to_string (Types.Server_cursor.to_int64 cursor)) ; "expectedDatoms", `Int (List.length expected) ; "actualDatoms", `Int (List.length actual) ; "beforeFields", `List before_fields - ; "actualDelta", `List - (Hashtbl.to_seq delta |> List.of_seq |> List.sort compare - |> List.map (fun (attribute, (added, retracted)) -> - `Assoc [ "attribute", str attribute; "added", `Int added; "retracted", `Int retracted ])) - ; "protectedPairs", `Assoc - [ "block/title", `Int (pair_count "block/title") - ; "block/name", `Int (pair_count "block/name") ] + ; ( "actualDelta" + , `List + (Hashtbl.to_seq delta + |> List.of_seq + |> List.sort compare + |> List.map (fun (attribute, (added, retracted)) -> + `Assoc + [ "attribute", str attribute + ; "added", `Int added + ; "retracted", `Int retracted + ])) ) + ; ( "protectedPairs" + , `Assoc + [ "block/title", `Int (pair_count "block/title") + ; "block/name", `Int (pair_count "block/name") + ] ) ] @ List.map (fun (k, v) -> k, str v) (integrity_record_context record) @ Option.fold ~none:[] ~some:(fun day -> [ "day", `Int day ]) day) @@ -8284,10 +9107,17 @@ let report_integrity_evidence owner batch record cursor root_before expected act let encoded = Yojson.Safe.to_string evidence in if String.length encoded <= 8192 then report_warning ("journal-integrity: " ^ encoded ^ "\n") - with _ -> report_warning "journal-integrity: {\"reason\":\"evidence-unavailable\"}\n" + with + | _ -> report_warning "journal-integrity: {\"reason\":\"evidence-unavailable\"}\n" ;; -let validate_active_transaction_origins owner batch transaction_data roots_after decoded_transactions = +let validate_active_transaction_origins + owner + batch + transaction_data + roots_after + decoded_transactions + = let cursors = Types.authoritative_batch_transactions batch |> List.map Types.authoritative_transaction_cursor @@ -8314,7 +9144,7 @@ let validate_active_transaction_origins owner batch transaction_data roots_after cursor_rest data_rest root_rest - ((cursor, data, previous) :: reversed) + ((cursor, data, previous, root_after) :: reversed) | [], [], [] -> List.rev reversed | _ -> invalid_arg "authoritative transaction origin context mismatch" in @@ -8323,7 +9153,7 @@ let validate_active_transaction_origins owner batch transaction_data roots_after in let validated = List.fold_left - (fun result (cursor, data, root_before) -> + (fun result (cursor, data, root_before, root_after) -> Result.bind result (fun reversed -> Result.map (fun evidence -> @@ -8331,7 +9161,12 @@ let validate_active_transaction_origins owner batch transaction_data roots_after ~none:reversed ~some:(fun value -> value :: reversed) evidence) - (validate_active_transaction_origin owner cursor root_before data))) + (validate_active_transaction_origin + owner + cursor + root_before + root_after + data))) (Ok []) contexts in @@ -8377,17 +9212,19 @@ let validate_active_transaction_origins owner batch transaction_data roots_after else ( let context = List.find_opt - (fun (cursor, _, _) -> Types.Server_cursor.equal cursor expected) + (fun (cursor, _, _, _) -> + Types.Server_cursor.equal cursor expected) contexts in let reason, counts = match context with | None -> "expected-cursor-missing", [] - | Some (_, actual, root_before) -> + | Some (_, actual, root_before, _) -> (match expected_record_transaction_data root_before record with | Error cause -> ( "expected-simulation-failed" - , ("actualDatoms", string_of_int (List.length actual)) :: cause ) + , ("actualDatoms", string_of_int (List.length actual)) + :: cause ) | Ok expected_data -> let rec operations_at_cursor cursors transactions = match cursors, transactions with @@ -8397,11 +9234,18 @@ let validate_active_transaction_origins owner batch transaction_data roots_after else operations_at_cursor rest more | _ -> [] in - report_integrity_evidence owner batch record expected - root_before expected_data actual + report_integrity_evidence + owner + batch + record + expected + root_before + expected_data + actual (operations_at_cursor cursors decoded_transactions); ( "payload-mismatch" - , [ "expectedDatoms", string_of_int (List.length expected_data) + , [ ( "expectedDatoms" + , string_of_int (List.length expected_data) ) ; "actualDatoms", string_of_int (List.length actual) ] )) in @@ -8851,34 +9695,40 @@ let prepare_authoritative_candidate preparation ~decrypted = |> Authoritative_diagnostic.encode) | wire :: rest -> let current = - Result.bind (Sync_tx_codec.protected_values ~allow_empty:true wire) (fun protected -> - Result.bind - (split_prefix (List.length protected) remaining - |> Result.map_error (fun _ -> - Authoritative_diagnostic.create - ~stage:Decode - ~reason:Decrypted_value_count - () - |> Authoritative_diagnostic.encode)) - (fun (plain, remaining) -> - Result.bind - (Sync_tx_codec.decode ~allow_empty:true ~db:database ~decrypted_values:plain wire) - (fun operations -> - try - let report = - Datascript.with_tx - ~tx_meta:[ "skip-store?", Datascript.Bool true ] - database - operations - in - Ok (report, operations, remaining) - with - | exn -> - Error - (authoritative_application_error - ~transaction:index - operations - exn)))) + Result.bind + (Sync_tx_codec.protected_values ~allow_empty:true wire) + (fun protected -> + Result.bind + (split_prefix (List.length protected) remaining + |> Result.map_error (fun _ -> + Authoritative_diagnostic.create + ~stage:Decode + ~reason:Decrypted_value_count + () + |> Authoritative_diagnostic.encode)) + (fun (plain, remaining) -> + Result.bind + (Sync_tx_codec.decode + ~allow_empty:true + ~db:database + ~decrypted_values:plain + wire) + (fun operations -> + try + let report = + Datascript.with_tx + ~tx_meta:[ "skip-store?", Datascript.Bool true ] + database + operations + in + Ok (report, operations, remaining) + with + | exn -> + Error + (authoritative_application_error + ~transaction:index + operations + exn)))) |> Result.map_error (fun source -> match Authoritative_diagnostic.decode source with | Some detail -> @@ -8899,16 +9749,21 @@ let prepare_authoritative_candidate preparation ~decrypted = (report.db_after :: roots_after) remaining rest - with exn -> - Error (Authoritative_diagnostic.create ~stage:Decode ~reason:Unknown - ?transaction:(if rest=[] then None else Some(index+1)) - ~exception_kind:(authoritative_exception_kind exn) () - |> Authoritative_diagnostic.encode)) + with + | exn -> + Error + (Authoritative_diagnostic.create + ~stage:Decode + ~reason:Unknown + ?transaction:(if rest = [] then None else Some (index + 1)) + ~exception_kind:(authoritative_exception_kind exn) + () + |> Authoritative_diagnostic.encode)) in match decode 1 - (authoritative_database preparation.authoritative_owner) + preparation.authoritative_base [] [] [] @@ -8917,152 +9772,200 @@ let prepare_authoritative_candidate preparation ~decrypted = with | Error message -> Error (Types.Authoritative_decode_failed message) | Ok (transactions, expected_tx_data, roots_after, database) -> - (match - validate_active_transaction_origins - preparation.authoritative_owner - preparation.authoritative_batch - expected_tx_data - roots_after - transactions - with - | Error message -> Error (Types.Authoritative_integrity_failure - (integrity_batch_source preparation.authoritative_owner preparation.authoritative_batch message)) - | Ok validated_origins -> - preparation.authoritative_state <- Authoritative_consumed; - Option.iter - (fun request -> request.unprotection_consumed <- true) - preparation.authoritative_unprotection; - let unresolved_submitted_deletes = - List.filter_map - (fun (record : outbox_record) -> - match record.transport_state, record.mutation with - | Types.Submitted batch_id, Types.Delete_blocks _ -> - Some (record, batch_id) - | ( Submitted _ - , ( Save_block _ - | Insert_blocks _ - | Append_journal _ - | Create_journal_page _ - | Set_task_status _ - | Clear_task_status _ - | Publish_asset _ - | Set_asset_reference _ ) ) -> None - | ( ( Queued - | Accepted_pending_authoritative _ - | Stale_rejected_pending_authoritative _ - | Blocked ) - , _ ) -> None) - preparation.authoritative_owner.outbox - in - (match - classify_stale_deletes - preparation.authoritative_owner - preparation.authoritative_batch - transactions - roots_after - with - | Error message -> Error (Types.Authoritative_integrity_failure - (integrity_batch_source preparation.authoritative_owner preparation.authoritative_batch message)) - | Ok (stale_updates, stale_remote_won, stale_no_change, stale_blocked) -> - let rec transaction_contexts previous transactions transaction_data roots = - match transactions, transaction_data, roots with - | ( transaction :: transaction_rest - , data :: data_rest - , root_after :: root_rest ) -> - ( Types.authoritative_transaction_cursor transaction - , data - , previous - , root_after ) - :: transaction_contexts root_after transaction_rest data_rest root_rest - | [], [], [] -> [] - | _ -> invalid_arg "authoritative transaction context mismatch" - in - let contexts = - transaction_contexts - (authoritative_database preparation.authoritative_owner) - (Types.authoritative_batch_transactions preparation.authoritative_batch) - expected_tx_data - roots_after - in - let classify_submitted_delete (record, batch_id) = - match record.submission_t_before with - | None -> + let digest = + Authoritative_checksum.advance + ~maximum: + preparation.authoritative_owner.dependencies.limits.wire_batch_max_bytes + ~before:preparation.authoritative_base + ~after:database + ~changes:expected_tx_data + preparation.authoritative_digest + |> Result.map_error (fun message -> + Types.Authoritative_integrity_failure message) + in + Result.bind digest (fun digest -> + match Types.authoritative_batch_checksum preparation.authoritative_batch with + | Some expected + when not + (String.equal + (checksum_value expected) + (Authoritative_checksum.to_string digest)) -> + Error Types.Authoritative_checksum_mismatch + | None | Some _ -> + (match + validate_active_transaction_origins + preparation.authoritative_owner + preparation.authoritative_batch + expected_tx_data + roots_after + transactions + with + | Error message -> + Error + (Types.Authoritative_integrity_failure + (integrity_batch_source + preparation.authoritative_owner + preparation.authoritative_batch + message)) + | Ok validated_origins -> + preparation.authoritative_state <- Authoritative_consumed; + Option.iter + (fun request -> request.unprotection_consumed <- true) + preparation.authoritative_unprotection; + let unresolved_submitted_deletes = + List.filter_map + (fun (record : outbox_record) -> + match record.transport_state, record.mutation with + | Types.Submitted batch_id, Types.Delete_blocks _ -> + Some (record, batch_id) + | ( Submitted _ + , ( Save_block _ + | Insert_blocks _ + | Append_journal _ + | Create_journal_page _ + | Set_task_status _ + | Clear_task_status _ + | Publish_asset _ + | Set_asset_reference _ ) ) -> None + | ( ( Queued + | Accepted_pending_authoritative _ + | Stale_rejected_pending_authoritative _ + | Blocked ) + , _ ) -> None) + preparation.authoritative_owner.outbox + in + (match + classify_stale_deletes + preparation.authoritative_owner + preparation.authoritative_batch + transactions + roots_after + with + | Error message -> Error (Types.Authoritative_integrity_failure - (integrity_source ~reason:"submitted-delete-missing-t-before" ~record () - |> integrity_batch_source preparation.authoritative_owner preparation.authoritative_batch)) - | Some t_before -> - (match - List.find_opt - (fun (cursor, _, _, _) -> - Types.Server_cursor.compare cursor t_before > 0) - contexts - with - | None -> Ok (`Deferred batch_id) - | Some (cursor, transaction_data, root_before, root_after) -> - if - transaction_matches_record_at_cursor - record - cursor - root_before - transaction_data - then Ok (`Deferred batch_id) - else ( - let conflicts = - delete_conflict_kinds_between root_before root_after record - in - if conflicts = [] - then Ok (`Deferred batch_id) - else - Ok - (`Remote_won (record.mutation_id, batch_id, cursor, conflicts)))) - in - let classified = - List.fold_left - (fun result submitted -> - Result.bind result (fun outcomes -> - Result.map - (fun outcome -> outcome :: outcomes) - (classify_submitted_delete submitted))) - (Ok []) - unresolved_submitted_deletes - in - Result.bind classified (fun classified -> - match - List.find_map - (function - | `Deferred batch_id -> Some batch_id - | `Remote_won _ -> None) - (List.rev classified) - with - | Some batch_id -> - preparation.authoritative_state <- Authoritative_deferred_state; - Ok (`Deferred (Types.Await_submission_outcome batch_id)) - | None -> - let submitted_remote_won = - List.filter_map - (function - | `Remote_won outcome -> Some outcome - | `Deferred _ -> None) - (List.rev classified) + (integrity_batch_source + preparation.authoritative_owner + preparation.authoritative_batch + message)) + | Ok (stale_updates, stale_remote_won, stale_no_change, stale_blocked) -> + let rec transaction_contexts + previous + transactions + transaction_data + roots + = + match transactions, transaction_data, roots with + | ( transaction :: transaction_rest + , data :: data_rest + , root_after :: root_rest ) -> + ( Types.authoritative_transaction_cursor transaction + , data + , previous + , root_after ) + :: transaction_contexts + root_after + transaction_rest + data_rest + root_rest + | [], [], [] -> [] + | _ -> invalid_arg "authoritative transaction context mismatch" in - let candidate = - { authoritative_preparation = preparation - ; authoritative_decrypted_input = decrypted - ; authoritative_transactions = transactions - ; authoritative_expected_tx_data = expected_tx_data - ; authoritative_roots_after = roots_after - ; validated_origins - ; submitted_remote_won - ; stale_updates - ; stale_remote_won - ; stale_no_change - ; stale_blocked - ; authoritative_db_after = database - } + let contexts = + transaction_contexts + (authoritative_database preparation.authoritative_owner) + (Types.authoritative_batch_transactions + preparation.authoritative_batch) + expected_tx_data + roots_after in - preparation.authoritative_state <- Authoritative_ready candidate; - Ok (`Candidate candidate))))) + let classify_submitted_delete (record, batch_id) = + match record.submission_t_before with + | None -> + Error + (Types.Authoritative_integrity_failure + (integrity_source + ~reason:"submitted-delete-missing-t-before" + ~record + () + |> integrity_batch_source + preparation.authoritative_owner + preparation.authoritative_batch)) + | Some t_before -> + (match + List.find_opt + (fun (cursor, _, _, _) -> + Types.Server_cursor.compare cursor t_before > 0) + contexts + with + | None -> Ok (`Deferred batch_id) + | Some (cursor, transaction_data, root_before, root_after) -> + if + transaction_matches_record_at_cursor + record + cursor + root_before + root_after + transaction_data + then Ok (`Deferred batch_id) + else ( + let conflicts = + delete_conflict_kinds_between root_before root_after record + in + if conflicts = [] + then Ok (`Deferred batch_id) + else + Ok + (`Remote_won + (record.mutation_id, batch_id, cursor, conflicts)))) + in + let classified = + List.fold_left + (fun result submitted -> + Result.bind result (fun outcomes -> + Result.map + (fun outcome -> outcome :: outcomes) + (classify_submitted_delete submitted))) + (Ok []) + unresolved_submitted_deletes + in + Result.bind classified (fun classified -> + match + List.find_map + (function + | `Deferred batch_id -> Some batch_id + | `Remote_won _ -> None) + (List.rev classified) + with + | Some batch_id -> + preparation.authoritative_state <- Authoritative_deferred_state; + Ok (`Deferred (Types.Await_submission_outcome batch_id)) + | None -> + let submitted_remote_won = + List.filter_map + (function + | `Remote_won outcome -> Some outcome + | `Deferred _ -> None) + (List.rev classified) + in + let candidate = + { authoritative_preparation = preparation + ; authoritative_decrypted_input = decrypted + ; authoritative_transactions = transactions + ; authoritative_expected_tx_data = expected_tx_data + ; authoritative_roots_after = roots_after + ; validated_origins + ; submitted_remote_won + ; stale_updates + ; stale_remote_won + ; stale_no_change + ; stale_blocked + ; authoritative_digest_after = digest + ; authoritative_db_after = database + } + in + preparation.authoritative_state <- Authoritative_ready candidate; + Ok (`Candidate candidate)))))) ;; let rec logical_tree_is_present snapshot ~parent (tree : Types.block_tree) = @@ -9075,17 +9978,30 @@ let rec logical_tree_is_present snapshot ~parent (tree : Types.block_tree) = ;; let logical_snapshot_satisfies_mutation snapshot = function - | Types.Append_journal{root;journal_day;_}-> - let rec matches parent (node:Types.append_node)=match logical_block_at snapshot node.uuid with - | Some value->(match parent with None->true|Some parent->value.block.parent=parent) - && value.block.title=node.title && value.task_status=Option.map(fun id->Types.Status_uuid id)node.task_status - && Rrbvec.for_all(matches(Some node.uuid))node.children - | None->false in - matches None root && (match logical_block_at snapshot root.uuid with - | Some value->(match logical_page_at snapshot value.block.page with - |Some page->not page.page.recycled && page.page.kind=Graph.Journal_page{journal_day} - && value.block.parent=page.page.uuid|None->false) - |None->false) + | Types.Append_journal { root; journal_day; _ } -> + let rec matches parent (node : Types.append_node) = + match logical_block_at snapshot node.uuid with + | Some value -> + (match parent with + | None -> true + | Some parent -> value.block.parent = parent) + && value.block.title = node.title + && value.task_status + = Option.map (fun id -> Types.Status_uuid id) node.task_status + && Rrbvec.for_all (matches (Some node.uuid)) node.children + | None -> false + in + matches None root + && + (match logical_block_at snapshot root.uuid with + | Some value -> + (match logical_page_at snapshot value.block.page with + | Some page -> + (not page.page.recycled) + && page.page.kind = Graph.Journal_page { journal_day } + && value.block.parent = page.page.uuid + | None -> false) + | None -> false) | Types.Set_asset_reference { block; asset; _ } -> logical_block_at snapshot block |> option_satisfies (fun holder -> asset_reference_matches holder (Some asset)) @@ -9133,10 +10049,18 @@ let logical_snapshot_satisfies_mutation snapshot = function ;; let ordinary_effect_footprint ?append_owner snapshot = function - | Types.Append_journal{page;root;journal_day;_}-> - let page=match append_owner with Some(page,_)->page|None->resolve_append_page snapshot ~candidate:page ~day:journal_day in - {block_uuids=append_uuids root;page_uuids=[page];structure_interests= - [Types.Children_interest page;Page_tree_interest page;Journal_index_interest]@append_interests root} + | Types.Append_journal { page; root; journal_day; _ } -> + let page = + match append_owner with + | Some (page, _) -> page + | None -> resolve_append_page snapshot ~candidate:page ~day:journal_day + in + { block_uuids = append_uuids root + ; page_uuids = [ page ] + ; structure_interests = + [ Types.Children_interest page; Page_tree_interest page; Journal_index_interest ] + @ append_interests root + } | Types.Set_asset_reference { block; _ } | Types.Publish_asset { block; _ } | Types.Save_block { block; _ } @@ -9164,7 +10088,6 @@ let ordinary_effect_footprint ?append_owner snapshot = function ;; let introduced_uuids = mutation_introduced_uuids -;; let unavailable_dependency unavailable uuid = List.find_map @@ -9176,16 +10099,31 @@ let unavailable_dependency unavailable uuid = let invalid_replan_dependency ?append_owner snapshot mutation = let missing uuid present = if present then None else Some uuid in match mutation with - | Types.Append_journal{page;root;journal_day;_}-> - let page,value=match append_owner with Some value->value|None-> - let page=resolve_append_page snapshot ~candidate:page ~day:journal_day in page,logical_page_at snapshot page in - let conflict=match value with - |Some value->value.page.recycled || value.page.kind<>Graph.Journal_page{journal_day} - |None->append_identity_present snapshot page in - if conflict then Some page else - Overlay_planner.append_nodes root |> Rrbvec.find_map(fun(node:Types.append_node)-> - if append_identity_present snapshot node.uuid then Some node.uuid - else match node.task_status with Some status when task_status_at snapshot status=None->Some status|_->None) + | Types.Append_journal { page; root; journal_day; _ } -> + let page, value = + match append_owner with + | Some value -> value + | None -> + let page = resolve_append_page snapshot ~candidate:page ~day:journal_day in + page, logical_page_at snapshot page + in + let conflict = + match value with + | Some value -> + value.page.recycled || value.page.kind <> Graph.Journal_page { journal_day } + | None -> append_identity_present snapshot page + in + if conflict + then Some page + else + Overlay_planner.append_nodes root + |> Rrbvec.find_map (fun (node : Types.append_node) -> + if append_identity_present snapshot node.uuid + then Some node.uuid + else ( + match node.task_status with + | Some status when task_status_at snapshot status = None -> Some status + | _ -> None)) | Types.Set_asset_reference { block; previous; asset; _ } -> (match logical_block_at snapshot block, asset_descriptor_at snapshot asset with | Some holder, Ok (Some _) when asset_reference_matches holder previous -> None @@ -9268,55 +10206,112 @@ let replan_queued_ordinary database authoritative_database records = (* An in-flight operation retains its dependency facts independently of sequence position. Sample those shadow tails once, before earlier queued Appends are replanned; a late ACK cannot make a visible sibling vanish. *) - let shadow_tails=Hashtbl.create 16 and shadow_presence=Hashtbl.create 16 in - let shadow_cache=hydration_cache ~index_idents:false authoritative_database in - let maximum_order lower order=Some(Option.fold ~none:order - ~some:(fun old->if String.compare old order>=0 then old else order)lower) in - List.iter(fun(record:outbox_record)-> - if record_has_active_dependency_shadows record && record_is_logically_active record then - List.iter(fun shadow-> - let key=Graph.Uuid.to_string shadow.shadow_block_uuid in - let present=match Hashtbl.find_opt shadow_presence key with Some present->present|None-> - let present=authoritative_block_present authoritative_database shadow_cache shadow.shadow_block_uuid in - Hashtbl.add shadow_presence key present;present in - if not present then ( - let parent=Graph.Uuid.to_string shadow.shadow_parent in - let lower=Hashtbl.find_opt shadow_tails parent |> Option.join in - Hashtbl.replace shadow_tails parent(maximum_order lower shadow.shadow_order))) - record.dependency_shadows.shadow_blocks)records; - let local_days=Hashtbl.create 16 and remote_days=Hashtbl.create 16 - and day_owners=Hashtbl.create 16 and parent_tails=Hashtbl.create 16 in + let shadow_tails = Hashtbl.create 16 + and shadow_presence = Hashtbl.create 16 in + let shadow_cache = hydration_cache ~index_idents:false authoritative_database in + let maximum_order lower order = + Some + (Option.fold + ~none:order + ~some:(fun old -> if String.compare old order >= 0 then old else order) + lower) + in + List.iter + (fun (record : outbox_record) -> + if record_has_active_dependency_shadows record && record_is_logically_active record + then + List.iter + (fun shadow -> + let key = Graph.Uuid.to_string shadow.shadow_block_uuid in + let present = + match Hashtbl.find_opt shadow_presence key with + | Some present -> present + | None -> + let present = + authoritative_block_present + authoritative_database + shadow_cache + shadow.shadow_block_uuid + in + Hashtbl.add shadow_presence key present; + present + in + if not present + then ( + let parent = Graph.Uuid.to_string shadow.shadow_parent in + let lower = Hashtbl.find_opt shadow_tails parent |> Option.join in + Hashtbl.replace + shadow_tails + parent + (maximum_order lower shadow.shadow_order))) + record.dependency_shadows.shadow_blocks) + records; + let local_days = Hashtbl.create 16 + and remote_days = Hashtbl.create 16 + and day_owners = Hashtbl.create 16 + and parent_tails = Hashtbl.create 16 in let append_owner snapshot page day = match Hashtbl.find_opt day_owners day with - | Some owner->owner - | None-> - let remote=match Hashtbl.find_opt remote_days day with Some value->value|None-> - let value=authoritative_append_page snapshot day in Hashtbl.add remote_days day value;value in - let existing=match remote with Some _->remote|None->Hashtbl.find_opt local_days day in - let resolved=Option.value existing ~default:page in - let owner=resolved,logical_page_at snapshot resolved in - if existing<>None then Hashtbl.replace day_owners day owner; - owner in - let record_append_indexes (record:outbox_record) = - let day_page=match record.mutation with - | Types.Create_journal_page{page;journal_day;_}->Some(journal_day,page) - | Append_journal{journal_day;_}->Some(journal_day,List.hd record.effect_footprint.page_uuids) - | _->None in - Option.iter(fun(day,page)-> - match Hashtbl.find_opt local_days day with - | Some old when Graph.Uuid.compare old page<=0->() - | _->Hashtbl.replace local_days day page;Hashtbl.remove day_owners day)day_page; - let inserted=match record.mutation with - | Types.Insert_blocks{parent;tree;_}->Some(parent,tree.uuid) - | Append_journal{root;_}->Some(List.hd record.effect_footprint.page_uuids,root.uuid) - | _->None in - Option.iter(fun(parent,root)-> - let key=Graph.Uuid.to_string parent in - match Hashtbl.find_opt parent_tails key with - | None->() - | Some lower->let order=assigned_order(assigned_orders record)root in - Hashtbl.replace parent_tails key (Some(Option.fold ~none:order - ~some:(fun old->if String.compare old order>=0 then old else order)lower))) inserted in + | Some owner -> owner + | None -> + let remote = + match Hashtbl.find_opt remote_days day with + | Some value -> value + | None -> + let value = authoritative_append_page snapshot day in + Hashtbl.add remote_days day value; + value + in + let existing = + match remote with + | Some _ -> remote + | None -> Hashtbl.find_opt local_days day + in + let resolved = Option.value existing ~default:page in + let owner = resolved, logical_page_at snapshot resolved in + if existing <> None then Hashtbl.replace day_owners day owner; + owner + in + let record_append_indexes (record : outbox_record) = + let day_page = + match record.mutation with + | Types.Create_journal_page { page; journal_day; _ } -> Some (journal_day, page) + | Append_journal { journal_day; _ } -> + Some (journal_day, List.hd record.effect_footprint.page_uuids) + | _ -> None + in + Option.iter + (fun (day, page) -> + match Hashtbl.find_opt local_days day with + | Some old when Graph.Uuid.compare old page <= 0 -> () + | _ -> + Hashtbl.replace local_days day page; + Hashtbl.remove day_owners day) + day_page; + let inserted = + match record.mutation with + | Types.Insert_blocks { parent; tree; _ } -> Some (parent, tree.uuid) + | Append_journal { root; _ } -> + Some (List.hd record.effect_footprint.page_uuids, root.uuid) + | _ -> None + in + Option.iter + (fun (parent, root) -> + let key = Graph.Uuid.to_string parent in + match Hashtbl.find_opt parent_tails key with + | None -> () + | Some lower -> + let order = assigned_order (assigned_orders record) root in + Hashtbl.replace + parent_tails + key + (Some + (Option.fold + ~none:order + ~some:(fun old -> if String.compare old order >= 0 then old else order) + lower))) + inserted + in let add_record records block_effects page_effects children_effects record = if not (record_is_logically_active record) then records, block_effects, page_effects, children_effects @@ -9326,19 +10321,46 @@ let replan_queued_ordinary database authoritative_database records = add_indexed_record block_effects (fun value -> value.block_uuids) record in let page_effects = - let pages=page_projection_effects record @ - (if record_has_active_dependency_shadows record then - List.map(fun shadow->shadow.shadow_page_uuid)record.dependency_shadows.shadow_pages else []) in - add_indexed_record page_effects (fun _->List.sort_uniq Graph.Uuid.compare pages)record + let pages = + page_projection_effects record + @ + if record_has_active_dependency_shadows record + then + List.map + (fun shadow -> shadow.shadow_page_uuid) + record.dependency_shadows.shadow_pages + else [] + in + add_indexed_record + page_effects + (fun _ -> List.sort_uniq Graph.Uuid.compare pages) + record + in + let parents = + List.filter_map + (function + | Types.Children_interest parent -> Some parent + | _ -> None) + record.effect_footprint.structure_interests + @ + if record_has_active_dependency_shadows record + then + List.map + (fun shadow -> shadow.shadow_parent) + record.dependency_shadows.shadow_blocks + else [] + in + let children_effects = + List.fold_left + (fun index parent -> + let key = Graph.Uuid.to_string parent in + Uuid_map.add + key + (record :: Option.value (Uuid_map.find_opt key index) ~default:[]) + index) + children_effects + (List.sort_uniq Graph.Uuid.compare parents) in - let parents=List.filter_map(function Types.Children_interest parent->Some parent|_->None) - record.effect_footprint.structure_interests - @ (if record_has_active_dependency_shadows record then - List.map(fun shadow->shadow.shadow_parent)record.dependency_shadows.shadow_blocks else []) in - let children_effects=List.fold_left(fun index parent-> - let key=Graph.Uuid.to_string parent in - Uuid_map.add key (record::Option.value(Uuid_map.find_opt key index) ~default:[]) index) - children_effects (List.sort_uniq Graph.Uuid.compare parents) in record :: records, block_effects, page_effects, children_effects) in let rec loop @@ -9388,9 +10410,12 @@ let replan_queued_ordinary database authoritative_database records = ; drain_waiters = [] } in - let append_owner=match mutation with - | Types.Append_journal{page;journal_day;_}->Some(append_owner snapshot page journal_day) - | _->None in + let append_owner = + match mutation with + | Types.Append_journal { page; journal_day; _ } -> + Some (append_owner snapshot page journal_day) + | _ -> None + in if logical_snapshot_satisfies_mutation snapshot mutation then loop @@ -9429,28 +10454,45 @@ let replan_queued_ordinary database authoritative_database records = (record :: blocked) rest | None -> - let effect_footprint = ordinary_effect_footprint ?append_owner snapshot mutation in + let effect_footprint = + ordinary_effect_footprint ?append_owner snapshot mutation + in let planned_tx = authoritative_database.Datascript.max_tx + 1 in - let append_create=match mutation,effect_footprint.page_uuids with - | Types.Append_journal _,[_]->Option.bind append_owner snd=None - | _->false in - let orders=match mutation,append_owner with - | Types.Append_journal _,Some(parent,_)-> - let key=Graph.Uuid.to_string parent in - let lower=match Hashtbl.find_opt parent_tails key with Some lower->lower|None-> - let lower=snd(read_siblings(read_context snapshot)parent) in - let lower=match Hashtbl.find_opt shadow_tails key |> Option.join with - |None->lower|Some shadow->maximum_order lower shadow in - Hashtbl.replace parent_tails key lower;lower in - insertion_orders ~append_parent:(parent,lower)(read_context snapshot)mutation - | _->assigned_orders record in + let append_create = + match mutation, effect_footprint.page_uuids with + | Types.Append_journal _, [ _ ] -> Option.bind append_owner snd = None + | _ -> false + in + let orders = + match mutation, append_owner with + | Types.Append_journal _, Some (parent, _) -> + let key = Graph.Uuid.to_string parent in + let lower = + match Hashtbl.find_opt parent_tails key with + | Some lower -> lower + | None -> + let lower = snd (read_siblings (read_context snapshot) parent) in + let lower = + match Hashtbl.find_opt shadow_tails key |> Option.join with + | None -> lower + | Some shadow -> maximum_order lower shadow + in + Hashtbl.replace parent_tails key lower; + lower + in + insertion_orders + ~append_parent:(parent, lower) + (read_context snapshot) + mutation + | _ -> assigned_orders record + in let record = { record with normalized_transaction = normalized_transaction_in authoritative_database ~append_create - ~fingerprint:record.fingerprint + ~fingerprint:record.fingerprint ~intent_time_ms:record.intent_time_ms ~planned_tx ~orders @@ -10221,7 +11263,8 @@ let logical_change_footprint ~transaction_data ~outbox_effects = - let before_context=read_context before and after_context=read_context after in + let before_context = read_context before + and after_context = read_context after in let direct = changed_entity_uuids authoritative_before authoritative_after transaction_data in @@ -10415,636 +11458,702 @@ let replay_authoritative database prepared = | exn -> Error (Printexc.to_string exn)) ;; -let report_timeline_commit graph_uuid checkpoint_before (commit : Types.authoritative_commit) = +let report_timeline_commit + graph_uuid + checkpoint_before + (commit : Types.authoritative_commit) + = try - let kind, blocks, pages, interests = match commit.logical_change_summary with + let kind, blocks, pages, interests = + match commit.logical_change_summary with | Types.No_logical_change -> "none", [], [], [] | Logical_resync_required _ -> "resync", [], [], [] | Exact_logical_change value -> "exact", value.block_uuids, value.page_uuids, value.structure_interests in - let uuids values = `List (List.take 16 values |> List.map (fun u -> `String (Graph.Uuid.to_string u))) in + let uuids values = + `List (List.take 16 values |> List.map (fun u -> `String (Graph.Uuid.to_string u))) + in let interest = function - | Types.Children_interest u -> `Assoc ["kind", `String "children"; "target", `String (Graph.Uuid.to_string u)] - | Page_tree_interest u -> `Assoc ["kind", `String "page-tree"; "target", `String (Graph.Uuid.to_string u)] - | Journal_index_interest -> `Assoc ["kind", `String "journal-index"] + | Types.Children_interest u -> + `Assoc [ "kind", `String "children"; "target", `String (Graph.Uuid.to_string u) ] + | Page_tree_interest u -> + `Assoc [ "kind", `String "page-tree"; "target", `String (Graph.Uuid.to_string u) ] + | Journal_index_interest -> `Assoc [ "kind", `String "journal-index" ] in let fields = [ "stage", `String "overlay-commit" ; "graph", `String (Graph.Uuid.to_string graph_uuid) ; "generation", `String (Types.Generation.to_string commit.generation) - ; "checkpointBefore", `String (Int64.to_string (Types.Server_cursor.to_int64 checkpoint_before)) - ; "checkpoint", `String (Int64.to_string (Types.Server_cursor.to_int64 commit.checkpoint)) - ; "beforeProjection", `String (Types.Projection_revision.to_string commit.before_projection_revision) - ; "projection", `String (Types.Projection_revision.to_string commit.after_projection_revision) + ; ( "checkpointBefore" + , `String (Int64.to_string (Types.Server_cursor.to_int64 checkpoint_before)) ) + ; ( "checkpoint" + , `String (Int64.to_string (Types.Server_cursor.to_int64 commit.checkpoint)) ) + ; ( "beforeProjection" + , `String (Types.Projection_revision.to_string commit.before_projection_revision) + ) + ; ( "projection" + , `String (Types.Projection_revision.to_string commit.after_projection_revision) ) ; "changeKind", `String kind - ; "blockCount", `Int (List.length blocks); "blocks", uuids blocks - ; "pageCount", `Int (List.length pages); "pages", uuids pages + ; "blockCount", `Int (List.length blocks) + ; "blocks", uuids blocks + ; "pageCount", `Int (List.length pages) + ; "pages", uuids pages ; "interestCount", `Int (List.length interests) ; "interests", `List (List.take 16 interests |> List.map interest) - ; "truncated", `Bool (List.length blocks > 16 || List.length pages > 16 || List.length interests > 16) - ] in + ; ( "truncated" + , `Bool + (List.length blocks > 16 + || List.length pages > 16 + || List.length interests > 16) ) + ] + in let text = Yojson.Safe.to_string (`Assoc fields) in if String.length text <= 8192 then report_warning ("journal-timeline: " ^ text ^ "\n") - with _ -> () + with + | _ -> () ;; let commit_authoritative_candidate database prepared = let checkpoint_before = ref None in - let outcome = serialize_commit database (fun () -> - let change, outcome = - Eio.Mutex.use_rw ~protect:true database.lock (fun () -> - let preparation = prepared.authoritative_preparation in - if preparation.authoritative_owner != database - then None, Error Types.Authoritative_commit_generation_invalidated - else if database.closed - then None, Error Types.Authoritative_commit_database_closed - else if database.sync_revision <> preparation.authoritative_revision - then None, Error Types.Authoritative_commit_token_conflict - else ( - match - Logseq_db_storage.Storage_session.stage_transact_batch - database.storage_session - ~authoritative_before:(authoritative_database database) - prepared.authoritative_transactions - with - | Error error -> - ( None - , Error - (Types.Authoritative_commit_persistence_failed - (storage_session_error error)) ) - | Ok staged -> - let through = - Types.authoritative_batch_through preparation.authoritative_batch - in - let authoritative_after = prepared.authoritative_db_after in - let authoritative_before = authoritative_database database in - (* Origin validation compares complete transaction data at its expected + let outcome = + serialize_commit database (fun () -> + let change, outcome = + Eio.Mutex.use_rw ~protect:true database.lock (fun () -> + let preparation = prepared.authoritative_preparation in + if preparation.authoritative_owner != database + then None, Error Types.Authoritative_commit_generation_invalidated + else if database.closed + then None, Error Types.Authoritative_commit_database_closed + else if database.sync_revision <> preparation.authoritative_revision + then None, Error Types.Authoritative_commit_token_conflict + else ( + match + Logseq_db_storage.Storage_session.stage_transact_batch + database.storage_session + ~authoritative_before:(authoritative_database database) + prepared.authoritative_transactions + with + | Error error -> + ( None + , Error + (Types.Authoritative_commit_persistence_failed + (storage_session_error error)) ) + | Ok staged -> + let through = + Types.authoritative_batch_through preparation.authoritative_batch + in + let authoritative_after = prepared.authoritative_db_after in + let authoritative_before = authoritative_database database in + (* Origin validation compares complete transaction data at its expected cursor. Earlier local effects were already visible when a queued delete was authored; their echoes are not competing remote edits. *) - let local_sequences = Hashtbl.create (List.length database.outbox) in - List.iter (fun (record : outbox_record) -> - Hashtbl.replace local_sequences (Graph.Uuid.to_string record.mutation_id) - record.sequence) database.outbox; - let origin_sequences = Hashtbl.create (List.length prepared.validated_origins) in - List.iter (fun (id, cursor) -> - Option.iter (fun sequence -> - Hashtbl.replace origin_sequences (Types.Server_cursor.to_int64 cursor) sequence) - (Hashtbl.find_opt local_sequences (Graph.Uuid.to_string id))) - prepared.validated_origins; - let _, conflict_contexts = List.fold_left2 - (fun (before, contexts) transaction after -> - let origin = Hashtbl.find_opt origin_sequences - (Types.Server_cursor.to_int64 - (Types.authoritative_transaction_cursor transaction)) in - after, Rrbvec.push_back contexts (before, after, origin)) - (authoritative_before, Rrbvec.empty) - (Types.authoritative_batch_transactions preparation.authoritative_batch) - prepared.authoritative_roots_after in - let module Conflict_set = Set.Make (struct - type t = Types.delete_conflict_kind - let compare = Stdlib.compare - end) in - let queued_delete_conflicts (record : outbox_record) = - let earlier = function Some sequence -> sequence < record.sequence | None -> false in - if not (Rrbvec.exists (fun (_, _, origin) -> earlier origin) conflict_contexts) - then delete_conflict_kinds_between authoritative_before authoritative_after record - else - Rrbvec.fold_left - (fun conflicts (before, after, origin) -> - if earlier origin then conflicts - else List.fold_left - (fun conflicts kind -> Conflict_set.add kind conflicts) - conflicts (delete_conflict_kinds_between before after record)) - Conflict_set.empty conflict_contexts - |> Conflict_set.elements - in - let remote_won_queued = - List.filter_map + let local_sequences = Hashtbl.create (List.length database.outbox) in + List.iter (fun (record : outbox_record) -> - match record.transport_state, record.mutation with - | Queued, Types.Delete_blocks { root; _ } -> - let after = block_of_database authoritative_after root in - let conflicts = - queued_delete_conflicts record + Hashtbl.replace + local_sequences + (Graph.Uuid.to_string record.mutation_id) + record.sequence) + database.outbox; + let origin_sequences = + Hashtbl.create (List.length prepared.validated_origins) + in + List.iter + (fun (id, cursor) -> + Option.iter + (fun sequence -> + Hashtbl.replace + origin_sequences + (Types.Server_cursor.to_int64 cursor) + sequence) + (Hashtbl.find_opt local_sequences (Graph.Uuid.to_string id))) + prepared.validated_origins; + let _, conflict_contexts = + List.fold_left2 + (fun (before, contexts) transaction after -> + let origin = + Hashtbl.find_opt + origin_sequences + (Types.Server_cursor.to_int64 + (Types.authoritative_transaction_cursor transaction)) in - if conflicts <> [] && Option.is_some after - then Some (record, conflicts) - else None - | _ -> None) - database.outbox - in - let remote_won_submitted = - List.filter_map - (fun (mutation_id, batch_id, cursor, conflicts) -> - List.find_opt - (fun (record : outbox_record) -> - Graph.Uuid.equal record.mutation_id mutation_id) - database.outbox - |> Option.map (fun record -> record, batch_id, cursor, conflicts)) - prepared.submitted_remote_won - in - let record_for id = - List.find_opt - (fun (record : outbox_record) -> Graph.Uuid.equal record.mutation_id id) - database.outbox - in - let stale_remote_won = - List.filter_map - (fun (id, batch_id, earliest, rejection_through, conflicts) -> - record_for id - |> Option.map (fun record -> - record, batch_id, earliest, rejection_through, conflicts)) - prepared.stale_remote_won - in - let resolved_stale = - List.filter_map - (fun (id, batch_id) -> - record_for id |> Option.map (fun record -> record, batch_id)) - prepared.stale_no_change - in - let blocked_stale = - List.filter_map - (fun (id, batch_id) -> - record_for id |> Option.map (fun record -> record, batch_id)) - prepared.stale_blocked - in - let covered_accepted = - List.filter - (fun (record : outbox_record) -> - match record.transport_state, record.acceptance_barrier with - | Accepted_pending_authoritative _, Some barrier -> - Types.Server_cursor.compare barrier.through through <= 0 - | _ -> false) - database.outbox - in - let authoritative_root_at barrier = - let target = barrier.Types.through in - if Types.Server_cursor.compare target database.checkpoint <= 0 - then Some authoritative_before - else - List.combine + after, Rrbvec.push_back contexts (before, after, origin)) + (authoritative_before, Rrbvec.empty) (Types.authoritative_batch_transactions preparation.authoritative_batch) prepared.authoritative_roots_after - |> List.find_map (fun (transaction, root) -> - if - Types.Server_cursor.equal - (Types.authoritative_transaction_cursor transaction) - target - then Some root - else None) - in - let accepted_requirement_satisfied (record : outbox_record) = - match record.acceptance_barrier with - | None -> false - | Some barrier -> - let barrier_cursor = barrier.through in - (match record.mutation with - | Save_block _ - | Insert_blocks _ - | Append_journal _ - | Create_journal_page _ - | Set_task_status _ - | Clear_task_status _ - when Types.Server_cursor.compare barrier_cursor database.checkpoint < 0 - -> true - | _ -> - (match authoritative_root_at barrier, record.transport_state with - | Some root, Accepted_pending_authoritative batch_id -> - accepted_record_satisfied - root - (batch_records database batch_id) - record - | None, _ | Some _, _ -> false)) - in - let resolved_accepted, mismatched_accepted = - List.partition accepted_requirement_satisfied covered_accepted - in - let resolved_records = List.map fst resolved_stale @ resolved_accepted in - let candidate_outbox = - List.filter_map - (fun (record : outbox_record) -> - if - List.exists (fun resolved -> resolved == record) resolved_records - || List.exists - (fun (remote_won, _) -> remote_won == record) - remote_won_queued - || List.exists - (fun (remote_won, _, _, _) -> remote_won == record) - remote_won_submitted - || List.exists - (fun (remote_won, _, _, _, _) -> remote_won == record) - stale_remote_won - then None - else if - List.exists - (fun mismatched -> mismatched == record) - mismatched_accepted - then - Some - { record with - transport_state = Blocked - ; dependency_shadows = empty_dependency_shadows - ; blocked_prior_state = Some record.transport_state - ; blocked_reason = Some Authoritative_mismatch - ; same_id_retry_eligible = false - ; acceptance_barrier = None - } - else if - List.exists (fun (blocked, _) -> blocked == record) blocked_stale - then - Some - { record with - transport_state = Blocked - ; dependency_shadows = empty_dependency_shadows - ; blocked_prior_state = Some record.transport_state - ; blocked_reason = Some Stale_barrier - ; same_id_retry_eligible = false - ; acceptance_barrier = None - } - else ( - let record_with_origin_evidence () = - match - List.find_opt - (fun (id, _) -> Graph.Uuid.equal id record.mutation_id) - prepared.validated_origins - with - | None -> record - | Some (_, cursor) -> - { record with observed_origin_cursor = Some cursor } - in - match - List.find_opt - (fun (id, _, _) -> Graph.Uuid.equal id record.mutation_id) - prepared.stale_updates - with - | None -> Some (record_with_origin_evidence ()) - | Some (_, earliest, conflicts) -> - Some - { (record_with_origin_evidence ()) with - stale_earliest_conflict_cursor = earliest - ; stale_conflicts = conflicts - })) - database.outbox - in - let candidate_outbox, stale_executed, stale_batches = - settle_stale_records - ~checkpoint:through - ~origins:prepared.validated_origins - candidate_outbox - in - let resolved_records = resolved_records @ stale_executed in - let candidate_outbox, replanned_queued, no_change_queued, blocked_queued = - replan_queued_ordinary database authoritative_after candidate_outbox - in - let receipt_revision = projection_revision database.projection in - let candidate_receipts = - List.fold_left - (fun receipts (record : outbox_record) -> - let commit = - { Types.mutation_id = record.mutation_id - ; status = - (if - List.exists (fun (stale, _) -> stale == record) resolved_stale - then No_change - else Applied) - ; generation = database.generation - ; before_projection_revision = receipt_revision - ; after_projection_revision = receipt_revision - ; logical_change_summary = No_logical_change - } - in - (record.mutation_id, record.fingerprint, Commit_receipt commit) - :: List.filter - (fun (candidate, _, _) -> - not (Graph.Uuid.equal candidate record.mutation_id)) - receipts) - database.receipts - resolved_records - in - let candidate_receipts = - List.fold_left - (fun receipts ((record : outbox_record), conflicts) -> - ( record.mutation_id - , record.fingerprint - , Remote_won_entry - (remote_won_before_submission ~conflict_kinds:conflicts record) ) - :: List.filter - (fun (candidate, _, _) -> - not (Graph.Uuid.equal candidate record.mutation_id)) - receipts) - candidate_receipts - remote_won_queued - in - let candidate_receipts = - List.fold_left - (fun receipts (record, batch_id, cursor, conflicts) -> - ( record.mutation_id - , record.fingerprint - , Remote_won_entry - (remote_won_proven_unexecuted - ~conflict_kinds:conflicts - record - ~batch_id - ~earliest_conflict_cursor:cursor) ) - :: List.filter - (fun (candidate, _, _) -> - not (Graph.Uuid.equal candidate record.mutation_id)) - receipts) - candidate_receipts - remote_won_submitted - in - let candidate_receipts = - List.fold_left - (fun receipts - (record, batch_id, earliest, rejection_through, conflicts) -> - ( record.mutation_id - , record.fingerprint - , Remote_won_entry - (remote_won_proven_unexecuted - ~rejection_through - ~conflict_kinds:conflicts + in + let module Conflict_set = Set.Make (struct + type t = Types.delete_conflict_kind + + let compare = Stdlib.compare + end) + in + let queued_delete_conflicts (record : outbox_record) = + let earlier = function + | Some sequence -> sequence < record.sequence + | None -> false + in + if + not + (Rrbvec.exists + (fun (_, _, origin) -> earlier origin) + conflict_contexts) + then + delete_conflict_kinds_between + authoritative_before + authoritative_after + record + else + Rrbvec.fold_left + (fun conflicts (before, after, origin) -> + if earlier origin + then conflicts + else + List.fold_left + (fun conflicts kind -> Conflict_set.add kind conflicts) + conflicts + (delete_conflict_kinds_between before after record)) + Conflict_set.empty + conflict_contexts + |> Conflict_set.elements + in + let remote_won_queued = + List.filter_map + (fun (record : outbox_record) -> + match record.transport_state, record.mutation with + | Queued, Types.Delete_blocks { root; _ } -> + let after = block_of_database authoritative_after root in + let conflicts = queued_delete_conflicts record in + if conflicts <> [] && Option.is_some after + then Some (record, conflicts) + else None + | _ -> None) + database.outbox + in + let remote_won_submitted = + List.filter_map + (fun (mutation_id, batch_id, cursor, conflicts) -> + List.find_opt + (fun (record : outbox_record) -> + Graph.Uuid.equal record.mutation_id mutation_id) + database.outbox + |> Option.map (fun record -> record, batch_id, cursor, conflicts)) + prepared.submitted_remote_won + in + let record_for id = + List.find_opt + (fun (record : outbox_record) -> Graph.Uuid.equal record.mutation_id id) + database.outbox + in + let stale_remote_won = + List.filter_map + (fun (id, batch_id, earliest, rejection_through, conflicts) -> + record_for id + |> Option.map (fun record -> + record, batch_id, earliest, rejection_through, conflicts)) + prepared.stale_remote_won + in + let resolved_stale = + List.filter_map + (fun (id, batch_id) -> + record_for id |> Option.map (fun record -> record, batch_id)) + prepared.stale_no_change + in + let blocked_stale = + List.filter_map + (fun (id, batch_id) -> + record_for id |> Option.map (fun record -> record, batch_id)) + prepared.stale_blocked + in + let covered_accepted = + List.filter + (fun (record : outbox_record) -> + match record.transport_state, record.acceptance_barrier with + | Accepted_pending_authoritative _, Some barrier -> + Types.Server_cursor.compare barrier.through through <= 0 + | _ -> false) + database.outbox + in + let authoritative_root_at barrier = + let target = barrier.Types.through in + if Types.Server_cursor.compare target database.checkpoint <= 0 + then Some authoritative_before + else + List.combine + (Types.authoritative_batch_transactions + preparation.authoritative_batch) + prepared.authoritative_roots_after + |> List.find_map (fun (transaction, root) -> + if + Types.Server_cursor.equal + (Types.authoritative_transaction_cursor transaction) + target + then Some root + else None) + in + let accepted_requirement_satisfied (record : outbox_record) = + match record.acceptance_barrier with + | None -> false + | Some barrier -> + let barrier_cursor = barrier.through in + (match record.mutation with + | Save_block _ + | Insert_blocks _ + | Append_journal _ + | Create_journal_page _ + | Set_task_status _ + | Clear_task_status _ + when Types.Server_cursor.compare barrier_cursor database.checkpoint + < 0 -> true + | _ -> + (match authoritative_root_at barrier, record.transport_state with + | Some root, Accepted_pending_authoritative batch_id -> + accepted_record_satisfied + root + (batch_records database batch_id) record - ~batch_id - ~earliest_conflict_cursor:earliest) ) - :: List.filter - (fun (candidate, _, _) -> - not (Graph.Uuid.equal candidate record.mutation_id)) - receipts) - candidate_receipts - stale_remote_won - in - let candidate_receipts = - List.fold_left - (fun receipts (record : outbox_record) -> - let commit = - { Types.mutation_id = record.mutation_id - ; status = No_change - ; generation = database.generation - ; before_projection_revision = receipt_revision - ; after_projection_revision = receipt_revision - ; logical_change_summary = No_logical_change - } - in - (record.mutation_id, record.fingerprint, Commit_receipt commit) - :: List.filter - (fun (candidate, _, _) -> - not (Graph.Uuid.equal candidate record.mutation_id)) - receipts) - candidate_receipts - no_change_queued - in - let terminal_receipts = - List.map - (fun (record : outbox_record) -> - let receipt = - if List.exists (fun (stale, _) -> stale == record) resolved_stale + | None, _ | Some _, _ -> false)) + in + let resolved_accepted, mismatched_accepted = + List.partition accepted_requirement_satisfied covered_accepted + in + let resolved_records = List.map fst resolved_stale @ resolved_accepted in + let candidate_outbox = + List.filter_map + (fun (record : outbox_record) -> + if + List.exists (fun resolved -> resolved == record) resolved_records + || List.exists + (fun (remote_won, _) -> remote_won == record) + remote_won_queued + || List.exists + (fun (remote_won, _, _, _) -> remote_won == record) + remote_won_submitted + || List.exists + (fun (remote_won, _, _, _, _) -> remote_won == record) + stale_remote_won + then None + else if + List.exists + (fun mismatched -> mismatched == record) + mismatched_accepted then - Types.No_change_receipt - { mutation_id = record.mutation_id - ; fingerprint = typed_mutation_fingerprint record.fingerprint + Some + { record with + transport_state = Blocked + ; dependency_shadows = empty_dependency_shadows + ; blocked_prior_state = Some record.transport_state + ; blocked_reason = Some Authoritative_mismatch + ; same_id_retry_eligible = false + ; acceptance_barrier = None } - else - Applied_receipt - { mutation_id = record.mutation_id - ; fingerprint = typed_mutation_fingerprint record.fingerprint + else if + List.exists (fun (blocked, _) -> blocked == record) blocked_stale + then + Some + { record with + transport_state = Blocked + ; dependency_shadows = empty_dependency_shadows + ; blocked_prior_state = Some record.transport_state + ; blocked_reason = Some Stale_barrier + ; same_id_retry_eligible = false + ; acceptance_barrier = None } - in - Types.{ receipt; transport_disposition = Clear_transport_owner }) - resolved_records - in - let terminal_receipts = - terminal_receipts - @ List.map - (fun ((record : outbox_record), conflicts) -> - Types. - { receipt = - Remote_won_receipt - (remote_won_before_submission - ~conflict_kinds:conflicts - record) - ; transport_disposition = No_transport_owner - }) + else ( + let record_with_origin_evidence () = + match + List.find_opt + (fun (id, _) -> Graph.Uuid.equal id record.mutation_id) + prepared.validated_origins + with + | None -> record + | Some (_, cursor) -> + { record with observed_origin_cursor = Some cursor } + in + match + List.find_opt + (fun (id, _, _) -> Graph.Uuid.equal id record.mutation_id) + prepared.stale_updates + with + | None -> Some (record_with_origin_evidence ()) + | Some (_, earliest, conflicts) -> + Some + { (record_with_origin_evidence ()) with + stale_earliest_conflict_cursor = earliest + ; stale_conflicts = conflicts + })) + database.outbox + in + let candidate_outbox, stale_executed, stale_batches = + settle_stale_records + ~checkpoint:through + ~origins:prepared.validated_origins + candidate_outbox + in + let resolved_records = resolved_records @ stale_executed in + let candidate_outbox, replanned_queued, no_change_queued, blocked_queued = + replan_queued_ordinary database authoritative_after candidate_outbox + in + let receipt_revision = projection_revision database.projection in + let candidate_receipts = + List.fold_left + (fun receipts (record : outbox_record) -> + let commit = + { Types.mutation_id = record.mutation_id + ; status = + (if + List.exists + (fun (stale, _) -> stale == record) + resolved_stale + then No_change + else Applied) + ; generation = database.generation + ; before_projection_revision = receipt_revision + ; after_projection_revision = receipt_revision + ; logical_change_summary = No_logical_change + } + in + (record.mutation_id, record.fingerprint, Commit_receipt commit) + :: List.filter + (fun (candidate, _, _) -> + not (Graph.Uuid.equal candidate record.mutation_id)) + receipts) + database.receipts + resolved_records + in + let candidate_receipts = + List.fold_left + (fun receipts ((record : outbox_record), conflicts) -> + ( record.mutation_id + , record.fingerprint + , Remote_won_entry + (remote_won_before_submission ~conflict_kinds:conflicts record) + ) + :: List.filter + (fun (candidate, _, _) -> + not (Graph.Uuid.equal candidate record.mutation_id)) + receipts) + candidate_receipts remote_won_queued - in - let terminal_receipts = - terminal_receipts - @ List.map - (fun (record, batch_id, cursor, conflicts) -> - Types. - { receipt = - Remote_won_receipt - (remote_won_proven_unexecuted - ~conflict_kinds:conflicts - record - ~batch_id - ~earliest_conflict_cursor:cursor) - ; transport_disposition = - Retain_terminal_owner_until_response batch_id - }) + in + let candidate_receipts = + List.fold_left + (fun receipts (record, batch_id, cursor, conflicts) -> + ( record.mutation_id + , record.fingerprint + , Remote_won_entry + (remote_won_proven_unexecuted + ~conflict_kinds:conflicts + record + ~batch_id + ~earliest_conflict_cursor:cursor) ) + :: List.filter + (fun (candidate, _, _) -> + not (Graph.Uuid.equal candidate record.mutation_id)) + receipts) + candidate_receipts remote_won_submitted - in - let terminal_receipts = - terminal_receipts - @ List.map - (fun (record, batch_id, earliest, rejection_through, conflicts) -> - Types. - { receipt = - Remote_won_receipt - (remote_won_proven_unexecuted - ~rejection_through - ~conflict_kinds:conflicts - record - ~batch_id - ~earliest_conflict_cursor:earliest) - ; transport_disposition = Clear_transport_owner - }) + in + let candidate_receipts = + List.fold_left + (fun receipts + (record, batch_id, earliest, rejection_through, conflicts) -> + ( record.mutation_id + , record.fingerprint + , Remote_won_entry + (remote_won_proven_unexecuted + ~rejection_through + ~conflict_kinds:conflicts + record + ~batch_id + ~earliest_conflict_cursor:earliest) ) + :: List.filter + (fun (candidate, _, _) -> + not (Graph.Uuid.equal candidate record.mutation_id)) + receipts) + candidate_receipts stale_remote_won - in - let terminal_receipts = - terminal_receipts - @ List.map - (fun (record : outbox_record) -> - Types. - { receipt = - No_change_receipt - { mutation_id = record.mutation_id - ; fingerprint = typed_mutation_fingerprint record.fingerprint - } - ; transport_disposition = No_transport_owner - }) + in + let candidate_receipts = + List.fold_left + (fun receipts (record : outbox_record) -> + let commit = + { Types.mutation_id = record.mutation_id + ; status = No_change + ; generation = database.generation + ; before_projection_revision = receipt_revision + ; after_projection_revision = receipt_revision + ; logical_change_summary = No_logical_change + } + in + (record.mutation_id, record.fingerprint, Commit_receipt commit) + :: List.filter + (fun (candidate, _, _) -> + not (Graph.Uuid.equal candidate record.mutation_id)) + receipts) + candidate_receipts no_change_queued - in - let terminal_batches = - stale_batches - @ List.map - (fun (record, batch_id, _cursor, _conflicts) -> - { terminal_batch_id = batch_id - ; terminal_outcome = - Terminal_proven_unexecuted - { mutation_id = record.mutation_id - ; fingerprint = record.fingerprint - } - }) - remote_won_submitted - @ List.map - (fun (record, batch_id, _, _, _) -> - { terminal_batch_id = batch_id - ; terminal_outcome = - Terminal_proven_unexecuted + in + let terminal_receipts = + List.map + (fun (record : outbox_record) -> + let receipt = + if List.exists (fun (stale, _) -> stale == record) resolved_stale + then + Types.No_change_receipt { mutation_id = record.mutation_id - ; fingerprint = record.fingerprint + ; fingerprint = typed_mutation_fingerprint record.fingerprint } - }) - stale_remote_won - @ List.map - (fun (record, batch_id) -> - { terminal_batch_id = batch_id - ; terminal_outcome = - Terminal_proven_unexecuted + else + Applied_receipt { mutation_id = record.mutation_id - ; fingerprint = record.fingerprint + ; fingerprint = typed_mutation_fingerprint record.fingerprint } - }) - (resolved_stale @ blocked_stale) - in - let checksum = - Types.authoritative_batch_checksum preparation.authoritative_batch - |> Option.map Types.Checksum.to_string - |> Option.map (strip_versioned_prefix "checksum") - |> Option.value ~default:database.checkpoint_metadata.checksum - in - let metadata = - Logseq_db_types.Sync_checkpoint.create - ~graph_id:database.graph_uuid - ~schema:database.schema - ~applied_server_t:through - ~checksum - |> Result.get_ok - in - let outbox_delta, durable_outbox = - outbox_write_set - database - ~revision:(database.sync_revision + 1) - candidate_outbox - in - let receipts = - List.filter_map Persistence_receipt_v1.encode_mutation candidate_receipts - in - (match - Logseq_db_storage.Storage_session - .commit_staged_with_sync_metadata_outbox_and_receipts - ~outbox_delta - database.storage_session - staged - metadata - [] - receipts - (List.map Persistence_receipt_v1.encode_terminal_batch terminal_batches) - with - | Error error -> - ( None - , Error - (Types.Authoritative_commit_persistence_failed - (storage_session_error error)) ) - | Ok () -> - database.durable_outbox <- durable_outbox; - database.durable_sync_revision <- database.sync_revision + 1; - preparation.authoritative_state <- Authoritative_consumed; - checkpoint_before := Some database.checkpoint; - let before_projection = database.projection in - let before_snapshot = snapshot_of_database database in - (match replay_authoritative database prepared with - | Error message -> - database.closed <- true; - None, Error (Types.Authoritative_commit_fatal_state message) - | Ok () -> - database.outbox <- candidate_outbox; - database.receipts <- []; - refresh_queryable_outbox database; - let candidate_projection = before_projection + 1 in - let after_snapshot = - snapshot_for_sources database authoritative_after candidate_outbox - in - let transitioned_records = - resolved_records - @ List.map fst remote_won_queued - @ List.map (fun (record, _, _, _) -> record) remote_won_submitted - @ List.map (fun (record, _, _, _, _) -> record) stale_remote_won - @ List.map fst blocked_stale - @ mismatched_accepted - @ no_change_queued - @ blocked_queued - @ replanned_queued - in - let block_uuids, page_uuids, structure_interests, fanout_overflow = - logical_change_footprint - ~before:before_snapshot - ~after:after_snapshot - ~authoritative_before - ~authoritative_after - ~transaction_data:prepared.authoritative_expected_tx_data - ~outbox_effects: - (List.map - (fun (record : outbox_record) -> record.effect_footprint) - transitioned_records) - in - let changed = - fanout_overflow - || block_uuids <> [] - || page_uuids <> [] - || structure_interests <> [] - in - database.projection - <- (if changed then candidate_projection else before_projection); - database.checkpoint <- through; - database.checkpoint_metadata <- metadata; - database.sync_revision <- database.sync_revision + 1; - let before_revision = projection_revision before_projection in - let after_revision = projection_revision database.projection in - let logical_change_summary = - if fanout_overflow - then Types.Logical_resync_required Change_limit_exceeded - else if changed - then - bounded_logical_summary - database - ~block_uuids - ~page_uuids - ~structure_interests - else No_logical_change - in - let change = - Change_dispatcher.event - ~generation:database.generation - ~before_revision - ~after_revision - logical_change_summary - in - ( change - , Ok - { Types.generation = database.generation - ; before_projection_revision = before_revision - ; after_projection_revision = after_revision - ; checkpoint = through - ; sync_token = sync_token database.sync_revision - ; terminal_receipts - ; replanned_queued_ids = - List.map - (fun (record : outbox_record) -> record.mutation_id) - replanned_queued - ; blocked_ids = - List.map - (fun (record : outbox_record) -> record.mutation_id) - mismatched_accepted - @ List.map (fun (record, _) -> record.mutation_id) blocked_stale - @ List.map + in + Types.{ receipt; transport_disposition = Clear_transport_owner }) + resolved_records + in + let terminal_receipts = + terminal_receipts + @ List.map + (fun ((record : outbox_record), conflicts) -> + Types. + { receipt = + Remote_won_receipt + (remote_won_before_submission + ~conflict_kinds:conflicts + record) + ; transport_disposition = No_transport_owner + }) + remote_won_queued + in + let terminal_receipts = + terminal_receipts + @ List.map + (fun (record, batch_id, cursor, conflicts) -> + Types. + { receipt = + Remote_won_receipt + (remote_won_proven_unexecuted + ~conflict_kinds:conflicts + record + ~batch_id + ~earliest_conflict_cursor:cursor) + ; transport_disposition = + Retain_terminal_owner_until_response batch_id + }) + remote_won_submitted + in + let terminal_receipts = + terminal_receipts + @ List.map + (fun (record, batch_id, earliest, rejection_through, conflicts) -> + Types. + { receipt = + Remote_won_receipt + (remote_won_proven_unexecuted + ~rejection_through + ~conflict_kinds:conflicts + record + ~batch_id + ~earliest_conflict_cursor:earliest) + ; transport_disposition = Clear_transport_owner + }) + stale_remote_won + in + let terminal_receipts = + terminal_receipts + @ List.map + (fun (record : outbox_record) -> + Types. + { receipt = + No_change_receipt + { mutation_id = record.mutation_id + ; fingerprint = + typed_mutation_fingerprint record.fingerprint + } + ; transport_disposition = No_transport_owner + }) + no_change_queued + in + let terminal_batches = + stale_batches + @ List.map + (fun (record, batch_id, _cursor, _conflicts) -> + { terminal_batch_id = batch_id + ; terminal_outcome = + Terminal_proven_unexecuted + { mutation_id = record.mutation_id + ; fingerprint = record.fingerprint + } + }) + remote_won_submitted + @ List.map + (fun (record, batch_id, _, _, _) -> + { terminal_batch_id = batch_id + ; terminal_outcome = + Terminal_proven_unexecuted + { mutation_id = record.mutation_id + ; fingerprint = record.fingerprint + } + }) + stale_remote_won + @ List.map + (fun (record, batch_id) -> + { terminal_batch_id = batch_id + ; terminal_outcome = + Terminal_proven_unexecuted + { mutation_id = record.mutation_id + ; fingerprint = record.fingerprint + } + }) + (resolved_stale @ blocked_stale) + in + let checksum = + Types.authoritative_batch_checksum preparation.authoritative_batch + |> Option.map Types.Checksum.to_string + |> Option.map (strip_versioned_prefix "checksum") + |> Option.value ~default:database.checkpoint_metadata.checksum + in + let metadata = + Logseq_db_types.Sync_checkpoint.create + ~graph_id:database.graph_uuid + ~schema:database.schema + ~applied_server_t:through + ~checksum + |> Result.get_ok + in + let outbox_delta, durable_outbox = + outbox_write_set + database + ~revision:(database.sync_revision + 1) + candidate_outbox + in + let receipts = + List.filter_map Persistence_receipt_v1.encode_mutation candidate_receipts + in + (match + Logseq_db_storage.Storage_session + .commit_staged_with_sync_metadata_outbox_and_receipts + ~outbox_delta + database.storage_session + staged + metadata + [] + receipts + (List.map + Persistence_receipt_v1.encode_terminal_batch + terminal_batches) + with + | Error error -> + ( None + , Error + (Types.Authoritative_commit_persistence_failed + (storage_session_error error)) ) + | Ok () -> + database.durable_outbox <- durable_outbox; + database.durable_sync_revision <- database.sync_revision + 1; + preparation.authoritative_state <- Authoritative_consumed; + checkpoint_before := Some database.checkpoint; + let before_projection = database.projection in + let before_snapshot = snapshot_of_database database in + (match replay_authoritative database prepared with + | Error message -> + database.closed <- true; + None, Error (Types.Authoritative_commit_fatal_state message) + | Ok () -> + database.outbox <- candidate_outbox; + database.receipts <- []; + refresh_queryable_outbox database; + let candidate_projection = before_projection + 1 in + let after_snapshot = + snapshot_for_sources database authoritative_after candidate_outbox + in + let transitioned_records = + resolved_records + @ List.map fst remote_won_queued + @ List.map (fun (record, _, _, _) -> record) remote_won_submitted + @ List.map (fun (record, _, _, _, _) -> record) stale_remote_won + @ List.map fst blocked_stale + @ mismatched_accepted + @ no_change_queued + @ blocked_queued + @ replanned_queued + in + let block_uuids, page_uuids, structure_interests, fanout_overflow = + logical_change_footprint + ~before:before_snapshot + ~after:after_snapshot + ~authoritative_before + ~authoritative_after + ~transaction_data:prepared.authoritative_expected_tx_data + ~outbox_effects: + (List.map + (fun (record : outbox_record) -> record.effect_footprint) + transitioned_records) + in + let changed = + fanout_overflow + || block_uuids <> [] + || page_uuids <> [] + || structure_interests <> [] + in + database.projection + <- (if changed then candidate_projection else before_projection); + database.authoritative_checksum <- prepared.authoritative_digest_after; + database.checkpoint <- through; + database.checkpoint_metadata <- metadata; + database.sync_revision <- database.sync_revision + 1; + Eio.Condition.broadcast database.sync_changed; + let before_revision = projection_revision before_projection in + let after_revision = projection_revision database.projection in + let logical_change_summary = + if fanout_overflow + then Types.Logical_resync_required Change_limit_exceeded + else if changed + then + bounded_logical_summary + database + ~block_uuids + ~page_uuids + ~structure_interests + else No_logical_change + in + let change = + Change_dispatcher.event + ~generation:database.generation + ~before_revision + ~after_revision + logical_change_summary + in + ( change + , Ok + { Types.generation = database.generation + ; before_projection_revision = before_revision + ; after_projection_revision = after_revision + ; checkpoint = through + ; sync_token = sync_token database.sync_revision + ; terminal_receipts + ; replanned_queued_ids = + List.map (fun (record : outbox_record) -> record.mutation_id) - blocked_queued - ; logical_change_summary - } ))))) - in - change, outcome) in + replanned_queued + ; blocked_ids = + List.map + (fun (record : outbox_record) -> record.mutation_id) + mismatched_accepted + @ List.map + (fun (record, _) -> record.mutation_id) + blocked_stale + @ List.map + (fun (record : outbox_record) -> record.mutation_id) + blocked_queued + ; logical_change_summary + } ))))) + in + change, outcome) + in (match outcome, !checkpoint_before with | Ok commit, Some before -> report_timeline_commit database.graph_uuid before commit | _ -> ()); @@ -11412,8 +12521,7 @@ let inspect_local_mutation database mutation = Eio.Mutex.use_rw ~protect:true database.lock (fun () -> if database.closed then Error Types.Local_database_closed - else - existing_mutation database (mutation_id mutation) (fingerprint mutation)) + else existing_mutation database (mutation_id mutation) (fingerprint mutation)) ;; let set_asset_reference database ~expected ~mutation_id ~block ~previous ~asset = diff --git a/logseq_overlay_db/spec/database.mli b/logseq_overlay_db/spec/database.mli index 3ae5fa58..1093f135 100644 --- a/logseq_overlay_db/spec/database.mli +++ b/logseq_overlay_db/spec/database.mli @@ -10,9 +10,15 @@ module Graph = Logseq_db_types.Graph_types +val inspect_snapshot_storage + : check_current:(unit -> unit) + -> scratch_bytes:int64 + -> encrypted:bool + -> path:string + -> (Types.Checksum.t, string) result + type dependencies type mirror_inspection - type t type snapshot @@ -134,9 +140,12 @@ val prepare_snapshot_activation (** Returns the next bounded ciphertext batch that the caller must decrypt, or [None] once snapshot unprotection is complete. Each returned request must be - supplied exactly once before asking for another batch. *) + supplied exactly once before asking for another batch. Positive caller bounds + can only tighten the dependency limits. *) val next_snapshot_unprotection_batch - : prepared_snapshot_activation + : ?maximum_items:int + -> ?maximum_bytes:int + -> prepared_snapshot_activation -> (unprotection_request option, Types.snapshot_activation_error) result (** Supplies plaintexts for the outstanding snapshot request. Foreign, stale, @@ -212,6 +221,7 @@ val cancel_snapshot_activation : prepared_snapshot_activation -> unit (** A deletion transaction identity, distinct from a graph UUID. *) type retirement_id = Graph.Retirement_id.t + val retirement_id_of_string : string -> (retirement_id, string) result val retirement_id_to_string : retirement_id -> string @@ -220,11 +230,14 @@ val retirement_id_to_string : retirement_id -> string contents, schema or checkpoint and does not delete data. Replaying the same identity recovers its durable commit receipt after a replacement was created. *) type prepared_mirror_deletion + val prepare_mirror_deletion : application_support_directory:string -> graph_id:Graph.Uuid.t -> retirement:retirement_id - -> (prepared_mirror_deletion * Types.mirror_generation, Types.mirror_delete_error) result + -> ( prepared_mirror_deletion * Types.mirror_generation + , Types.mirror_delete_error ) + result (** Whether this exact retirement has durable commit evidence. An unrelated missing or replaced canonical mirror is never commit evidence. *) @@ -245,7 +258,7 @@ val cancel_mirror_deletion : prepared_mirror_deletion -> unit (** Checked release for a handoff owner. A failed physical ownership close keeps the exact preparation and its cleanup obligation; success is the release ACK. This neither restores a committed namespace nor removes its durable proof. *) -val cancel_mirror_deletion_checked : prepared_mirror_deletion -> (unit,string)result +val cancel_mirror_deletion_checked : prepared_mirror_deletion -> (unit, string) result (** Collects eligible durable garbage without removing mutation receipts needed to classify late synchronization responses. *) @@ -369,12 +382,22 @@ type timeline_scope = | Parent_children of Graph.block_uuid type timeline_scope_version -val timeline_scope_version : snapshot -> timeline_scope -> (timeline_scope_version, Types.read_error) result -val equal_timeline_scope_version : timeline_scope_version -> timeline_scope_version -> bool + +val timeline_scope_version + : snapshot + -> timeline_scope + -> (timeline_scope_version, Types.read_error) result + +val equal_timeline_scope_version + : timeline_scope_version + -> timeline_scope_version + -> bool + val timeline_scope_version_to_string : timeline_scope_version -> string type journal_position type member_position + val journal_position_to_string : journal_position -> string val journal_position_of_string : string -> (journal_position, Types.read_error) result val member_position_to_string : member_position -> string @@ -388,14 +411,25 @@ val member_position_of_string : string -> (member_position, Types.read_error) re All operations enforce item, structural and ancestry bounds; errors never silently turn truncated data into exhaustion. *) val get_timeline_journals - : snapshot -> from_day:int -> through_day:int -> limit:int + : snapshot + -> from_day:int + -> through_day:int + -> limit:int -> after:journal_position option -> (Types.journal_item list * journal_position option, Types.read_error) result + val get_timeline_roots - : snapshot -> page:Graph.page_uuid -> limit:int -> after:member_position option + : snapshot + -> page:Graph.page_uuid + -> limit:int + -> after:member_position option -> (Types.tree_member list * member_position option, Types.read_error) result + val get_timeline_children - : snapshot -> parent:Graph.block_uuid -> limit:int -> after:member_position option + : snapshot + -> parent:Graph.block_uuid + -> limit:int + -> after:member_position option -> (Types.child_member list * member_position option, Types.read_error) result (** {3 Example: reading one coherent projection} @@ -503,15 +537,28 @@ val commit_local mutation and preconditions. No original commit point is inferred from a legacy idempotency receipt that did not store one. *) type local_intent + type intent_commit -type intent_resolution = Intent_pending | Intent_committed of intent_commit + +type intent_resolution = + | Intent_pending + | Intent_committed of intent_commit | Intent_rejected of Types.local_commit_error val persist_local_intent - : t -> namespace:string -> payload:string -> expected:write_precondition - -> Types.local_mutation -> (local_intent, Types.local_commit_error) result -val find_local_intent : t -> namespace:string -> mutation_id:Graph.Uuid.t + : t + -> namespace:string + -> payload:string + -> expected:write_precondition + -> Types.local_mutation + -> (local_intent, Types.local_commit_error) result + +val find_local_intent + : t + -> namespace:string + -> mutation_id:Graph.Uuid.t -> (local_intent option, Types.local_commit_error) result + val local_intent_id : local_intent -> Graph.Uuid.t val local_intent_payload : local_intent -> string @@ -528,26 +575,42 @@ val local_intent_payload : local_intent -> string transaction. Retrying returns that same evidence, including after reopening; it never replaces the commit revision with a current read revision. *) val commit_local_intent - : t -> local_intent -> (intent_commit, Types.local_commit_error) result + : t + -> local_intent + -> (intent_commit, Types.local_commit_error) result + val inspect_local_intent - : t -> local_intent -> (intent_resolution, Types.local_commit_error) result + : t + -> local_intent + -> (intent_resolution, Types.local_commit_error) result + val intent_commit_outcome : intent_commit -> Types.local_commit val intent_commit_targets : intent_commit -> Graph.Uuid.t Rrbvec.t -type local_intent_page = private { items : local_intent list; next : int64 option } +type local_intent_page = private + { items : local_intent list + ; next : int64 option + } val pending_local_intents - : t -> namespace:string -> after:int64 option -> limit:int + : t + -> namespace:string + -> after:int64 option + -> limit:int -> (local_intent_page, Types.local_commit_error) result (** Acknowledges only an existing immutable known-not-applied rejection. Pending or committed intents cannot be discarded through this operation. *) val acknowledge_rejected_local_intent - : t -> local_intent -> (unit, Types.local_commit_error) result + : t + -> local_intent + -> (unit, Types.local_commit_error) result (** Retains the commit proof while removing this exact intent from recovery. *) val acknowledge_local_intent - : t -> intent_commit -> (unit, Types.local_commit_error) result + : t + -> intent_commit + -> (unit, Types.local_commit_error) result (** Requeues an eligible blocked mutation after validating a fresh read set. *) val retry_blocked @@ -633,6 +696,20 @@ val unprotection_ciphertexts (** Returns the current transport view and synchronization token. *) val inspect_sync : t -> (Types.sync_view, Types.sync_read_error) result +(** Waits for a different sync token, including ACK-only changes; cancellation releases the wait. *) +val await_sync_change + : t + -> after:Types.sync_token + -> (Types.sync_view, Types.sync_read_error) result + +(** Compares the computed authoritative digest without graph reads. + [None] means a different checkpoint. *) +val check_checkpoint_checksum + : t + -> through:Types.server_cursor + -> expected:Types.checksum + -> (bool option, Types.sync_read_error) result + (** {2 Sending the outbox} *) type prepared_outbox_transition diff --git a/logseq_overlay_db/test/test_overlay_storage.ml b/logseq_overlay_db/test/test_overlay_storage.ml index 1620cbe0..3424f6a5 100644 --- a/logseq_overlay_db/test/test_overlay_storage.ml +++ b/logseq_overlay_db/test/test_overlay_storage.ml @@ -5,11 +5,15 @@ let server_cursor_of_int value = module Database = Logseq_overlay_db.Database let retirement_sequence = Atomic.make 0 + let next_retirement () = Database.retirement_id_of_string - (Printf.sprintf "97000000-0000-4000-8000-%012d" (Atomic.fetch_and_add retirement_sequence 1)) + (Printf.sprintf + "97000000-0000-4000-8000-%012d" + (Atomic.fetch_and_add retirement_sequence 1)) |> Result.get_ok ;; + module T = Test_support module Types = Logseq_overlay_db.Types open Types @@ -34,7 +38,9 @@ let block_precondition database block behavior = ~finally:(fun () -> Database.release_snapshot snapshot) (fun () -> let revision = - match Database.get_outline_blocks snapshot [ block ] |> T.require_ok ~behavior with + match + Database.get_outline_blocks snapshot [ block ] |> T.require_ok ~behavior + with | [ Present_block { revision; _ } ] | [ Missing_block { revision; _ } ] -> revision | _ -> Alcotest.fail "block precondition returned the wrong cardinality" @@ -135,16 +141,24 @@ let rec supply_snapshot_crypto ~behavior prepared batch_count = supply_snapshot_crypto ~behavior prepared (batch_count + 1) ;; -let prepare_snapshot ?wire_batch_max_bytes ~behavior root support = +let prepare_snapshot ?wire_batch_max_bytes ?change_max_items ~behavior root support = let path, applied_server_cursor, expected_checksum, expected_rows = snapshot_input_for root in let dependencies, location, inspection = absent_inspection ~behavior support in let dependencies = - match wire_batch_max_bytes with - | None -> dependencies - | Some wire_batch_max_bytes -> - let limits = { (T.limits ~behavior) with wire_batch_max_bytes } in + match wire_batch_max_bytes, change_max_items with + | None, None -> dependencies + | _ -> + let defaults = T.limits ~behavior in + let limits = + { defaults with + wire_batch_max_bytes = + Option.value wire_batch_max_bytes ~default:defaults.wire_batch_max_bytes + ; change_max_items = + Option.value change_max_items ~default:defaults.change_max_items + } + in Database.dependencies ~epoch_ms:(fun () -> 1_704_067_200_000L) ~monotonic_ns:(fun () -> 1_000_000L) @@ -213,40 +227,78 @@ let snapshot_input_validation_is_fail_closed () = let snapshot_rejection_preserves_schema_versions () = let module Storage = Logseq_db_storage.Logseq_sqlite_storage in let module Session = Logseq_db_storage.Storage_session in - List.iter (fun (major, minor) -> - T.with_temp_directory "overlay-old-schema-" (fun root -> - let source = Filename.concat root "source" in - Unix.mkdir source 0o700; - let database_path = T.seed_mirror source in - let connection = Storage.open_database database_path |> T.require_ok ~behavior:"open synthetic schema" in - let db = Storage.restore_database connection |> T.require_ok ~behavior:"restore synthetic schema" in - let session = Session.create ~tail:(Datascript.Storage.restore_tail_groups (Storage.datascript_storage connection)) - ~callbacks:(Storage.connection_callbacks connection) in - Fun.protect ~finally:(fun () -> ignore (Session.close session)) (fun () -> - let staged = Session.stage_transact session ~authoritative_before:db - [Datascript.Add (Ident "logseq.kv/schema-version", "kv/value", - Map [Keyword "major", Int64 (Int64.of_int major); Keyword "minor", Int64 (Int64.of_int minor)])] - |> T.require_ok ~behavior:"stage old synthetic schema" in - Session.commit_staged session staged |> T.require_ok ~behavior:"commit old synthetic schema"); - let path = Filename.concat root "snapshot.transit" in - let expected_rows = T.write_snapshot_from_database ~database_path ~snapshot_path:path in - let support = Filename.concat root "target" in - Unix.mkdir support 0o700; - let dependencies, _, inspection = absent_inspection ~behavior:"old schema admission" support in - match Database.prepare_snapshot_activation dependencies inspection ~path - ~applied_server_cursor:(server_cursor 0) ~expected_checksum:None ~expected_rows with - | Error (Snapshot_parse_error message) -> - Alcotest.(check string) "version evidence preserved at admission boundary" + List.iter + (fun (major, minor) -> + T.with_temp_directory "overlay-old-schema-" (fun root -> + let source = Filename.concat root "source" in + Unix.mkdir source 0o700; + let database_path = T.seed_mirror source in + let connection = + Storage.open_database database_path + |> T.require_ok ~behavior:"open synthetic schema" + in + let db = + Storage.restore_database connection + |> T.require_ok ~behavior:"restore synthetic schema" + in + let session = + Session.create + ~tail: + (Datascript.Storage.restore_tail_groups + (Storage.datascript_storage connection)) + ~callbacks:(Storage.connection_callbacks connection) + in + Fun.protect + ~finally:(fun () -> ignore (Session.close session)) + (fun () -> + let staged = + Session.stage_transact + session + ~authoritative_before:db + [ Datascript.Add + ( Ident "logseq.kv/schema-version" + , "kv/value" + , Map + [ Keyword "major", Int64 (Int64.of_int major) + ; Keyword "minor", Int64 (Int64.of_int minor) + ] ) + ] + |> T.require_ok ~behavior:"stage old synthetic schema" + in + Session.commit_staged session staged + |> T.require_ok ~behavior:"commit old synthetic schema"); + let path = Filename.concat root "snapshot.transit" in + let expected_rows = + T.write_snapshot_from_database ~database_path ~snapshot_path:path + in + let support = Filename.concat root "target" in + Unix.mkdir support 0o700; + let dependencies, _, inspection = + absent_inspection ~behavior:"old schema admission" support + in + match + Database.prepare_snapshot_activation + dependencies + inspection + ~path + ~applied_server_cursor:(server_cursor 0) + ~expected_checksum:None + ~expected_rows + with + | Error (Snapshot_parse_error message) -> + Alcotest.(check string) + "version evidence preserved at admission boundary" (Printf.sprintf "Graph schema %d.%d is not supported. Journal requires schema 65.33 or \ later." major minor) - message - | Error _ -> Alcotest.fail "old schema returned a different error category" - | Ok prepared -> Database.cancel_snapshot_activation prepared; - Alcotest.fail "unsupported schema was admitted")) - [65, 29; 64, 200] + message + | Error _ -> Alcotest.fail "old schema returned a different error category" + | Ok prepared -> + Database.cancel_snapshot_activation prepared; + Alcotest.fail "unsupported schema was admitted")) + [ 65, 29; 64, 200 ] ;; let with_snapshot_metadata changes f = @@ -261,14 +313,20 @@ let with_snapshot_metadata changes f = let database = Storage.restore_database connection |> T.require_ok ~behavior in let session = Session.create - ~tail:(Datascript.Storage.restore_tail_groups (Storage.datascript_storage connection)) + ~tail: + (Datascript.Storage.restore_tail_groups (Storage.datascript_storage connection)) ~callbacks:(Storage.connection_callbacks connection) in - Fun.protect ~finally:(fun () -> ignore (Session.close session)) (fun () -> - if changes <> [] then ( - let staged = Session.stage_transact session ~authoritative_before:database changes - |> T.require_ok ~behavior in - Session.commit_staged session staged |> T.require_ok ~behavior)); + Fun.protect + ~finally:(fun () -> ignore (Session.close session)) + (fun () -> + if changes <> [] + then ( + let staged = + Session.stage_transact session ~authoritative_before:database changes + |> T.require_ok ~behavior + in + Session.commit_staged session staged |> T.require_ok ~behavior)); f root database_path) ;; @@ -278,13 +336,22 @@ let snapshot_uuid_metadata_case changes expected_error () = let behavior = "optional snapshot UUID metadata" in with_snapshot_metadata changes (fun root database_path -> let path = Filename.concat root "snapshot.transit" in - let expected_rows = T.write_snapshot_from_database ~database_path ~snapshot_path:path in + let expected_rows = + T.write_snapshot_from_database ~database_path ~snapshot_path:path + in let support = Filename.concat root "target" in Unix.mkdir support 0o700; let dependencies, _, inspection = absent_inspection ~behavior support in let cursor = server_cursor 42 in - let result = Database.prepare_snapshot_activation dependencies inspection ~path - ~applied_server_cursor:cursor ~expected_checksum:None ~expected_rows in + let result = + Database.prepare_snapshot_activation + dependencies + inspection + ~path + ~applied_server_cursor:cursor + ~expected_checksum:None + ~expected_rows + in match expected_error, result with | Some expected, Error (Snapshot_parse_error actual) -> Alcotest.(check string) "metadata rejection is preserved" expected actual; @@ -298,64 +365,122 @@ let snapshot_uuid_metadata_case changes expected_error () = | None, Error (Snapshot_parse_error message) -> Alcotest.fail message | None, Error _ -> Alcotest.fail "valid optional UUID metadata was rejected" | None, Ok prepared -> - Fun.protect ~finally:(fun () -> Database.cancel_snapshot_activation prepared) (fun () -> - ignore (supply_snapshot_crypto ~behavior prepared 0); - ignore (Database.commit_snapshot_activation prepared |> T.require_ok ~behavior)); + Fun.protect + ~finally:(fun () -> Database.cancel_snapshot_activation prepared) + (fun () -> + ignore (supply_snapshot_crypto ~behavior prepared 0); + ignore (Database.commit_snapshot_activation prepared |> T.require_ok ~behavior)); (* The fixture helper only exports kvs, so this checkpoint must have been created from the requested target and baseline, not copied from input. *) - let target_path = Filename.concat support - ("logseq-db-worker/synced-graphs/" ^ Logseq_db_types.Graph_types.Uuid.to_string T.graph_uuid ^ "/db.sqlite") in + let target_path = + Filename.concat + support + ("logseq-db-worker/synced-graphs/" + ^ Logseq_db_types.Graph_types.Uuid.to_string T.graph_uuid + ^ "/db.sqlite") + in let checkpoint = Checkpoint.read_path target_path |> T.require_ok ~behavior in - T.require (Logseq_db_types.Graph_types.Uuid.equal checkpoint.graph_id T.graph_uuid) + T.require + (Logseq_db_types.Graph_types.Uuid.equal checkpoint.graph_id T.graph_uuid) "checkpoint did not retain the download target"; - T.require (Server_cursor.equal checkpoint.applied_server_t cursor) + T.require + (Server_cursor.equal checkpoint.applied_server_t cursor) "checkpoint did not retain the baseline cursor"; let metadata_values path = let connection = Storage.open_database path |> T.require_ok ~behavior in - Fun.protect ~finally:(fun () -> ignore (Storage.close (Storage.connection_callbacks connection))) (fun () -> - let database = Storage.restore_database connection |> T.require_ok ~behavior in - List.map (fun ident -> - Datascript.datoms database Datascript.Avet ~a:"db/ident" - ~v:(Datascript.Keyword ident) () - |> Seq.flat_map (fun datom -> - Datascript.datoms database Datascript.Eavt ~e:datom.Datascript.e - ~a:"kv/value" ()) - |> Seq.map (fun datom -> datom.Datascript.v) - |> List.of_seq) - ["logseq.kv/local-graph-uuid"; "logseq.kv/graph-uuid"]) + Fun.protect + ~finally:(fun () -> + ignore (Storage.close (Storage.connection_callbacks connection))) + (fun () -> + let database = + Storage.restore_database connection |> T.require_ok ~behavior + in + List.map + (fun ident -> + Datascript.datoms + database + Datascript.Avet + ~a:"db/ident" + ~v:(Datascript.Keyword ident) + () + |> Seq.flat_map (fun datom -> + Datascript.datoms + database + Datascript.Eavt + ~e:datom.Datascript.e + ~a:"kv/value" + ()) + |> Seq.map (fun datom -> datom.Datascript.v) + |> List.of_seq) + [ "logseq.kv/local-graph-uuid"; "logseq.kv/graph-uuid" ]) in - Alcotest.(check bool) "activation does not invent UUID metadata" true + Alcotest.(check bool) + "activation does not invent UUID metadata" + true (metadata_values database_path = metadata_values target_path); - Eio_main.run (fun _ -> Eio.Switch.run (fun sw -> - let database = Database.open_ ~sw dependencies (inspect_mirror support behavior) - ~graph_name:"optional-uuid-graph" |> T.require_ok ~behavior in - Fun.protect ~finally:(fun () -> ignore (Database.close database)) (fun () -> - let snapshot = Database.current_snapshot database |> T.require_ok ~behavior in - Fun.protect ~finally:(fun () -> Database.release_snapshot snapshot) (fun () -> - let info = Database.graph_info snapshot |> T.require_ok ~behavior in - T.require (Logseq_db_types.Graph_types.Uuid.equal info.graph_uuid T.graph_uuid) - "reopened mirror lost target identity"; - match Database.get_outline_blocks snapshot [T.authoritative_block_uuid] - |> T.require_ok ~behavior with - | [Present_block _] -> () - | _ -> Alcotest.fail "reopened snapshot lost graph data"))))) + Eio_main.run (fun _ -> + Eio.Switch.run (fun sw -> + let database = + Database.open_ + ~sw + dependencies + (inspect_mirror support behavior) + ~graph_name:"optional-uuid-graph" + |> T.require_ok ~behavior + in + Fun.protect + ~finally:(fun () -> ignore (Database.close database)) + (fun () -> + let snapshot = + Database.current_snapshot database |> T.require_ok ~behavior + in + Fun.protect + ~finally:(fun () -> Database.release_snapshot snapshot) + (fun () -> + let info = Database.graph_info snapshot |> T.require_ok ~behavior in + T.require + (Logseq_db_types.Graph_types.Uuid.equal + info.graph_uuid + T.graph_uuid) + "reopened mirror lost target identity"; + match + Database.get_outline_blocks snapshot [ T.authoritative_block_uuid ] + |> T.require_ok ~behavior + with + | [ Present_block _ ] -> () + | _ -> Alcotest.fail "reopened snapshot lost graph data"))))) ;; let local_admission_keeps_uuid_requirement missing () = let module Storage = Logseq_db_storage.Logseq_sqlite_storage in let behavior = "local graph admission remains unchanged" in - let changes = Datascript.[RetractEntity (Ident "logseq.kv/graph-uuid"); - Add (Ident "logseq.kv/graph-remote?", "kv/value", Bool false)] - @ (if missing then [Datascript.RetractEntity (Ident "logseq.kv/local-graph-uuid")] else []) in + let changes = + Datascript. + [ RetractEntity (Ident "logseq.kv/graph-uuid") + ; Add (Ident "logseq.kv/graph-remote?", "kv/value", Bool false) + ] + @ + if missing + then [ Datascript.RetractEntity (Ident "logseq.kv/local-graph-uuid") ] + else [] + in with_snapshot_metadata changes (fun _ database_path -> let connection = Storage.open_database database_path |> T.require_ok ~behavior in - Fun.protect ~finally:(fun () -> ignore (Storage.close (Storage.connection_callbacks connection))) (fun () -> - let database = Storage.restore_database connection |> T.require_ok ~behavior in - let startup = Storage.startup_metadata connection |> T.require_ok ~behavior in - match Logseq_db_storage.Admission.inspect ~target:Local_target ~db:database - ~storage_schema:startup.schema, missing with - | Error Corrupt_storage, true | Ok _, false -> () - | _ -> Alcotest.fail "local graph UUID requirement changed")) + Fun.protect + ~finally:(fun () -> + ignore (Storage.close (Storage.connection_callbacks connection))) + (fun () -> + let database = Storage.restore_database connection |> T.require_ok ~behavior in + let startup = Storage.startup_metadata connection |> T.require_ok ~behavior in + match + ( Logseq_db_storage.Admission.inspect + ~target:Local_target + ~db:database + ~storage_schema:startup.schema + , missing ) + with + | Error Corrupt_storage, true | Ok _, false -> () + | _ -> Alcotest.fail "local graph UUID requirement changed")) ;; let snapshot_uuid_metadata_cases = @@ -364,29 +489,48 @@ let snapshot_uuid_metadata_cases = let set ident value = Add (Ident ("logseq.kv/" ^ ident), "kv/value", value) in let corrupt = Some "The downloaded graph storage is corrupt." in let identity = Some "The downloaded graph has an invalid synchronization identity." in - List.map (fun (name, changes, expected) -> - Alcotest.test_case name `Quick (snapshot_uuid_metadata_case changes expected)) + List.map + (fun (name, changes, expected) -> + Alcotest.test_case name `Quick (snapshot_uuid_metadata_case changes expected)) [ "snapshot UUIDs present", [], None - ; "snapshot remote UUID absent", [remove "graph-uuid"], None - ; "snapshot local UUID absent", [remove "local-graph-uuid"], None - ; "snapshot both UUIDs absent", [remove "graph-uuid"; remove "local-graph-uuid"], None - ; "snapshot present remote UUID conflict", [set "graph-uuid" (Uuid "70000000-0000-4000-8000-000000000001")], identity - ; "snapshot absent local UUID retains remote conflict rejection", [remove "local-graph-uuid"; set "graph-uuid" (Uuid "70000000-0000-4000-8000-000000000001")], identity - ; "snapshot malformed remote UUID", [set "graph-uuid" (String "invalid-uuid")], corrupt - ; "snapshot malformed local UUID", [set "local-graph-uuid" (String "invalid-uuid")], corrupt - ; "snapshot remote UUID wrong type", [set "graph-uuid" (Bool false)], corrupt - ; "snapshot local UUID wrong type", [set "local-graph-uuid" (Bool false)], corrupt - ; "snapshot missing db type", [remove "db-type"], corrupt - ; "snapshot invalid db type", [set "db-type" (String "file")], corrupt - ; "snapshot missing schema version", [remove "schema-version"], corrupt - ; "snapshot invalid schema version", [set "schema-version" (String "65.33")], corrupt - ; "snapshot missing remote flag", [remove "graph-remote?"], identity - ; "snapshot false remote flag", [set "graph-remote?" (Bool false)], identity - ; "snapshot malformed remote flag", [set "graph-remote?" (String "true")], identity + ; "snapshot remote UUID absent", [ remove "graph-uuid" ], None + ; "snapshot local UUID absent", [ remove "local-graph-uuid" ], None + ; ( "snapshot both UUIDs absent" + , [ remove "graph-uuid"; remove "local-graph-uuid" ] + , None ) + ; ( "snapshot present remote UUID conflict" + , [ set "graph-uuid" (Uuid "70000000-0000-4000-8000-000000000001") ] + , identity ) + ; ( "snapshot absent local UUID retains remote conflict rejection" + , [ remove "local-graph-uuid" + ; set "graph-uuid" (Uuid "70000000-0000-4000-8000-000000000001") + ] + , identity ) + ; ( "snapshot malformed remote UUID" + , [ set "graph-uuid" (String "invalid-uuid") ] + , corrupt ) + ; ( "snapshot malformed local UUID" + , [ set "local-graph-uuid" (String "invalid-uuid") ] + , corrupt ) + ; "snapshot remote UUID wrong type", [ set "graph-uuid" (Bool false) ], corrupt + ; "snapshot local UUID wrong type", [ set "local-graph-uuid" (Bool false) ], corrupt + ; "snapshot missing db type", [ remove "db-type" ], corrupt + ; "snapshot invalid db type", [ set "db-type" (String "file") ], corrupt + ; "snapshot missing schema version", [ remove "schema-version" ], corrupt + ; ( "snapshot invalid schema version" + , [ set "schema-version" (String "65.33") ] + , corrupt ) + ; "snapshot missing remote flag", [ remove "graph-remote?" ], identity + ; "snapshot false remote flag", [ set "graph-remote?" (Bool false) ], identity + ; "snapshot malformed remote flag", [ set "graph-remote?" (String "true") ], identity ] - @ [ Alcotest.test_case "local admission still requires local UUID" `Quick + @ [ Alcotest.test_case + "local admission still requires local UUID" + `Quick (local_admission_keeps_uuid_requirement true) - ; Alcotest.test_case "local admission accepts existing local UUID" `Quick + ; Alcotest.test_case + "local admission accepts existing local UUID" + `Quick (local_admission_keeps_uuid_requirement false) ] ;; @@ -395,23 +539,43 @@ let snapshot_import_error_does_not_disclose_payload () = T.with_temp_directory "overlay-invalid-payload-" (fun root -> let support = Filename.concat root "target" in Unix.mkdir support 0o700; - let dependencies, _, inspection = absent_inspection ~behavior:"safe import rejection" support in + let dependencies, _, inspection = + absent_inspection ~behavior:"safe import rejection" support + in let path = Filename.concat root "snapshot.transit" in let payload = {|{"SENSITIVE-synthetic-body":invalid}|} in let header = Bytes.create 4 in Bytes.set_int32_be header 0 (Int32.of_int (String.length payload)); Out_channel.with_open_bin path (fun channel -> - output_bytes channel header; output_string channel payload); - match Database.prepare_snapshot_activation dependencies inspection ~path - ~applied_server_cursor:(server_cursor 0) ~expected_checksum:None ~expected_rows:1 with + output_bytes channel header; + output_string channel payload); + match + Database.prepare_snapshot_activation + dependencies + inspection + ~path + ~applied_server_cursor:(server_cursor 0) + ~expected_checksum:None + ~expected_rows:1 + with | Error (Snapshot_parse_error message) -> - let module D=Logseq_db_types.Failure_diagnostic in - let diagnostic=match D.decode message with Some d->d | None->Alcotest.fail "snapshot parser reason was discarded" in - Alcotest.(check string) "safe parser category survives" "parseFailed" (D.reason diagnostic); - Alcotest.(check bool) "raw parser body is excluded" false + let module D = Logseq_db_types.Failure_diagnostic in + let diagnostic = + match D.decode message with + | Some d -> d + | None -> Alcotest.fail "snapshot parser reason was discarded" + in + Alcotest.(check string) + "safe parser category survives" + "parseFailed" + (D.reason diagnostic); + Alcotest.(check bool) + "raw parser body is excluded" + false (String.contains (D.message diagnostic) '{') | Error _ -> Alcotest.fail "malformed snapshot returned another category" - | Ok prepared -> Database.cancel_snapshot_activation prepared; + | Ok prepared -> + Database.cancel_snapshot_activation prepared; Alcotest.fail "malformed snapshot was admitted") ;; @@ -455,6 +619,80 @@ let snapshot_activation_installs_queryable_mirror () = "activated mirror was not queryable"))) ;; +let snapshot_crypto_batches_respect_item_capacity () = + let behavior = "snapshot crypto batches respect item capacity" in + T.with_temp_directory "overlay-snapshot-item-bound-" (fun root -> + let support = Filename.concat root "target" in + Unix.mkdir support 0o700; + let _, _, prepared = prepare_snapshot ~change_max_items:1 ~behavior root support in + Fun.protect + ~finally:(fun () -> Database.cancel_snapshot_activation prepared) + (fun () -> + match + Database.next_snapshot_unprotection_batch prepared |> T.require_ok ~behavior + with + | None -> Alcotest.fail "protected snapshot omitted crypto" + | Some request -> + Alcotest.(check int) + "snapshot item limit" + 1 + (List.length (Database.unprotection_ciphertexts request)))) +;; + +let snapshot_crypto_batches_tighten_caller_bounds () = + let behavior = "snapshot crypto batches tighten caller bounds" in + T.with_temp_directory "overlay-snapshot-caller-bound-" (fun root -> + let support = Filename.concat root "target" in + Unix.mkdir support 0o700; + let _, _, prepared = prepare_snapshot ~behavior root support in + Fun.protect + ~finally:(fun () -> Database.cancel_snapshot_activation prepared) + (fun () -> + List.iter + (fun (maximum_items, maximum_bytes) -> + match + Database.next_snapshot_unprotection_batch + ~maximum_items + ~maximum_bytes + prepared + with + | Error Snapshot_limit_exceeded -> () + | _ -> + Alcotest.fail "invalid caller bound must reject without consuming a batch") + [ 0, 128; -1, 128; 2, 0; 2, -1 ]; + let rec supply total batches = + match + Database.next_snapshot_unprotection_batch + ~maximum_items:2 + ~maximum_bytes:128 + prepared + |> T.require_ok ~behavior + with + | None -> total, batches + | Some request -> + let items = Database.unprotection_ciphertexts request in + T.require + (items <> [] && List.length items <= 2) + "caller item bound exceeded"; + T.require + (List.fold_left + (fun bytes (_, value) -> bytes + String.length value) + 0 + items + <= 128) + "caller byte bound exceeded"; + Database.supply_snapshot_unprotection_batch + prepared + ~request + ~plaintexts:items + |> T.require_ok ~behavior; + supply (total + List.length items) (batches + 1) + in + let total, batches = supply 0 0 in + T.require (total > 2 && batches > 1) "fixture did not exercise multiple batches"; + ignore (Database.commit_snapshot_activation prepared |> T.require_ok ~behavior))) +;; + let canceled_snapshot_activation_removes_temporary_artifacts () = let behavior = "canceled snapshot activation removes temporary artifacts" in T.with_temp_directory "overlay-snapshot-cancel-" (fun root -> @@ -497,29 +735,50 @@ let snapshot_state_errors_keep_their_reason () = let support = Filename.concat root "target" in Unix.mkdir support 0o700; let _, _, prepared = prepare_snapshot ~behavior root support in - Fun.protect ~finally:(fun () -> Database.cancel_snapshot_activation prepared) (fun () -> - let request = Database.next_snapshot_unprotection_batch prepared - |> T.require_ok ~behavior |> Option.get in - expect Snapshot_crypto_batch_pending (Database.next_snapshot_unprotection_batch prepared); - expect Snapshot_persist_awaiting_crypto (Database.commit_snapshot_activation prepared); - let plaintexts = Database.unprotection_ciphertexts request in - Database.supply_snapshot_unprotection_batch prepared ~request ~plaintexts - |> T.require_ok ~behavior; - ignore (supply_snapshot_crypto ~behavior prepared 0); - ignore (Database.commit_snapshot_activation prepared |> T.require_ok ~behavior); - expect Snapshot_next_after_persist_or_commit (Database.next_snapshot_unprotection_batch prepared); - expect Snapshot_supply_after_persist_or_commit - (Database.supply_snapshot_unprotection_batch prepared ~request ~plaintexts))); + Fun.protect + ~finally:(fun () -> Database.cancel_snapshot_activation prepared) + (fun () -> + let request = + Database.next_snapshot_unprotection_batch prepared + |> T.require_ok ~behavior + |> Option.get + in + expect + Snapshot_crypto_batch_pending + (Database.next_snapshot_unprotection_batch prepared); + expect + Snapshot_persist_awaiting_crypto + (Database.commit_snapshot_activation prepared); + let plaintexts = Database.unprotection_ciphertexts request in + Database.supply_snapshot_unprotection_batch prepared ~request ~plaintexts + |> T.require_ok ~behavior; + ignore (supply_snapshot_crypto ~behavior prepared 0); + ignore (Database.commit_snapshot_activation prepared |> T.require_ok ~behavior); + expect + Snapshot_next_after_persist_or_commit + (Database.next_snapshot_unprotection_batch prepared); + expect + Snapshot_supply_after_persist_or_commit + (Database.supply_snapshot_unprotection_batch prepared ~request ~plaintexts))); T.with_temp_directory "overlay-snapshot-root-reason-" (fun root -> let support = Filename.concat root "target" in Unix.mkdir support 0o700; let dependencies, _, inspection = absent_inspection ~behavior support in - let path, applied_server_cursor, expected_checksum, expected_rows = snapshot_input_for root in + let path, applied_server_cursor, expected_checksum, expected_rows = + snapshot_input_for root + in let blocked = Filename.concat support "logseq-db-worker" in - let channel = open_out_bin blocked in close_out channel; - expect Snapshot_root_unavailable - (Database.prepare_snapshot_activation dependencies inspection ~path ~applied_server_cursor - ~expected_checksum ~expected_rows)) + let channel = open_out_bin blocked in + close_out channel; + expect + Snapshot_root_unavailable + (Database.prepare_snapshot_activation + dependencies + inspection + ~path + ~applied_server_cursor + ~expected_checksum + ~expected_rows)) ;; let snapshot_crypto_results_are_validated_before_staging () = @@ -792,6 +1051,131 @@ let checksum_vectors_preserve_snapshot_semantics () = ] ;; +let incremental_checksum_matches_snapshot_verification () = + let behavior = "incremental checksum matches snapshot verification" in + List.iter + (fun e2ee -> + T.with_temp_directory "overlay-checksum-delta-" (fun root -> + let support = Filename.concat root "live" in + Unix.mkdir support 0o700; + let database_path = T.seed_mirror support in + if e2ee then T.mark_snapshot_e2ee database_path; + let dependencies, _, inspection = location_and_inspection support behavior in + Eio_main.run (fun _ -> + Eio.Switch.run (fun sw -> + let database = + Database.open_ ~sw dependencies inspection ~graph_name:"checksum-delta" + |> T.require_ok ~behavior + in + let module J = Transit_core.Json in + let lookup uuid = + J.Array [ J.Keyword "block/uuid"; J.Uuid (Graph.Uuid.to_string uuid) ] + in + let block = lookup T.authoritative_block_uuid in + let renamed_page = T.mutation_uuid 991 in + let orphan = T.mutation_uuid 992 in + let tag = T.mutation_uuid 993 in + let add entity attr value = + J.Array [ J.Keyword "db/add"; entity; J.Keyword attr; value ] + in + let retract entity attr value = + J.Array [ J.Keyword "db/retract"; entity; J.Keyword attr; value ] + in + let steps = + [ [ add block "block/title" (J.String "checksum delta 🚀") + ; add block "block/order" (J.String "a9") + ] + ; [ add + (lookup T.page_uuid) + "block/uuid" + (J.Uuid (Graph.Uuid.to_string renamed_page)) + ] + ; [ add (J.Int (-1)) "block/uuid" (J.Uuid (Graph.Uuid.to_string tag)) + ; add (J.Int (-1)) "db/ident" (J.Keyword "checksum/temporary-tag") + ; add (J.Int (-2)) "block/uuid" (J.Uuid (Graph.Uuid.to_string orphan)) + ; add + (J.Int (-2)) + "block/tags" + (J.Array [ J.Keyword "db/ident"; J.Keyword "logseq.class/Page" ]) + ] + ; [ retract + (lookup orphan) + "block/tags" + (J.Array [ J.Keyword "db/ident"; J.Keyword "logseq.class/Page" ]) + ; add (lookup tag) "db/ident" (J.Keyword "checksum/not-a-page") + ] + ; [ add block "logseq.property/built-in?" (J.Bool true) ] + ; [ add block "logseq.property/built-in?" (J.Bool false) ] + ; [ retract block "block/order" (J.String "a9") ] + ; [ J.Array [ J.Keyword "db/retractEntity"; lookup orphan ] ] + ; [ J.Array [ J.Keyword "db/retractEntity"; block ] ] + ] + in + let verify cursor = + let target = Filename.concat root (Printf.sprintf "verified-%d" cursor) in + Unix.mkdir target 0o700; + let path = Filename.concat target "snapshot.transit" in + let expected_rows = + T.write_snapshot_from_database ~database_path ~snapshot_path:path + in + let dependencies, _, inspection = absent_inspection ~behavior target in + let prepared = + Database.prepare_snapshot_activation + dependencies + inspection + ~path + ~applied_server_cursor:(server_cursor cursor) + ~expected_checksum:None + ~expected_rows + |> T.require_ok ~behavior + in + ignore (supply_snapshot_crypto ~behavior prepared 0); + Database.commit_snapshot_activation prepared + |> T.require_ok ~behavior + |> ignore; + let expected = + match Database.mirror_presence (inspect_mirror target behavior) with + | Available { checksum = Some checksum; _ } -> checksum + | _ -> Alcotest.fail "snapshot verification omitted computed digest" + in + T.require + (Database.check_checkpoint_checksum + database + ~through:(server_cursor cursor) + ~expected + |> T.require_ok ~behavior + = Some true) + "incremental digest differs from full snapshot verification"; + expected + in + ignore (verify 0); + let expected = ref None in + List.iteri + (fun index operations -> + T.apply_test_transaction database ~cursor:(index + 1) operations; + expected := Some (verify (index + 1))) + steps; + Database.close database |> T.require_ok ~behavior; + let reopened = + Database.open_ + ~sw + dependencies + (inspect_mirror support behavior) + ~graph_name:"checksum-delta" + |> T.require_ok ~behavior + in + T.require + (Database.check_checkpoint_checksum + reopened + ~through:(server_cursor (List.length steps)) + ~expected:(Option.get !expected) + |> T.require_ok ~behavior + = Some true) + "restored digest differs from committed incremental digest"; + Database.close reopened |> T.require_ok ~behavior)))) + [ false; true ] +;; + let unused_preparation_checksum_defers_invalid_utf8_without_publication () = let behavior = "unused preparation checksum defers invalid UTF-8" in T.with_temp_directory "overlay-checksum-late-failure-" (fun root -> @@ -940,15 +1324,25 @@ let stale_delete_cannot_remove_replacement () = T.with_temp_directory "overlay-stale-delete-" (fun support -> let database_path = T.seed_mirror support in let directory = Filename.dirname database_path in - let prepared, _ = Database.prepare_mirror_deletion ~application_support_directory:support - ~graph_id:T.graph_uuid ~retirement:(next_retirement ()) |> T.require_ok ~behavior in - Fun.protect ~finally:(fun () -> Database.cancel_mirror_deletion prepared) (fun () -> - Unix.rename directory (Filename.concat support "captured-original"); - ignore (T.seed_mirror support); - (match Database.commit_mirror_deletion prepared with - | Error Mirror_delete_stale -> () - | _ -> Alcotest.fail "a prepared deletion removed its replacement"); - Alcotest.(check bool) "replacement remains present" true (Sys.file_exists database_path))) + let prepared, _ = + Database.prepare_mirror_deletion + ~application_support_directory:support + ~graph_id:T.graph_uuid + ~retirement:(next_retirement ()) + |> T.require_ok ~behavior + in + Fun.protect + ~finally:(fun () -> Database.cancel_mirror_deletion prepared) + (fun () -> + Unix.rename directory (Filename.concat support "captured-original"); + ignore (T.seed_mirror support); + (match Database.commit_mirror_deletion prepared with + | Error Mirror_delete_stale -> () + | _ -> Alcotest.fail "a prepared deletion removed its replacement"); + Alcotest.(check bool) + "replacement remains present" + true + (Sys.file_exists database_path))) ;; (* No pure reducer owns the filesystem rename, purge, or stderr capability. @@ -960,43 +1354,82 @@ let retired_cleanup_retries_without_touching_a_new_mirror ?(closed_stderr = fals let blocked_name = "synthetic-cleanup-blocked" in let blocked = Filename.concat (Filename.dirname database_path) blocked_name in Unix.mkdir blocked 0o700; - Out_channel.with_open_bin (Filename.concat blocked "pending") (fun out -> output_string out "synthetic"); + Out_channel.with_open_bin (Filename.concat blocked "pending") (fun out -> + output_string out "synthetic"); let rec blocked_paths path = match Unix.lstat path with | { Unix.st_kind = Unix.S_DIR; _ } -> - let children = Sys.readdir path |> Array.to_list - |> List.concat_map (fun name -> blocked_paths (Filename.concat path name)) in + let children = + Sys.readdir path + |> Array.to_list + |> List.concat_map (fun name -> blocked_paths (Filename.concat path name)) + in if Filename.basename path = blocked_name then path :: children else children | _ -> [] - | exception Unix.Unix_error (Unix.ENOENT, _, _) -> [] in - let unblock () = List.iter (fun path -> Unix.chmod path 0o700) (blocked_paths support) in + | exception Unix.Unix_error (Unix.ENOENT, _, _) -> [] + in + let unblock () = + List.iter (fun path -> Unix.chmod path 0o700) (blocked_paths support) + in Fun.protect ~finally:unblock (fun () -> let dependencies = T.dependencies ~behavior in Unix.chmod blocked 0o500; let commit () = - match Database.prepare_mirror_deletion ~application_support_directory:support ~graph_id:T.graph_uuid ~retirement:(next_retirement ()) with + match + Database.prepare_mirror_deletion + ~application_support_directory:support + ~graph_id:T.graph_uuid + ~retirement:(next_retirement ()) + with | Error _ as error -> error - | Ok (preparation, _) -> Fun.protect ~finally:(fun () -> Database.cancel_mirror_deletion preparation) - (fun () -> Database.commit_mirror_deletion preparation) in - let result = if not closed_stderr then commit () else ( - flush stderr; - let saved = Unix.dup Unix.stderr in - let unreadable = Unix.openfile "/dev/null" [Unix.O_RDONLY] 0 in - Fun.protect ~finally:(fun () -> - Unix.dup2 saved Unix.stderr; Unix.close saved; Unix.close unreadable) - (fun () -> Unix.dup2 unreadable Unix.stderr; commit ())) in + | Ok (preparation, _) -> + Fun.protect + ~finally:(fun () -> Database.cancel_mirror_deletion preparation) + (fun () -> Database.commit_mirror_deletion preparation) + in + let result = + if not closed_stderr + then commit () + else ( + flush stderr; + let saved = Unix.dup Unix.stderr in + let unreadable = Unix.openfile "/dev/null" [ Unix.O_RDONLY ] 0 in + Fun.protect + ~finally:(fun () -> + Unix.dup2 saved Unix.stderr; + Unix.close saved; + Unix.close unreadable) + (fun () -> + Unix.dup2 unreadable Unix.stderr; + commit ())) + in ignore (result |> T.require_ok ~behavior); - Alcotest.(check bool) "canonical copy was committed absent" false (Sys.file_exists database_path); - Alcotest.(check bool) "failed purge remains durably retryable" true (blocked_paths support <> []); + Alcotest.(check bool) + "canonical copy was committed absent" + false + (Sys.file_exists database_path); + Alcotest.(check bool) + "failed purge remains durably retryable" + true + (blocked_paths support <> []); ignore (T.seed_mirror support); unblock (); let fresh = inspect_mirror support behavior in - Eio_main.run (fun _ -> Eio.Switch.run (fun sw -> - let database = Database.open_ ~sw dependencies fresh ~graph_name:"New synthetic copy" - |> T.require_ok ~behavior in - ignore (Database.close database |> T.require_ok ~behavior))); - Alcotest.(check bool) "new live copy survives retired cleanup" true (Sys.file_exists database_path); - Alcotest.(check bool) "ordinary producer lifecycle retries retired purge" true (blocked_paths support = []))) + Eio_main.run (fun _ -> + Eio.Switch.run (fun sw -> + let database = + Database.open_ ~sw dependencies fresh ~graph_name:"New synthetic copy" + |> T.require_ok ~behavior + in + ignore (Database.close database |> T.require_ok ~behavior))); + Alcotest.(check bool) + "new live copy survives retired cleanup" + true + (Sys.file_exists database_path); + Alcotest.(check bool) + "ordinary producer lifecycle retries retired purge" + true + (blocked_paths support = []))) ;; let mirror_preparation_cancel_and_commit_replay () = @@ -1005,39 +1438,78 @@ let mirror_preparation_cancel_and_commit_replay () = let database_path = T.seed_mirror support in let dependencies, _, inspection = location_and_inspection support behavior in let retirement = next_retirement () in - let prepared, _ = Database.prepare_mirror_deletion ~application_support_directory:support ~graph_id:T.graph_uuid ~retirement |> T.require_ok ~behavior in - Eio_main.run (fun _ -> Eio.Switch.run (fun sw -> - match Database.open_ ~sw dependencies inspection ~graph_name:"Busy synthetic mirror" with - | Error Ownership_conflict -> () - | _ -> Alcotest.fail "preparation did not hold exclusive ownership")); - ignore(Database.cancel_mirror_deletion_checked prepared |> T.require_ok ~behavior); - ignore(Database.cancel_mirror_deletion_checked prepared |> T.require_ok ~behavior); - Alcotest.(check bool) "cancel retains canonical mirror" true (Sys.file_exists database_path); - Alcotest.(check bool) "cancelled preparation cannot commit" true + let prepared, _ = + Database.prepare_mirror_deletion + ~application_support_directory:support + ~graph_id:T.graph_uuid + ~retirement + |> T.require_ok ~behavior + in + Eio_main.run (fun _ -> + Eio.Switch.run (fun sw -> + match + Database.open_ ~sw dependencies inspection ~graph_name:"Busy synthetic mirror" + with + | Error Ownership_conflict -> () + | _ -> Alcotest.fail "preparation did not hold exclusive ownership")); + ignore (Database.cancel_mirror_deletion_checked prepared |> T.require_ok ~behavior); + ignore (Database.cancel_mirror_deletion_checked prepared |> T.require_ok ~behavior); + Alcotest.(check bool) + "cancel retains canonical mirror" + true + (Sys.file_exists database_path); + Alcotest.(check bool) + "cancelled preparation cannot commit" + true (Result.is_error (Database.commit_mirror_deletion prepared)); let committed_retirement = next_retirement () in - let prepared, _ = Database.prepare_mirror_deletion ~application_support_directory:support ~graph_id:T.graph_uuid ~retirement:committed_retirement |> T.require_ok ~behavior in + let prepared, _ = + Database.prepare_mirror_deletion + ~application_support_directory:support + ~graph_id:T.graph_uuid + ~retirement:committed_retirement + |> T.require_ok ~behavior + in let committed_id = prepared in ignore (Database.commit_mirror_deletion prepared |> T.require_ok ~behavior); Database.cancel_mirror_deletion prepared; - Alcotest.(check bool) "commit has durable evidence" true (Database.mirror_deletion_committed prepared); + Alcotest.(check bool) + "commit has durable evidence" + true + (Database.mirror_deletion_committed prepared); ignore (T.seed_mirror support); - let replay, _ = Database.prepare_mirror_deletion ~application_support_directory:support ~graph_id:T.graph_uuid ~retirement:committed_retirement |> T.require_ok ~behavior in - Alcotest.(check bool) "fresh handle recovers durable commit" true (Database.mirror_deletion_committed replay); + let replay, _ = + Database.prepare_mirror_deletion + ~application_support_directory:support + ~graph_id:T.graph_uuid + ~retirement:committed_retirement + |> T.require_ok ~behavior + in + Alcotest.(check bool) + "fresh handle recovers durable commit" + true + (Database.mirror_deletion_committed replay); ignore (Database.commit_mirror_deletion replay |> T.require_ok ~behavior); Database.cancel_mirror_deletion replay; ignore (Database.commit_mirror_deletion committed_id |> T.require_ok ~behavior); Database.cancel_mirror_deletion committed_id; - Alcotest.(check bool) "late commit/cancel preserves new canonical mirror" true (Sys.file_exists database_path)) + Alcotest.(check bool) + "late commit/cancel preserves new canonical mirror" + true + (Sys.file_exists database_path)) ;; let deletion_does_not_follow_external_paths kind () = let behavior = "local deletion stays inside the selected physical graph" in T.with_temp_directory "overlay-delete-paths-" (fun root -> - let support = Filename.concat root "selected" and outside = Filename.concat root "outside" in - Unix.mkdir support 0o700; Unix.mkdir outside 0o700; - let database_path = T.seed_mirror support and external_database = T.seed_mirror outside in - let directory = Filename.dirname database_path and external_directory = Filename.dirname external_database in + let support = Filename.concat root "selected" + and outside = Filename.concat root "outside" in + Unix.mkdir support 0o700; + Unix.mkdir outside 0o700; + let database_path = T.seed_mirror support + and external_database = T.seed_mirror outside in + let directory = Filename.dirname database_path + and external_directory = Filename.dirname external_database in let mirrors = Filename.dirname directory in (match kind with | `Graph -> @@ -1047,20 +1519,46 @@ let deletion_does_not_follow_external_paths kind () = Unix.rename mirrors (Filename.concat root "original-mirrors"); Unix.symlink (Filename.dirname external_directory) mirrors | `Retirement -> Unix.symlink outside (Filename.concat mirrors ".mirror-deletions") - | `Lease -> Unix.symlink external_database (Filename.concat directory ".logseq-overlay-owner.sqlite") - | `Contained -> Unix.symlink external_directory (Filename.concat directory "attachment-link")); - let prepared = Database.prepare_mirror_deletion ~application_support_directory:support ~graph_id:T.graph_uuid ~retirement:(next_retirement ()) in - if kind = `Contained then ( + | `Lease -> + Unix.symlink + external_database + (Filename.concat directory ".logseq-overlay-owner.sqlite") + | `Contained -> + Unix.symlink external_directory (Filename.concat directory "attachment-link")); + let prepared = + Database.prepare_mirror_deletion + ~application_support_directory:support + ~graph_id:T.graph_uuid + ~retirement:(next_retirement ()) + in + if kind = `Contained + then ( let prepared, _ = prepared |> T.require_ok ~behavior in - Fun.protect ~finally:(fun () -> Database.cancel_mirror_deletion prepared) (fun () -> - ignore (Database.commit_mirror_deletion prepared |> T.require_ok ~behavior)); - Alcotest.(check bool) "selected directory was removed" false (Sys.file_exists directory)) + Fun.protect + ~finally:(fun () -> Database.cancel_mirror_deletion prepared) + (fun () -> + ignore (Database.commit_mirror_deletion prepared |> T.require_ok ~behavior)); + Alcotest.(check bool) + "selected directory was removed" + false + (Sys.file_exists directory)) else ( - Option.iter (fun (preparation, _) -> Database.cancel_mirror_deletion preparation) (Result.to_option prepared); - Alcotest.(check bool) "unsafe path is rejected before mutation" true (Result.is_error prepared)); - Alcotest.(check bool) "external target database survives" true (Sys.file_exists external_database); - Alcotest.(check bool) "external target was never leased" false - (Sys.file_exists (Filename.concat external_directory ".logseq-overlay-owner.sqlite"))) + Option.iter + (fun (preparation, _) -> Database.cancel_mirror_deletion preparation) + (Result.to_option prepared); + Alcotest.(check bool) + "unsafe path is rejected before mutation" + true + (Result.is_error prepared)); + Alcotest.(check bool) + "external target database survives" + true + (Sys.file_exists external_database); + Alcotest.(check bool) + "external target was never leased" + false + (Sys.file_exists + (Filename.concat external_directory ".logseq-overlay-owner.sqlite"))) ;; let legacy_retirement_replays_without_content_load mode () = @@ -1068,66 +1566,129 @@ let legacy_retirement_replays_without_content_load mode () = T.with_temp_directory "overlay-legacy-retirement-" (fun support -> let database_path = T.seed_mirror support in let directory = Filename.dirname database_path in - let fixture = Yojson.Safe.from_file (Filename.concat T.root - "logseq_overlay_db/test/fixtures/legacy-retirement-v1.json") in + let fixture = + Yojson.Safe.from_file + (Filename.concat + T.root + "logseq_overlay_db/test/fixtures/legacy-retirement-v1.json") + in let open Yojson.Safe.Util in let receipt = fixture |> member "receipt" in let id = receipt |> member "retirement" |> to_string in let retirement = Database.retirement_id_of_string id |> T.require_ok ~behavior in - let previous = receipt |> member "previous" |> to_string - |> Mirror_generation.of_string |> T.require_ok ~behavior in - let root = Filename.concat (Filename.dirname directory) (".mirror-deletions/" ^ Graph.Uuid.to_string T.graph_uuid) in - List.iter (fun path -> Unix.mkdir path 0o700) - [Filename.dirname root; root; Filename.concat root "states"; Filename.concat root "pending"]; + let previous = + receipt + |> member "previous" + |> to_string + |> Mirror_generation.of_string + |> T.require_ok ~behavior + in + let root = + Filename.concat + (Filename.dirname directory) + (".mirror-deletions/" ^ Graph.Uuid.to_string T.graph_uuid) + in + List.iter + (fun path -> Unix.mkdir path 0o700) + [ Filename.dirname root + ; root + ; Filename.concat root "states" + ; Filename.concat root "pending" + ]; let receipt_path = Filename.concat root ("states/" ^ id ^ ".json") in - Out_channel.with_open_bin receipt_path (fun output -> output_string output (Yojson.Safe.to_string receipt)); + Out_channel.with_open_bin receipt_path (fun output -> + output_string output (Yojson.Safe.to_string receipt)); (* This is captured v1 wire data from the actual pre-upgrade public producer, not a reimplementation of its former generation algorithm. *) let dependencies = T.dependencies ~behavior in - Eio_main.run (fun _ -> Eio.Switch.run (fun sw -> - let inspection = inspect_mirror support behavior in - let database = Database.open_ ~sw dependencies inspection ~graph_name:"Synthetic legacy lease" - |> T.require_ok ~behavior in - if mode = `Before then ( - let busy = Database.prepare_mirror_deletion ~application_support_directory:support - ~graph_id:T.graph_uuid ~retirement in - Alcotest.(check bool) "legacy cancellation waits for live ownership" true + Eio_main.run (fun _ -> + Eio.Switch.run (fun sw -> + let inspection = inspect_mirror support behavior in + let database = + Database.open_ ~sw dependencies inspection ~graph_name:"Synthetic legacy lease" + |> T.require_ok ~behavior + in + if mode = `Before + then ( + let busy = + Database.prepare_mirror_deletion + ~application_support_directory:support + ~graph_id:T.graph_uuid + ~retirement + in + Alcotest.(check bool) + "legacy cancellation waits for live ownership" + true (match busy with | Error Mirror_delete_busy -> true | _ -> false)); - ignore (Database.close database |> T.require_ok ~behavior))); + ignore (Database.close database |> T.require_ok ~behavior))); let data = Filename.concat root ("pending/" ^ id ^ ".data") in - if mode = `Missing then Unix.rename directory (Filename.concat support "preserved-before-missing"); - if mode = `After || mode = `Linked then ( + if mode = `Missing + then Unix.rename directory (Filename.concat support "preserved-before-missing"); + if mode = `After || mode = `Linked + then ( Unix.rename directory data; - Out_channel.with_open_bin (Filename.concat data "db.sqlite") - (fun output -> output_string output "synthetic corrupted legacy retired contents"); + Out_channel.with_open_bin (Filename.concat data "db.sqlite") (fun output -> + output_string output "synthetic corrupted legacy retired contents"); ignore (T.seed_mirror support); - if mode = `Linked then ( + if mode = `Linked + then ( let external_directory = Filename.concat support "captured-external" in - Unix.rename data external_directory; Unix.symlink (Filename.concat external_directory "missing") data)); - let observed = Database.prepare_mirror_deletion ~application_support_directory:support - ~graph_id:T.graph_uuid ~retirement in - if mode = `Linked then ( - Option.iter (fun (prepared, _) -> Database.cancel_mirror_deletion prepared) (Result.to_option observed); - Alcotest.(check bool) "legacy dangling pending symlink is rejected" true (Result.is_error observed); - Alcotest.(check bool) "captured external contents survive" true + Unix.rename data external_directory; + Unix.symlink (Filename.concat external_directory "missing") data)); + let observed = + Database.prepare_mirror_deletion + ~application_support_directory:support + ~graph_id:T.graph_uuid + ~retirement + in + if mode = `Linked + then ( + Option.iter + (fun (prepared, _) -> Database.cancel_mirror_deletion prepared) + (Result.to_option observed); + Alcotest.(check bool) + "legacy dangling pending symlink is rejected" + true + (Result.is_error observed); + Alcotest.(check bool) + "captured external contents survive" + true (Sys.file_exists (Filename.concat support "captured-external/db.sqlite"))) else ( let prepared, generation = observed |> T.require_ok ~behavior in - Fun.protect ~finally:(fun () -> Database.cancel_mirror_deletion prepared) (fun () -> - Alcotest.(check bool) "legacy generation remains an opaque journal identity" true - (Mirror_generation.equal generation previous); - if mode = `Before || mode = `Missing then ( - Alcotest.(check bool) "uncommitted legacy preparation is canceled" false (Database.mirror_deletion_committed prepared); - Alcotest.(check bool) "old preparation cannot rename a live replacement" true - (Result.is_error (Database.commit_mirror_deletion prepared))) - else ( - Alcotest.(check bool) "legacy retired namespace yields durable commit evidence" true - (Database.mirror_deletion_committed prepared); - ignore (Database.commit_mirror_deletion prepared |> T.require_ok ~behavior); - Alcotest.(check bool) "legacy retired contents are collected" false (Sys.file_exists data)))); - Alcotest.(check bool) "canonical is preserved without manufacturing a missing copy" (mode <> `Missing) (Sys.file_exists database_path)) + Fun.protect + ~finally:(fun () -> Database.cancel_mirror_deletion prepared) + (fun () -> + Alcotest.(check bool) + "legacy generation remains an opaque journal identity" + true + (Mirror_generation.equal generation previous); + if mode = `Before || mode = `Missing + then ( + Alcotest.(check bool) + "uncommitted legacy preparation is canceled" + false + (Database.mirror_deletion_committed prepared); + Alcotest.(check bool) + "old preparation cannot rename a live replacement" + true + (Result.is_error (Database.commit_mirror_deletion prepared))) + else ( + Alcotest.(check bool) + "legacy retired namespace yields durable commit evidence" + true + (Database.mirror_deletion_committed prepared); + ignore (Database.commit_mirror_deletion prepared |> T.require_ok ~behavior); + Alcotest.(check bool) + "legacy retired contents are collected" + false + (Sys.file_exists data)))); + Alcotest.(check bool) + "canonical is preserved without manufacturing a missing copy" + (mode <> `Missing) + (Sys.file_exists database_path)) ;; let current_retirement_rejects_replaced_pending_data () = @@ -1136,27 +1697,59 @@ let current_retirement_rejects_replaced_pending_data () = let database_path = T.seed_mirror support in let directory = Filename.dirname database_path in let retirement = next_retirement () in - let prepared, _ = Database.prepare_mirror_deletion ~application_support_directory:support - ~graph_id:T.graph_uuid ~retirement |> T.require_ok ~behavior in - let root = Filename.concat (Filename.dirname directory) (".mirror-deletions/" ^ Graph.Uuid.to_string T.graph_uuid) in + let prepared, _ = + Database.prepare_mirror_deletion + ~application_support_directory:support + ~graph_id:T.graph_uuid + ~retirement + |> T.require_ok ~behavior + in + let root = + Filename.concat + (Filename.dirname directory) + (".mirror-deletions/" ^ Graph.Uuid.to_string T.graph_uuid) + in let states = Filename.concat root "states" in Unix.chmod states 0o500; - Fun.protect ~finally:(fun () -> Unix.chmod states 0o700; Database.cancel_mirror_deletion prepared) - (fun () -> ignore (Database.commit_mirror_deletion prepared |> T.require_ok ~behavior)); - let data = Filename.concat root ("pending/" ^ Database.retirement_id_to_string retirement ^ ".data") in + Fun.protect + ~finally:(fun () -> + Unix.chmod states 0o700; + Database.cancel_mirror_deletion prepared) + (fun () -> + ignore (Database.commit_mirror_deletion prepared |> T.require_ok ~behavior)); + let data = + Filename.concat + root + ("pending/" ^ Database.retirement_id_to_string retirement ^ ".data") + in Unix.rename data (Filename.concat support "captured-old"); ignore (T.seed_mirror support); let dependencies, _, inspection = location_and_inspection support behavior in - Eio_main.run (fun _ -> Eio.Switch.run (fun sw -> - let database = Database.open_ ~sw dependencies inspection ~graph_name:"Synthetic replacement" - |> T.require_ok ~behavior in - ignore (Database.close database |> T.require_ok ~behavior))); + Eio_main.run (fun _ -> + Eio.Switch.run (fun sw -> + let database = + Database.open_ ~sw dependencies inspection ~graph_name:"Synthetic replacement" + |> T.require_ok ~behavior + in + ignore (Database.close database |> T.require_ok ~behavior))); Unix.rename directory data; - let replay = Database.prepare_mirror_deletion ~application_support_directory:support - ~graph_id:T.graph_uuid ~retirement in - Option.iter (fun (p, _) -> Database.cancel_mirror_deletion p) (Result.to_option replay); - Alcotest.(check bool) "replacement pending namespace is rejected" true (Result.is_error replay); - Alcotest.(check bool) "replacement pending contents survive" true (Sys.file_exists (Filename.concat data "db.sqlite"))) + let replay = + Database.prepare_mirror_deletion + ~application_support_directory:support + ~graph_id:T.graph_uuid + ~retirement + in + Option.iter + (fun (p, _) -> Database.cancel_mirror_deletion p) + (Result.to_option replay); + Alcotest.(check bool) + "replacement pending namespace is rejected" + true + (Result.is_error replay); + Alcotest.(check bool) + "replacement pending contents survive" + true + (Sys.file_exists (Filename.concat data "db.sqlite"))) ;; let corrupt_outbox_fails_closed () = @@ -1394,7 +1987,9 @@ let require_generated_mutation_receipt database_path outcome expected_keys = | `List [ `String _tag; `Assoc fields ] as json -> (match List.assoc_opt "status" fields with | Some (`List [ `String actual ]) - when String.equal actual (if outcome = "noChange" then "No_change" else "Applied") -> + when String.equal + actual + (if outcome = "noChange" then "No_change" else "Applied") -> Some (source, json, fields) | _ -> None) | _ -> None) @@ -1435,7 +2030,8 @@ let duplicate_outbox_member_fails_closed () = Database.close database |> T.require_ok ~behavior; let sqlite = Sqlite3.db_open ~mode:`NO_CREATE database_path in let records = - Logseq_db_storage.Sync_outbox_store.read_database sqlite |> T.require_ok ~behavior + Logseq_db_storage.Sync_outbox_store.read_database sqlite + |> T.require_ok ~behavior in Sqlite3.Rc.check (Sqlite3.exec @@ -1459,8 +2055,10 @@ let duplicate_outbox_member_fails_closed () = let corrupt_single_outbox_record sqlite record = let statement = Sqlite3.prepare sqlite "UPDATE sync_outbox SET record=?" in - Fun.protect ~finally:(fun () -> ignore (Sqlite3.finalize statement)) (fun () -> - Sqlite3.Rc.check (Sqlite3.bind_text statement 1 record); + Fun.protect + ~finally:(fun () -> ignore (Sqlite3.finalize statement)) + (fun () -> + Sqlite3.Rc.check (Sqlite3.bind_text statement 1 record); Sqlite3.Rc.check (Sqlite3.step statement)) ;; @@ -1540,7 +2138,7 @@ let pre_uuid_status_outbox_preserves_identity () = "opening legacy outbox changed operation fingerprints") ;; -let noncanonical_outbox_payload_fails_closed ?(payload="tampered") () = +let noncanonical_outbox_payload_fails_closed ?(payload = "tampered") () = let behavior = "noncanonical durable outbox payload fails closed" in T.with_temp_directory "overlay-noncanonical-outbox-" (fun support -> let database_path = T.seed_mirror support in @@ -1957,10 +2555,30 @@ let no_change_receipt_survives_reopen () = Database.close second |> T.require_ok ~behavior))) ;; -let authoritative_checkpoint_and_root_survive_reopen () = +let authoritative_checkpoint_and_root_survive_reopen ?(mismatch = false) () = let behavior = "authoritative checkpoint and root survive reopen" in T.with_temp_directory "overlay-authoritative-reopen-" (fun support -> - ignore (T.seed_mirror support); + let database_path = T.seed_mirror support in + let durable_state () = + let sqlite = Sqlite3.db_open ~mode:`READONLY database_path in + Fun.protect + ~finally:(fun () -> ignore (Sqlite3.db_close sqlite)) + (fun () -> + List.map + (fun table -> + let rows = ref [] in + Sqlite3.Rc.check + (Sqlite3.exec + sqlite + ~cb:(fun row _ -> rows := Array.to_list row :: !rows) + ("SELECT * FROM " ^ table ^ " ORDER BY rowid")); + table, List.rev !rows) + [ "sync_meta" + ; "sync_outbox" + ; "overlay_mutation_receipts" + ; "overlay_terminal_batch_receipts" + ]) + in let dependencies, location, inspection = location_and_inspection support behavior in let opening_inspection = inspection in Eio_main.run (fun _environment -> @@ -1991,7 +2609,8 @@ let authoritative_checkpoint_and_root_survive_reopen () = in let cursor = Server_cursor.of_int64 1L - |> Result.map_error (fun `Negative_cursor -> "negative cursor") |> T.require_ok ~behavior + |> Result.map_error (fun `Negative_cursor -> "negative cursor") + |> T.require_ok ~behavior in let batch = authoritative_batch @@ -1999,9 +2618,46 @@ let authoritative_checkpoint_and_root_survive_reopen () = ~maximum_bytes:4_096 ~transactions:[ authoritative_transaction ~cursor ~transaction:wire ] ~through:cursor - ~checksum:None + ~checksum:(if mismatch then Some (checksum "ffffffffffffffff") else None) |> T.require_ok ~behavior in + if mismatch + then ( + Database.commit_local + first + ~expected:(T.insert_precondition first ~parent:T.page_uuid ~behavior) + (T.insert_blocks ~ordinal:980 ()) + |> T.require_ok ~behavior + |> ignore; + let snapshot = Database.current_snapshot first |> T.require_ok ~behavior in + let title = + match + Database.get_outline_blocks snapshot [ T.authoritative_block_uuid ] + |> T.require_ok ~behavior + with + | [ Present_block { value; _ } ] -> value.block.title + | _ -> Alcotest.fail "missing checksum fixture block" + in + Database.release_snapshot snapshot; + Database.commit_local + first + ~expected:(block_precondition first T.authoritative_block_uuid behavior) + (Save_block + { mutation_id = T.mutation_uuid 981 + ; block = T.authoritative_block_uuid + ; title + }) + |> T.require_ok ~behavior + |> ignore); + T.require + (Database.check_checkpoint_checksum + first + ~through:(server_cursor 0) + ~expected:(checksum "0000000000000000") + |> T.require_ok ~behavior + = Some false) + "restoration trusted the seed's server-claimed checksum"; + let durable_before = durable_state () in let sync = Database.inspect_sync first |> T.require_ok ~behavior in let prepared, request = Database.begin_authoritative first ~expected:(sync_view_token sync) batch @@ -2010,21 +2666,38 @@ let authoritative_checkpoint_and_root_survive_reopen () = T.require (Option.is_none request) "plaintext authoritative fixture requested crypto"; - let application = - match - Database.apply_authoritative first prepared ~decrypted:None - |> T.require_ok ~behavior - with - | Authoritative_applied commit -> commit - | Authoritative_deferred _ -> Alcotest.fail "authoritative fixture was deferred" + let snapshot = Database.current_snapshot first |> T.require_ok ~behavior in + let before = + Database.get_outline_blocks snapshot [ T.authoritative_block_uuid ] + |> T.require_ok ~behavior in - ignore application; + Database.release_snapshot snapshot; + let result = Database.apply_authoritative first prepared ~decrypted:None in + if mismatch + then ( + T.require + (durable_state () = durable_before) + "checksum mismatch changed durable checkpoint, outbox or receipts"; + (match result with + | Error Authoritative_checksum_mismatch -> () + | Error _ -> Alcotest.fail "checksum mismatch returned the wrong error" + | Ok _ -> Alcotest.fail "checksum mismatch was committed"); + let after = Database.inspect_sync first |> T.require_ok ~behavior in + T.require + (sync_token_equal (sync_view_token sync) (sync_view_token after)) + "checksum rejection changed sync state") + else ( + match T.require_ok ~behavior result with + | Authoritative_applied _ -> () + | Authoritative_deferred _ -> Alcotest.fail "authoritative fixture was deferred"); Database.close first |> T.require_ok ~behavior; let refreshed = inspect_mirror location behavior in (match Database.mirror_presence refreshed with | Available { checkpoint; _ } -> T.require - (Server_cursor.equal checkpoint cursor) + (Server_cursor.equal + checkpoint + (if mismatch then sync_view_checkpoint sync else cursor)) "reopen inspection lost authoritative checkpoint" | Absent _ -> Alcotest.fail "authoritative commit removed the mirror"); let opening_inspection = refreshed in @@ -2033,15 +2706,36 @@ let authoritative_checkpoint_and_root_survive_reopen () = |> T.require_ok ~behavior in let snapshot = Database.current_snapshot second |> T.require_ok ~behavior in - (match - Database.get_outline_blocks snapshot [ T.authoritative_block_uuid ] - |> T.require_ok ~behavior - with - | [ Present_block { value; _ } ] -> - T.require - (value.block.updated_at_ms = 1_704_067_200_456L) - "reopen lost authoritative block update" - | _ -> Alcotest.fail "reopen lost authoritative block"); + let after = + Database.get_outline_blocks snapshot [ T.authoritative_block_uuid ] + |> T.require_ok ~behavior + in + if mismatch + then ( + let values blocks = + List.filter_map + (function + | Present_block { value; _ } -> Some value + | _ -> None) + blocks + in + T.require + (values after = values before) + "rejected checksum changed durable graph"; + T.require + (durable_state () = durable_before) + "reopen changed rejected checkpoint, outbox or receipts"; + let reopened = Database.inspect_sync second |> T.require_ok ~behavior in + T.require + (sync_view_submissions reopened = sync_view_submissions sync) + "reopen changed rejected outbox") + else ( + match after with + | [ Present_block { value; _ } ] -> + T.require + (value.block.updated_at_ms = 1_704_067_200_456L) + "reopen lost authoritative block update" + | _ -> Alcotest.fail "reopen lost authoritative block"); Database.release_snapshot snapshot; Database.close second |> T.require_ok ~behavior))) ;; @@ -2082,7 +2776,9 @@ let outbox_persistence_failure_is_atomic () = before_version.projection_revision after_version.projection_revision) "failed outbox commit advanced projection revision"; - (match Database.get_outline_blocks after [ T.block_uuid ] |> T.require_ok ~behavior with + (match + Database.get_outline_blocks after [ T.block_uuid ] |> T.require_ok ~behavior + with | [ Missing_block _ ] -> () | _ -> Alcotest.fail "failed outbox commit exposed the pending block"); Database.release_snapshot after; @@ -2101,13 +2797,17 @@ let receipt_writer_schema_failure_is_atomic () = Fun.protect ~finally:(fun () -> ignore (Sqlite3.db_close sqlite)) (fun () -> - Sqlite3.Rc.check (Sqlite3.exec sqlite - "CREATE TABLE overlay_terminal_batch_receipts (batch_key TEXT PRIMARY KEY)")); + Sqlite3.Rc.check + (Sqlite3.exec + sqlite + "CREATE TABLE overlay_terminal_batch_receipts (batch_key TEXT PRIMARY KEY)")); let dependencies, _location, inspection = location_and_inspection support behavior in Eio_main.run (fun _environment -> Eio.Switch.run (fun sw -> - let database = Database.open_ ~sw dependencies inspection ~graph_name:"oracle-graph" - |> T.require_ok ~behavior in + let database = + Database.open_ ~sw dependencies inspection ~graph_name:"oracle-graph" + |> T.require_ok ~behavior + in Fun.protect ~finally:(fun () -> ignore (Database.close database)) (fun () -> @@ -2115,62 +2815,90 @@ let receipt_writer_schema_failure_is_atomic () = let before_version = Database.snapshot_version before in Database.release_snapshot before; let before_sync = Database.inspect_sync database |> T.require_ok ~behavior in - let expected = T.insert_precondition database ~parent:T.page_uuid ~behavior in + let expected = + T.insert_precondition database ~parent:T.page_uuid ~behavior + in let mutation = T.insert_blocks ~ordinal:411 () in (match Database.commit_local database ~expected mutation with | Error (Local_commit_persistence_failed message) -> - Alcotest.(check string) "receipt writer preserves the storage cause" - "Sqlite3.Error(\"Sqlite3.prepare: table overlay_terminal_batch_receipts has no column named receipt\")" + Alcotest.(check string) + "receipt writer preserves the storage cause" + "Sqlite3.Error(\"Sqlite3.prepare: table \ + overlay_terminal_batch_receipts has no column named receipt\")" message | Error _ -> Alcotest.fail "receipt writer returned the wrong commit error" | Ok _ -> Alcotest.fail "receipt writer exposed a successful commit"); let after = Database.current_snapshot database |> T.require_ok ~behavior in - T.require (Projection_revision.equal before_version.projection_revision - (Database.snapshot_version after).projection_revision) + T.require + (Projection_revision.equal + before_version.projection_revision + (Database.snapshot_version after).projection_revision) "receipt writer failure advanced projection revision"; - (match Database.get_outline_blocks after [ T.block_uuid ] |> T.require_ok ~behavior with + (match + Database.get_outline_blocks after [ T.block_uuid ] + |> T.require_ok ~behavior + with | [ Missing_block _ ] -> () | _ -> Alcotest.fail "receipt writer failure exposed the pending block"); Database.release_snapshot after; let after_sync = Database.inspect_sync database |> T.require_ok ~behavior in - T.require (sync_token_equal (sync_view_token before_sync) (sync_view_token after_sync)) + T.require + (sync_token_equal + (sync_view_token before_sync) + (sync_view_token after_sync)) "receipt writer failure advanced sync token"; - T.require (sync_view_submissions after_sync = []) + T.require + (sync_view_submissions after_sync = []) "receipt writer failure exposed a submission"; let sqlite = Sqlite3.db_open ~mode:`NO_CREATE database_path in Fun.protect ~finally:(fun () -> ignore (Sqlite3.db_close sqlite)) (fun () -> - Alcotest.(check int) "durable outbox rolled back" 0 - (List.length (Logseq_db_storage.Sync_outbox_store.read_database sqlite - |> T.require_ok ~behavior)); - Alcotest.(check int) "durable mutation receipt rolled back" 0 - (List.length (Logseq_db_storage.Mutation_receipt_store.read_database sqlite - |> T.require_ok ~behavior)); - Sqlite3.Rc.check (Sqlite3.exec sqlite "DROP TABLE overlay_terminal_batch_receipts"); + Alcotest.(check int) + "durable outbox rolled back" + 0 + (List.length + (Logseq_db_storage.Sync_outbox_store.read_database sqlite + |> T.require_ok ~behavior)); + Alcotest.(check int) + "durable mutation receipt rolled back" + 0 + (List.length + (Logseq_db_storage.Mutation_receipt_store.read_database sqlite + |> T.require_ok ~behavior)); + Sqlite3.Rc.check + (Sqlite3.exec sqlite "DROP TABLE overlay_terminal_batch_receipts"); Logseq_db_storage.Mutation_receipt_store.initialize_database sqlite |> T.require_ok ~behavior); - (match Database.commit_local database ~expected mutation |> T.require_ok ~behavior with - | Local_committed _ -> () - | _ -> Alcotest.fail "same mutation retry retained a failed receipt"))))) + match + Database.commit_local database ~expected mutation |> T.require_ok ~behavior + with + | Local_committed _ -> () + | _ -> Alcotest.fail "same mutation retry retained a failed receipt")))) ;; let receipt_writer_schema_errors_are_results () = let module Store = Logseq_db_storage.Mutation_receipt_store in let check_writer table key_column write = - List.iter (fun receipts -> - let sqlite = Sqlite3.db_open ":memory:" in - Fun.protect - ~finally:(fun () -> ignore (Sqlite3.db_close sqlite)) - (fun () -> - Sqlite3.Rc.check (Sqlite3.exec sqlite - ("CREATE TABLE " ^ table ^ " (" ^ key_column ^ " TEXT PRIMARY KEY)")); - match write sqlite receipts with - | Error message -> - Alcotest.(check string) "receipt writer preserves the prepare cause" - ("Sqlite3.Error(\"Sqlite3.prepare: table " ^ table ^ " has no column named receipt\")") - message - | Ok () -> Alcotest.fail "malformed receipt writer reported success")) + List.iter + (fun receipts -> + let sqlite = Sqlite3.db_open ":memory:" in + Fun.protect + ~finally:(fun () -> ignore (Sqlite3.db_close sqlite)) + (fun () -> + Sqlite3.Rc.check + (Sqlite3.exec + sqlite + ("CREATE TABLE " ^ table ^ " (" ^ key_column ^ " TEXT PRIMARY KEY)")); + match write sqlite receipts with + | Error message -> + Alcotest.(check string) + "receipt writer preserves the prepare cause" + ("Sqlite3.Error(\"Sqlite3.prepare: table " + ^ table + ^ " has no column named receipt\")") + message + | Ok () -> Alcotest.fail "malformed receipt writer reported success")) [ []; [ String.make 64 'a', "{}" ] ] in check_writer "overlay_mutation_receipts" "receipt_key" Store.upsert_mutations; @@ -2245,38 +2973,71 @@ let submit_persistence_failure_restores_queued_state () = -> () | _ -> Alcotest.fail "failed submit did not restore Queued state"); let sqlite = Sqlite3.db_open ~mode:`NO_CREATE database_path in - Fun.protect ~finally:(fun () -> ignore (Sqlite3.db_close sqlite)) (fun () -> - Alcotest.(check (list string)) "failed submit rolls back durable mutable record" - durable_before (Store.read_database sqlite |> T.require_ok ~behavior); - T.require (Store.read_revision sqlite |> T.require_ok ~behavior = revision_before) - "failed submit rolls back metadata revision"; - Sqlite3.Rc.check (Sqlite3.exec sqlite "DROP TRIGGER fail_overlay_submit"); - let prepared, request = Database.begin_outbox_transition database - ~expected:(sync_view_token after) (Submit_group [ mutation_id ]) - |> T.require_ok ~behavior in - let request = Option.get request in - let encrypted = Database.protection_plaintexts request - |> List.map (fun (id, plaintext) -> id, "encrypted:" ^ plaintext) in - let retried = Database.apply_outbox_transition database prepared - ~encrypted:(Some (request, encrypted)) |> T.require_ok ~behavior in - T.require (Store.read_database sqlite |> T.require_ok ~behavior <> durable_before) - "retry after rollback failed to persist its changed frozen baseline"; - let current_revision = Store.read_revision sqlite |> T.require_ok ~behavior |> Option.get in - T.require (current_revision = Option.get revision_before + 1) - "successful retry must advance metadata exactly once"; - let durable_after_retry = Store.read_database sqlite |> T.require_ok ~behavior in - (* A second public storage writer advances only its revision. Database + Fun.protect + ~finally:(fun () -> ignore (Sqlite3.db_close sqlite)) + (fun () -> + Alcotest.(check (list string)) + "failed submit rolls back durable mutable record" + durable_before + (Store.read_database sqlite |> T.require_ok ~behavior); + T.require + (Store.read_revision sqlite |> T.require_ok ~behavior = revision_before) + "failed submit rolls back metadata revision"; + Sqlite3.Rc.check (Sqlite3.exec sqlite "DROP TRIGGER fail_overlay_submit"); + let prepared, request = + Database.begin_outbox_transition + database + ~expected:(sync_view_token after) + (Submit_group [ mutation_id ]) + |> T.require_ok ~behavior + in + let request = Option.get request in + let encrypted = + Database.protection_plaintexts request + |> List.map (fun (id, plaintext) -> id, "encrypted:" ^ plaintext) + in + let retried = + Database.apply_outbox_transition + database + prepared + ~encrypted:(Some (request, encrypted)) + |> T.require_ok ~behavior + in + T.require + (Store.read_database sqlite |> T.require_ok ~behavior <> durable_before) + "retry after rollback failed to persist its changed frozen baseline"; + let current_revision = + Store.read_revision sqlite |> T.require_ok ~behavior |> Option.get + in + T.require + (current_revision = Option.get revision_before + 1) + "successful retry must advance metadata exactly once"; + let durable_after_retry = + Store.read_database sqlite |> T.require_ok ~behavior + in + (* A second public storage writer advances only its revision. Database owns executing the CAS and must reject its now stale durable baseline. *) - Sqlite3.Rc.check (Sqlite3.exec sqlite "BEGIN IMMEDIATE"); - Store.apply_delta sqlite - { expected_revision = current_revision; revision = current_revision + 1 - ; upserts = []; deletes = [] } |> T.require_ok ~behavior; - Sqlite3.Rc.check (Sqlite3.exec sqlite "COMMIT"); - let submitted_view = Database.inspect_sync database |> T.require_ok ~behavior in - let batch = Option.get retried.submission_batch in - let prepared, request = Database.begin_outbox_transition database - ~expected:(sync_view_token submitted_view) - (Retry_group (submission_batch_id batch)) |> T.require_ok ~behavior in + Sqlite3.Rc.check (Sqlite3.exec sqlite "BEGIN IMMEDIATE"); + Store.apply_delta + sqlite + { expected_revision = current_revision + ; revision = current_revision + 1 + ; upserts = [] + ; deletes = [] + } + |> T.require_ok ~behavior; + Sqlite3.Rc.check (Sqlite3.exec sqlite "COMMIT"); + let submitted_view = + Database.inspect_sync database |> T.require_ok ~behavior + in + let batch = Option.get retried.submission_batch in + let prepared, request = + Database.begin_outbox_transition + database + ~expected:(sync_view_token submitted_view) + (Retry_group (submission_batch_id batch)) + |> T.require_ok ~behavior + in let encrypted = Option.map (fun request -> @@ -2285,16 +3046,26 @@ let submit_persistence_failure_restores_queued_state () = |> List.map (fun (id, plaintext) -> id, "encrypted:" ^ plaintext) )) request in - (match Database.apply_outbox_transition database prepared ~encrypted with - | Error (Outbox_commit_persistence_failed _) -> () - | _ -> Alcotest.fail "stale durable metadata CAS exposed a successful retry"); - Alcotest.(check (list string)) "metadata CAS failure preserves every payload" - durable_after_retry (Store.read_database sqlite |> T.require_ok ~behavior); - T.require (Store.read_revision sqlite |> T.require_ok ~behavior = Some (current_revision + 1)) - "metadata CAS failure overwrote the winning writer revision"; - let after_conflict = Database.inspect_sync database |> T.require_ok ~behavior in - T.require (sync_token_equal (sync_view_token submitted_view) (sync_view_token after_conflict)) - "metadata CAS failure published an in-memory revision"); + (match Database.apply_outbox_transition database prepared ~encrypted with + | Error (Outbox_commit_persistence_failed _) -> () + | _ -> Alcotest.fail "stale durable metadata CAS exposed a successful retry"); + Alcotest.(check (list string)) + "metadata CAS failure preserves every payload" + durable_after_retry + (Store.read_database sqlite |> T.require_ok ~behavior); + T.require + (Store.read_revision sqlite + |> T.require_ok ~behavior + = Some (current_revision + 1)) + "metadata CAS failure overwrote the winning writer revision"; + let after_conflict = + Database.inspect_sync database |> T.require_ok ~behavior + in + T.require + (sync_token_equal + (sync_view_token submitted_view) + (sync_view_token after_conflict)) + "metadata CAS failure published an in-memory revision"); ignore (Database.close database)))) ;; @@ -2731,15 +3502,17 @@ let with_outbox_write_fixture behavior f = let outbox_write_count sqlite = let count = ref 0 in Sqlite3.Rc.check - (Sqlite3.exec sqlite "SELECT count(*) FROM outbox_write_audit" - ~cb:(fun row _ -> count := int_of_string (Option.get row.(0)))); + (Sqlite3.exec sqlite "SELECT count(*) FROM outbox_write_audit" ~cb:(fun row _ -> + count := int_of_string (Option.get row.(0)))); !count ;; let outbox_payload_bytes sqlite = let count = ref 0 in Sqlite3.Rc.check - (Sqlite3.exec sqlite "SELECT coalesce(sum(payload_bytes),0) FROM outbox_write_audit" + (Sqlite3.exec + sqlite + "SELECT coalesce(sum(payload_bytes),0) FROM outbox_write_audit" ~cb:(fun row _ -> count := int_of_string (Option.get row.(0)))); !count ;; @@ -2762,7 +3535,8 @@ let append_outbox_save database ordinal behavior = in let commit = match - Database.commit_local database + Database.commit_local + database ~expected:(block_precondition database T.authoritative_block_uuid behavior) mutation |> T.require_ok ~behavior @@ -2792,330 +3566,552 @@ let apply_outbox_event database transition behavior = let incremental_outbox_appends count () = let behavior = Printf.sprintf "incremental outbox appends %d" count in - with_outbox_write_fixture behavior + with_outbox_write_fixture + behavior (fun ~sw:_ ~dependencies:_ ~support:_ ~path:_ ~sqlite database -> - for ordinal = 1 to count do - ignore (append_outbox_save database ordinal behavior) - done; - Alcotest.(check int) "one durable row write per appended intent" count - (outbox_write_count sqlite); - let stored = persisted_outbox sqlite behavior in - Alcotest.(check int) "all intents recover in order" count (List.length stored); - let final_bytes = List.fold_left (fun total record -> total + String.length record) 0 stored in - Alcotest.(check int) "each appended payload is persisted exactly once" final_bytes - (outbox_payload_bytes sqlite); + for ordinal = 1 to count do + ignore (append_outbox_save database ordinal behavior) + done; + Alcotest.(check int) + "one durable row write per appended intent" + count + (outbox_write_count sqlite); + let stored = persisted_outbox sqlite behavior in + Alcotest.(check int) "all intents recover in order" count (List.length stored); + let final_bytes = + List.fold_left (fun total record -> total + String.length record) 0 stored + in + Alcotest.(check int) + "each appended payload is persisted exactly once" + final_bytes + (outbox_payload_bytes sqlite); Printf.printf "outbox append N=%d committed_row_changes=%d logical_payload_bytes=%d \ final_payload_bytes=%d\n\ %!" - count (outbox_write_count sqlite) (outbox_payload_bytes sqlite) final_bytes) + count + (outbox_write_count sqlite) + (outbox_payload_bytes sqlite) + final_bytes) ;; let incremental_outbox_preserves_unchanged_rows () = let behavior = "incremental outbox preserves unchanged rows" in - with_outbox_write_fixture behavior + with_outbox_write_fixture + behavior (fun ~sw:_ ~dependencies:_ ~support:_ ~path:_ ~sqlite database -> - ignore (append_outbox_save database 1 behavior); - let before = List.hd (persisted_outbox sqlite behavior) in - ignore (append_outbox_save database 2 behavior); - Alcotest.(check string) "unrelated revision does not reencode prior payload" before - (List.hd (persisted_outbox sqlite behavior))) + ignore (append_outbox_save database 1 behavior); + let before = List.hd (persisted_outbox sqlite behavior) in + ignore (append_outbox_save database 2 behavior); + Alcotest.(check string) + "unrelated revision does not reencode prior payload" + before + (List.hd (persisted_outbox sqlite behavior))) ;; let incremental_outbox_no_change_and_identity () = let behavior = "incremental outbox No_change and identity" in - with_outbox_write_fixture behavior + with_outbox_write_fixture + behavior (fun ~sw:_ ~dependencies:_ ~support:_ ~path:_ ~sqlite database -> - let mutation, _ = append_outbox_save database 1 behavior in - for ordinal = 2 to 32 do - ignore (append_outbox_save database ordinal behavior) - done; - reset_outbox_write_count sqlite; - let no_change = - Save_block - { mutation_id = T.mutation_uuid 20_001 - ; block = T.block_uuid - ; title = "Missing target" - } - in - (match Database.commit_local database - ~expected:(block_precondition database T.block_uuid behavior) no_change - |> T.require_ok ~behavior with - | Local_committed { status = No_change; _ } -> () - | _ -> Alcotest.fail "missing target must produce a durable No_change receipt"); - (match Database.commit_local database ~expected:(T.empty_precondition ~behavior) mutation - |> T.require_ok ~behavior with - | Local_existing _ -> () - | _ -> Alcotest.fail "same-ID replay appended another intent"); - (match Database.commit_local database ~expected:(T.empty_precondition ~behavior) - (Save_block { mutation_id = T.mutation_uuid 10_001 - ; block = T.authoritative_block_uuid; title = "Different intent" }) with - | Error Mutation_identity_conflict -> () - | _ -> Alcotest.fail "same-ID different intent must remain a conflict"); - Alcotest.(check int) "receipt and identity checks do not rewrite outbox rows" 0 - (outbox_write_count sqlite)) + let mutation, _ = append_outbox_save database 1 behavior in + for ordinal = 2 to 32 do + ignore (append_outbox_save database ordinal behavior) + done; + reset_outbox_write_count sqlite; + let no_change = + Save_block + { mutation_id = T.mutation_uuid 20_001 + ; block = T.block_uuid + ; title = "Missing target" + } + in + (match + Database.commit_local + database + ~expected:(block_precondition database T.block_uuid behavior) + no_change + |> T.require_ok ~behavior + with + | Local_committed { status = No_change; _ } -> () + | _ -> Alcotest.fail "missing target must produce a durable No_change receipt"); + (match + Database.commit_local + database + ~expected:(T.empty_precondition ~behavior) + mutation + |> T.require_ok ~behavior + with + | Local_existing _ -> () + | _ -> Alcotest.fail "same-ID replay appended another intent"); + (match + Database.commit_local + database + ~expected:(T.empty_precondition ~behavior) + (Save_block + { mutation_id = T.mutation_uuid 10_001 + ; block = T.authoritative_block_uuid + ; title = "Different intent" + }) + with + | Error Mutation_identity_conflict -> () + | _ -> Alcotest.fail "same-ID different intent must remain a conflict"); + Alcotest.(check int) + "receipt and identity checks do not rewrite outbox rows" + 0 + (outbox_write_count sqlite)) ;; let incremental_outbox_batch_updates () = let behavior = "incremental outbox batch updates" in - with_outbox_write_fixture behavior + with_outbox_write_fixture + behavior (fun ~sw:_ ~dependencies:_ ~support:_ ~path:_ ~sqlite database -> - let commits = List.init 64 (fun n -> snd (append_outbox_save database (n + 1) behavior)) in - let ids = List.take 32 commits |> List.map (fun (commit : local_commit) -> commit.mutation_id) in - reset_outbox_write_count sqlite; - let submitted = apply_outbox_event database (Submit_group ids) behavior in - Alcotest.(check int) "submit persists its members only" 32 (outbox_write_count sqlite); - let batch = Option.get submitted.submission_batch in - reset_outbox_write_count sqlite; - ignore - (apply_outbox_event database - (Accept_group - { batch_id = submission_batch_id batch - ; barrier = { through = server_cursor 32; checksum = checksum "0000000000000000" } - }) behavior); - Alcotest.(check int) "accept persists its members only" 32 (outbox_write_count sqlite); - let stored = persisted_outbox sqlite behavior in - let view = Database.inspect_sync database |> T.require_ok ~behavior in - Alcotest.(check int) "mutable transport updates remain durable" 32 - (List.fold_left - (fun count descriptor -> match descriptor.state with - | Accepted_pending_authoritative _ -> count + 1 | _ -> count) - 0 (sync_view_submissions view)); - Alcotest.(check int) "other queued rows remain present" 64 (List.length stored)) + let commits = + List.init 64 (fun n -> snd (append_outbox_save database (n + 1) behavior)) + in + let ids = + List.take 32 commits + |> List.map (fun (commit : local_commit) -> commit.mutation_id) + in + reset_outbox_write_count sqlite; + let submitted = apply_outbox_event database (Submit_group ids) behavior in + Alcotest.(check int) + "submit persists its members only" + 32 + (outbox_write_count sqlite); + let batch = Option.get submitted.submission_batch in + reset_outbox_write_count sqlite; + ignore + (apply_outbox_event + database + (Accept_group + { batch_id = submission_batch_id batch + ; barrier = + { through = server_cursor 32; checksum = checksum "0000000000000000" } + }) + behavior); + Alcotest.(check int) + "accept persists its members only" + 32 + (outbox_write_count sqlite); + let stored = persisted_outbox sqlite behavior in + let view = Database.inspect_sync database |> T.require_ok ~behavior in + Alcotest.(check int) + "mutable transport updates remain durable" + 32 + (List.fold_left + (fun count descriptor -> + match descriptor.state with + | Accepted_pending_authoritative _ -> count + 1 + | _ -> count) + 0 + (sync_view_submissions view)); + Alcotest.(check int) "other queued rows remain present" 64 (List.length stored)) ;; let reject_and_discard database (commit : local_commit) behavior = - let submitted = apply_outbox_event database (Submit_group [ commit.mutation_id ]) behavior in + let submitted = + apply_outbox_event database (Submit_group [ commit.mutation_id ]) behavior + in let batch = Option.get submitted.submission_batch in ignore - (apply_outbox_event database + (apply_outbox_event + database (Reject_group { batch_id = submission_batch_id batch - ; resolution = Definitive - { reason = Invalid_request - ; partition = { accepted_prefix = []; failed_member = Some commit.mutation_id - ; unexecuted_suffix = []; acceptance_barrier = None - ; missing_uuids = []; diagnostics = [] } - } - }) behavior); - Database.discard_blocked database ~mutation_id:commit.mutation_id |> T.require_ok ~behavior |> ignore + ; resolution = + Definitive + { reason = Invalid_request + ; partition = + { accepted_prefix = [] + ; failed_member = Some commit.mutation_id + ; unexecuted_suffix = [] + ; acceptance_barrier = None + ; missing_uuids = [] + ; diagnostics = [] + } + } + }) + behavior); + Database.discard_blocked database ~mutation_id:commit.mutation_id + |> T.require_ok ~behavior + |> ignore ;; let incremental_outbox_middle_delete_and_empty_revision () = let behavior = "incremental outbox middle delete and empty revision" in - with_outbox_write_fixture behavior + with_outbox_write_fixture + behavior (fun ~sw ~dependencies ~support ~path:_ ~sqlite database -> - let commits = List.init 3 (fun n -> snd (append_outbox_save database (n + 1) behavior)) in - let middle = List.nth commits 1 in - let submitted = apply_outbox_event database (Submit_group [ middle.mutation_id ]) behavior in - let batch = Option.get submitted.submission_batch in - ignore (apply_outbox_event database - (Reject_group { batch_id = submission_batch_id batch - ; resolution = Definitive - { reason = Invalid_request - ; partition = { accepted_prefix = []; failed_member = Some middle.mutation_id - ; unexecuted_suffix = []; acceptance_barrier = None - ; missing_uuids = []; diagnostics = [] } } }) behavior); - let before = persisted_outbox sqlite behavior in - reset_outbox_write_count sqlite; - Database.discard_blocked database ~mutation_id:middle.mutation_id |> T.require_ok ~behavior |> ignore; - Alcotest.(check int) "middle discard deletes one row" 1 (outbox_write_count sqlite); - Alcotest.(check (list string)) "remaining sequence holes and payloads are stable" - [ List.nth before 0; List.nth before 2 ] (persisted_outbox sqlite behavior); - reject_and_discard database (List.nth commits 0) behavior; - reject_and_discard database (List.nth commits 2) behavior; - let before_reopen = Database.inspect_sync database |> T.require_ok ~behavior in - Database.close database |> T.require_ok ~behavior; - let inspection = inspect_mirror support behavior in - let reopened = Database.open_ ~sw dependencies inspection ~graph_name:"empty-queue" - |> T.require_ok ~behavior in - let after_reopen = Database.inspect_sync reopened |> T.require_ok ~behavior in - T.require (sync_token_equal (sync_view_token before_reopen) (sync_view_token after_reopen)) - "empty durable queue lost its independent sync revision"; - Database.close reopened |> T.require_ok ~behavior) + let commits = + List.init 3 (fun n -> snd (append_outbox_save database (n + 1) behavior)) + in + let middle = List.nth commits 1 in + let submitted = + apply_outbox_event database (Submit_group [ middle.mutation_id ]) behavior + in + let batch = Option.get submitted.submission_batch in + ignore + (apply_outbox_event + database + (Reject_group + { batch_id = submission_batch_id batch + ; resolution = + Definitive + { reason = Invalid_request + ; partition = + { accepted_prefix = [] + ; failed_member = Some middle.mutation_id + ; unexecuted_suffix = [] + ; acceptance_barrier = None + ; missing_uuids = [] + ; diagnostics = [] + } + } + }) + behavior); + let before = persisted_outbox sqlite behavior in + reset_outbox_write_count sqlite; + Database.discard_blocked database ~mutation_id:middle.mutation_id + |> T.require_ok ~behavior + |> ignore; + Alcotest.(check int) "middle discard deletes one row" 1 (outbox_write_count sqlite); + Alcotest.(check (list string)) + "remaining sequence holes and payloads are stable" + [ List.nth before 0; List.nth before 2 ] + (persisted_outbox sqlite behavior); + reject_and_discard database (List.nth commits 0) behavior; + reject_and_discard database (List.nth commits 2) behavior; + let before_reopen = Database.inspect_sync database |> T.require_ok ~behavior in + Database.close database |> T.require_ok ~behavior; + let inspection = inspect_mirror support behavior in + let reopened = + Database.open_ ~sw dependencies inspection ~graph_name:"empty-queue" + |> T.require_ok ~behavior + in + let after_reopen = Database.inspect_sync reopened |> T.require_ok ~behavior in + T.require + (sync_token_equal (sync_view_token before_reopen) (sync_view_token after_reopen)) + "empty durable queue lost its independent sync revision"; + Database.close reopened |> T.require_ok ~behavior) ;; let incremental_outbox_empty_revision () = let behavior = "incremental outbox empty revision" in - with_outbox_write_fixture behavior + with_outbox_write_fixture + behavior (fun ~sw ~dependencies ~support ~path:_ ~sqlite:_ database -> - let _, commit = append_outbox_save database 1 behavior in - reject_and_discard database commit behavior; - let before = Database.inspect_sync database |> T.require_ok ~behavior in - Database.close database |> T.require_ok ~behavior; - let reopened = Database.open_ ~sw dependencies (inspect_mirror support behavior) - ~graph_name:"empty-revision" |> T.require_ok ~behavior in - let after = Database.inspect_sync reopened |> T.require_ok ~behavior in - T.require (sync_token_equal (sync_view_token before) (sync_view_token after)) - "empty queue must retain its durable sync revision"; - Database.close reopened |> T.require_ok ~behavior) + let _, commit = append_outbox_save database 1 behavior in + reject_and_discard database commit behavior; + let before = Database.inspect_sync database |> T.require_ok ~behavior in + Database.close database |> T.require_ok ~behavior; + let reopened = + Database.open_ + ~sw + dependencies + (inspect_mirror support behavior) + ~graph_name:"empty-revision" + |> T.require_ok ~behavior + in + let after = Database.inspect_sync reopened |> T.require_ok ~behavior in + T.require + (sync_token_equal (sync_view_token before) (sync_view_token after)) + "empty queue must retain its durable sync revision"; + Database.close reopened |> T.require_ok ~behavior) ;; let incremental_outbox_unrelated_authoritative () = let behavior = "incremental outbox unrelated authoritative" in - with_outbox_write_fixture behavior + with_outbox_write_fixture + behavior (fun ~sw:_ ~dependencies:_ ~support:_ ~path:_ ~sqlite database -> - let commits = List.init 32 (fun n -> snd (append_outbox_save database (n + 1) behavior)) in - let ids = List.map (fun (commit : local_commit) -> commit.mutation_id) commits in - ignore (apply_outbox_event database (Submit_group ids) behavior); - let before = persisted_outbox sqlite behavior in - reset_outbox_write_count sqlite; - let module Transit = Transit_core.Json in - let module Codec = Transit_native.Transit.Json in - let wire = - Transit.Array - [ Transit.Array - [ Transit.Keyword "db/add" - ; Transit.Array [ Transit.Keyword "block/uuid" - ; Transit.Uuid (Logseq_db_types.Graph_types.Uuid.to_string T.page_uuid) ] - ; Transit.Keyword "audit/unrelated" - ; Transit.Int 1 ] ] - |> Codec.to_string ~mode:Codec.Verbose - |> encoded_transaction_of_string ~maximum_bytes:4096 |> T.require_ok ~behavior - in - let batch = authoritative_batch ~maximum_count:16 ~maximum_bytes:4096 - ~transactions:[ authoritative_transaction ~cursor:(server_cursor 1) ~transaction:wire ] - ~through:(server_cursor 1) ~checksum:None |> T.require_ok ~behavior in - let view = Database.inspect_sync database |> T.require_ok ~behavior in - let prepared, request = Database.begin_authoritative database - ~expected:(sync_view_token view) batch |> T.require_ok ~behavior in - T.require (Option.is_none request) "unrelated transaction unexpectedly needs crypto"; - (match Database.apply_authoritative database prepared ~decrypted:None |> T.require_ok ~behavior with - | Authoritative_applied _ -> () - | _ -> Alcotest.fail "unrelated authoritative batch was deferred"); - Alcotest.(check int) "unrelated checkpoint does not rewrite frozen submissions" 0 - (outbox_write_count sqlite); - Alcotest.(check (list string)) "frozen submission payloads survive unchanged" before - (persisted_outbox sqlite behavior)) + let commits = + List.init 32 (fun n -> snd (append_outbox_save database (n + 1) behavior)) + in + let ids = List.map (fun (commit : local_commit) -> commit.mutation_id) commits in + ignore (apply_outbox_event database (Submit_group ids) behavior); + let before = persisted_outbox sqlite behavior in + reset_outbox_write_count sqlite; + let module Transit = Transit_core.Json in + let module Codec = Transit_native.Transit.Json in + let wire = + Transit.Array + [ Transit.Array + [ Transit.Keyword "db/add" + ; Transit.Array + [ Transit.Keyword "block/uuid" + ; Transit.Uuid (Logseq_db_types.Graph_types.Uuid.to_string T.page_uuid) + ] + ; Transit.Keyword "audit/unrelated" + ; Transit.Int 1 + ] + ] + |> Codec.to_string ~mode:Codec.Verbose + |> encoded_transaction_of_string ~maximum_bytes:4096 + |> T.require_ok ~behavior + in + let batch = + authoritative_batch + ~maximum_count:16 + ~maximum_bytes:4096 + ~transactions: + [ authoritative_transaction ~cursor:(server_cursor 1) ~transaction:wire ] + ~through:(server_cursor 1) + ~checksum:None + |> T.require_ok ~behavior + in + let view = Database.inspect_sync database |> T.require_ok ~behavior in + let prepared, request = + Database.begin_authoritative database ~expected:(sync_view_token view) batch + |> T.require_ok ~behavior + in + T.require + (Option.is_none request) + "unrelated transaction unexpectedly needs crypto"; + (match + Database.apply_authoritative database prepared ~decrypted:None + |> T.require_ok ~behavior + with + | Authoritative_applied _ -> () + | _ -> Alcotest.fail "unrelated authoritative batch was deferred"); + Alcotest.(check int) + "unrelated checkpoint does not rewrite frozen submissions" + 0 + (outbox_write_count sqlite); + Alcotest.(check (list string)) + "frozen submission payloads survive unchanged" + before + (persisted_outbox sqlite behavior)) ;; let incremental_outbox_legacy_migration () = let behavior = "incremental outbox legacy migration" in - with_outbox_write_fixture behavior + with_outbox_write_fixture + behavior (fun ~sw ~dependencies ~support ~path:_ ~sqlite database -> - ignore (append_outbox_save database 1 behavior); - ignore (append_outbox_save database 2 behavior); - let records = persisted_outbox sqlite behavior in - Database.close database |> T.require_ok ~behavior; + ignore (append_outbox_save database 1 behavior); + ignore (append_outbox_save database 2 behavior); + let records = persisted_outbox sqlite behavior in + Database.close database |> T.require_ok ~behavior; Sqlite3.Rc.check (Sqlite3.exec sqlite "DROP TABLE sync_outbox; DROP TABLE IF EXISTS sync_outbox_metadata; CREATE \ TABLE sync_outbox(position INTEGER PRIMARY KEY, record TEXT NOT NULL)"); - let legacy = List.map - (fun record -> match Yojson.Safe.from_string record with - | `Assoc fields -> `Assoc (("sync_revision", `Int 57) :: List.remove_assoc "sync_revision" fields) - |> Yojson.Safe.to_string - | _ -> Alcotest.fail "legacy fixture omitted record JSON") records in - Logseq_db_storage.Sync_outbox_store.replace_database sqlite legacy |> T.require_ok ~behavior; - let inspection = inspect_mirror support behavior in - let reopened = Database.open_ ~sw dependencies inspection ~graph_name:"legacy-queue" - |> T.require_ok ~behavior in - let view = Database.inspect_sync reopened |> T.require_ok ~behavior in - let expected = sync_token_of_string "sync-token:v1:57" |> T.require_ok ~behavior in - T.require (sync_token_equal expected (sync_view_token view)) "legacy migration lost global revision"; - ignore (append_outbox_save reopened 3 behavior); - Database.close reopened |> T.require_ok ~behavior; - let again = Database.open_ ~sw dependencies (inspect_mirror support behavior) - ~graph_name:"migrated-queue" |> T.require_ok ~behavior in - Alcotest.(check int) "migrated and new mutations all recover" 3 - (List.length (sync_view_submissions (Database.inspect_sync again |> T.require_ok ~behavior))); - T.require - (sync_token_equal (sync_token_of_string "sync-token:v1:58" |> T.require_ok ~behavior) - (sync_view_token (Database.inspect_sync again |> T.require_ok ~behavior))) - "migrated metadata did not advance atomically"; - Database.close again |> T.require_ok ~behavior) + let legacy = + List.map + (fun record -> + match Yojson.Safe.from_string record with + | `Assoc fields -> + `Assoc + (("sync_revision", `Int 57) :: List.remove_assoc "sync_revision" fields) + |> Yojson.Safe.to_string + | _ -> Alcotest.fail "legacy fixture omitted record JSON") + records + in + Logseq_db_storage.Sync_outbox_store.replace_database sqlite legacy + |> T.require_ok ~behavior; + let inspection = inspect_mirror support behavior in + let reopened = + Database.open_ ~sw dependencies inspection ~graph_name:"legacy-queue" + |> T.require_ok ~behavior + in + let view = Database.inspect_sync reopened |> T.require_ok ~behavior in + let expected = sync_token_of_string "sync-token:v1:57" |> T.require_ok ~behavior in + T.require + (sync_token_equal expected (sync_view_token view)) + "legacy migration lost global revision"; + ignore (append_outbox_save reopened 3 behavior); + Database.close reopened |> T.require_ok ~behavior; + let again = + Database.open_ + ~sw + dependencies + (inspect_mirror support behavior) + ~graph_name:"migrated-queue" + |> T.require_ok ~behavior + in + Alcotest.(check int) + "migrated and new mutations all recover" + 3 + (List.length + (sync_view_submissions + (Database.inspect_sync again |> T.require_ok ~behavior))); + T.require + (sync_token_equal + (sync_token_of_string "sync-token:v1:58" |> T.require_ok ~behavior) + (sync_view_token (Database.inspect_sync again |> T.require_ok ~behavior))) + "migrated metadata did not advance atomically"; + Database.close again |> T.require_ok ~behavior) ;; let incremental_outbox_migration_writer_lock () = let behavior = "incremental outbox migration writer lock" in - with_outbox_write_fixture behavior + with_outbox_write_fixture + behavior (fun ~sw ~dependencies ~support ~path:_ ~sqlite database -> - ignore (append_outbox_save database 1 behavior); - let legacy = persisted_outbox sqlite behavior in - Database.close database |> T.require_ok ~behavior; + ignore (append_outbox_save database 1 behavior); + let legacy = persisted_outbox sqlite behavior in + Database.close database |> T.require_ok ~behavior; Sqlite3.Rc.check (Sqlite3.exec sqlite "DROP TABLE sync_outbox; DROP TABLE IF EXISTS sync_outbox_metadata; CREATE \ TABLE sync_outbox(position INTEGER PRIMARY KEY, record TEXT NOT NULL)"); - Logseq_db_storage.Sync_outbox_store.replace_database sqlite legacy |> T.require_ok ~behavior; - Sqlite3.Rc.check (Sqlite3.exec sqlite "BEGIN IMMEDIATE"); - Fun.protect ~finally:(fun () -> Sqlite3.Rc.check (Sqlite3.exec sqlite "ROLLBACK")) - (fun () -> match Database.open_ ~sw dependencies (inspect_mirror support behavior) - ~graph_name:"locked-migration" with - | Error _ -> () - | Ok opened -> ignore (Database.close opened); - Alcotest.fail "write-required migration succeeded under another writer lock"); - Alcotest.(check (list string)) "failed migration retains complete legacy rows" legacy - (persisted_outbox sqlite behavior); - let reopened = Database.open_ ~sw dependencies (inspect_mirror support behavior) - ~graph_name:"migration-retry" |> T.require_ok ~behavior in - Alcotest.(check int) "migration retries without losing a row" 1 - (List.length (sync_view_submissions (Database.inspect_sync reopened |> T.require_ok ~behavior))); - Database.close reopened |> T.require_ok ~behavior) + Logseq_db_storage.Sync_outbox_store.replace_database sqlite legacy + |> T.require_ok ~behavior; + Sqlite3.Rc.check (Sqlite3.exec sqlite "BEGIN IMMEDIATE"); + Fun.protect + ~finally:(fun () -> Sqlite3.Rc.check (Sqlite3.exec sqlite "ROLLBACK")) + (fun () -> + match + Database.open_ + ~sw + dependencies + (inspect_mirror support behavior) + ~graph_name:"locked-migration" + with + | Error _ -> () + | Ok opened -> + ignore (Database.close opened); + Alcotest.fail "write-required migration succeeded under another writer lock"); + Alcotest.(check (list string)) + "failed migration retains complete legacy rows" + legacy + (persisted_outbox sqlite behavior); + let reopened = + Database.open_ + ~sw + dependencies + (inspect_mirror support behavior) + ~graph_name:"migration-retry" + |> T.require_ok ~behavior + in + Alcotest.(check int) + "migration retries without losing a row" + 1 + (List.length + (sync_view_submissions + (Database.inspect_sync reopened |> T.require_ok ~behavior))); + Database.close reopened |> T.require_ok ~behavior) ;; (* Database ownership cannot interpose a writer between its read and migration through a pure event. Exercise the public storage migration at that boundary. *) let incremental_outbox_migration_rejects_stale_rows () = let behavior = "incremental outbox migration rejects stale rows" in - with_outbox_write_fixture behavior + with_outbox_write_fixture + behavior (fun ~sw:_ ~dependencies:_ ~support:_ ~path:_ ~sqlite database -> - ignore (append_outbox_save database 1 behavior); - ignore (append_outbox_save database 2 behavior); - let all_records = persisted_outbox sqlite behavior in - Database.close database |> T.require_ok ~behavior; + ignore (append_outbox_save database 1 behavior); + ignore (append_outbox_save database 2 behavior); + let all_records = persisted_outbox sqlite behavior in + Database.close database |> T.require_ok ~behavior; Sqlite3.Rc.check (Sqlite3.exec sqlite "DROP TABLE sync_outbox; DROP TABLE sync_outbox_metadata; CREATE TABLE \ sync_outbox(position INTEGER PRIMARY KEY, record TEXT NOT NULL)"); - let module Store = Logseq_db_storage.Sync_outbox_store in - Store.replace_database sqlite [ List.hd all_records ] |> T.require_ok ~behavior; - let validated = Store.read_database sqlite |> T.require_ok ~behavior in - let rows_of records = List.map - (fun record -> - let open Yojson.Safe.Util in - let json = Yojson.Safe.from_string record in - let mutation_id = - json |> member "mutation" |> to_list |> fun parts -> - List.nth parts 1 |> member "mutation_id" |> to_string - in - Store.{ mutation_id - ; sequence = json |> member "sequence" |> to_int - ; record }) records in - let rows = rows_of validated in - Store.replace_database sqlite all_records |> T.require_ok ~behavior; - (match Store.migrate_database sqlite ~revision:0 ~expected_records:validated rows with - | Error _ -> () - | Ok () -> Alcotest.fail "migration discarded a row added after its validated snapshot"); - Alcotest.(check (list string)) "stale migration preserves all later rows" all_records - (Store.read_database sqlite |> T.require_ok ~behavior); - T.require (Store.read_revision sqlite |> T.require_ok ~behavior = None) - "stale migration must roll back its format change"; - Store.migrate_database sqlite ~revision:0 ~expected_records:all_records (rows_of all_records) - |> T.require_ok ~behavior; - (match Store.migrate_database sqlite ~revision:0 ~expected_records:validated rows with - | Error _ -> () - | Ok () -> Alcotest.fail "matching revision concealed stale incremental rows"); - Alcotest.(check (list string)) "incremental stale restore preserves all rows" all_records - (Store.read_database sqlite |> T.require_ok ~behavior)) + let module Store = Logseq_db_storage.Sync_outbox_store in + Store.replace_database sqlite [ List.hd all_records ] |> T.require_ok ~behavior; + let validated = Store.read_database sqlite |> T.require_ok ~behavior in + let rows_of records = + List.map + (fun record -> + let open Yojson.Safe.Util in + let json = Yojson.Safe.from_string record in + let mutation_id = + json + |> member "mutation" + |> to_list + |> fun parts -> List.nth parts 1 |> member "mutation_id" |> to_string + in + Store. + { mutation_id; sequence = json |> member "sequence" |> to_int; record }) + records + in + let rows = rows_of validated in + Store.replace_database sqlite all_records |> T.require_ok ~behavior; + (match + Store.migrate_database sqlite ~revision:0 ~expected_records:validated rows + with + | Error _ -> () + | Ok () -> + Alcotest.fail "migration discarded a row added after its validated snapshot"); + Alcotest.(check (list string)) + "stale migration preserves all later rows" + all_records + (Store.read_database sqlite |> T.require_ok ~behavior); + T.require + (Store.read_revision sqlite |> T.require_ok ~behavior = None) + "stale migration must roll back its format change"; + Store.migrate_database + sqlite + ~revision:0 + ~expected_records:all_records + (rows_of all_records) + |> T.require_ok ~behavior; + (match + Store.migrate_database sqlite ~revision:0 ~expected_records:validated rows + with + | Error _ -> () + | Ok () -> Alcotest.fail "matching revision concealed stale incremental rows"); + Alcotest.(check (list string)) + "incremental stale restore preserves all rows" + all_records + (Store.read_database sqlite |> T.require_ok ~behavior)) ;; let cases = - [ Alcotest.test_case "incremental outbox appends 1" `Quick (incremental_outbox_appends 1) - ; Alcotest.test_case "incremental outbox appends 32" `Quick (incremental_outbox_appends 32) - ; Alcotest.test_case "incremental outbox appends 1000" `Quick (incremental_outbox_appends 1000) - ; Alcotest.test_case "incremental outbox unchanged rows" `Quick incremental_outbox_preserves_unchanged_rows - ; Alcotest.test_case "incremental outbox No_change and identity" `Quick incremental_outbox_no_change_and_identity - ; Alcotest.test_case "incremental outbox batch updates" `Quick incremental_outbox_batch_updates - ; Alcotest.test_case "incremental outbox middle delete and empty revision" `Quick incremental_outbox_middle_delete_and_empty_revision - ; Alcotest.test_case "incremental outbox legacy migration" `Quick incremental_outbox_legacy_migration - ; Alcotest.test_case "incremental outbox empty revision" `Quick incremental_outbox_empty_revision - ; Alcotest.test_case "incremental outbox unrelated authoritative" `Quick incremental_outbox_unrelated_authoritative - ; Alcotest.test_case "incremental outbox migration writer lock" `Quick incremental_outbox_migration_writer_lock - ; Alcotest.test_case "incremental outbox migration rejects stale rows" `Quick incremental_outbox_migration_rejects_stale_rows + [ Alcotest.test_case + "incremental outbox appends 1" + `Quick + (incremental_outbox_appends 1) + ; Alcotest.test_case + "incremental outbox appends 32" + `Quick + (incremental_outbox_appends 32) + ; Alcotest.test_case + "incremental outbox appends 1000" + `Quick + (incremental_outbox_appends 1000) + ; Alcotest.test_case + "incremental outbox unchanged rows" + `Quick + incremental_outbox_preserves_unchanged_rows + ; Alcotest.test_case + "incremental outbox No_change and identity" + `Quick + incremental_outbox_no_change_and_identity + ; Alcotest.test_case + "incremental outbox batch updates" + `Quick + incremental_outbox_batch_updates + ; Alcotest.test_case + "incremental outbox middle delete and empty revision" + `Quick + incremental_outbox_middle_delete_and_empty_revision + ; Alcotest.test_case + "incremental outbox legacy migration" + `Quick + incremental_outbox_legacy_migration + ; Alcotest.test_case + "incremental outbox empty revision" + `Quick + incremental_outbox_empty_revision + ; Alcotest.test_case + "incremental outbox unrelated authoritative" + `Quick + incremental_outbox_unrelated_authoritative + ; Alcotest.test_case + "incremental outbox migration writer lock" + `Quick + incremental_outbox_migration_writer_lock + ; Alcotest.test_case + "incremental outbox migration rejects stale rows" + `Quick + incremental_outbox_migration_rejects_stale_rows ; Alcotest.test_case "legacy storage payloads roundtrip" `Quick @@ -3152,6 +4148,14 @@ let cases = "snapshot activation installs a queryable mirror" `Quick snapshot_activation_installs_queryable_mirror + ; Alcotest.test_case + "snapshot crypto batches respect item capacity" + `Quick + snapshot_crypto_batches_respect_item_capacity + ; Alcotest.test_case + "snapshot crypto batches tighten caller bounds" + `Quick + snapshot_crypto_batches_tighten_caller_bounds ; Alcotest.test_case "snapshot cancellation removes temporary artifacts" `Quick @@ -3176,6 +4180,10 @@ let cases = "snapshot checksum independent vectors" `Quick checksum_vectors_preserve_snapshot_semantics + ; Alcotest.test_case + "incremental checksum matches snapshot verification" + `Quick + incremental_checksum_matches_snapshot_verification ; Alcotest.test_case "unused preparation checksum defers invalid UTF-8" `Quick @@ -3215,9 +4223,12 @@ let cases = duplicate_outbox_member_fails_closed ; Alcotest.test_case "noncanonical outbox payload fails closed" - `Quick (noncanonical_outbox_payload_fails_closed ~payload:"tampered") - ; Alcotest.test_case "empty outgoing transaction stays corrupt" `Quick - (noncanonical_outbox_payload_fails_closed ~payload:"[]") + `Quick + (noncanonical_outbox_payload_fails_closed ~payload:"tampered") + ; Alcotest.test_case + "empty outgoing transaction stays corrupt" + `Quick + (noncanonical_outbox_payload_fails_closed ~payload:"[]") ; Alcotest.test_case "inconsistent origin evidence fails closed" `Quick @@ -3246,7 +4257,11 @@ let cases = ; Alcotest.test_case "authoritative checkpoint and root survive reopen" `Quick - authoritative_checkpoint_and_root_survive_reopen + (authoritative_checkpoint_and_root_survive_reopen ~mismatch:false) + ; Alcotest.test_case + "checksum mismatch preserves durable authoritative state" + `Quick + (authoritative_checkpoint_and_root_survive_reopen ~mismatch:true) ; Alcotest.test_case "outbox persistence failure is atomic" `Quick @@ -3328,6 +4343,21 @@ let receipt_history_open_is_bounded () = ~finally:(fun () -> ignore (Sqlite3.db_close sqlite)) (fun () -> Store.initialize_database sqlite |> T.require_ok ~behavior; + let empty_receipt_open_bytes = + Eio_main.run (fun _ -> + Eio.Switch.run (fun sw -> + let dependencies, _, inspection = + location_and_inspection support behavior + in + Gc.compact (); + let before = Gc.allocated_bytes () in + let database = + Database.open_ ~sw dependencies inspection ~graph_name:"receipt-history" + |> T.require_ok ~behavior + in + Database.close database |> T.require_ok ~behavior; + Gc.allocated_bytes () -. before)) + in Sqlite3.Rc.check (Sqlite3.exec sqlite "BEGIN"); for n = 1 to 50_000 do let id = Printf.sprintf "10000000-0000-4000-8000-%012d" n in @@ -3382,11 +4412,14 @@ let receipt_history_open_is_bounded () = let allocated_bytes = Gc.allocated_bytes () -. allocated_before in let peak_bytes = 8 * (!peak - baseline) in Printf.printf - "receipt history rows=100000 peak_live_growth_bytes=%d allocated_bytes=%.0f\n%!" + "receipt history rows=100000 peak_live_growth_bytes=%d \ + allocated_bytes=%.0f empty_receipt_open_bytes=%.0f\n\ + %!" peak_bytes - allocated_bytes; + allocated_bytes + empty_receipt_open_bytes; T.require - (allocated_bytes < 16. *. 1024. *. 1024.) + (allocated_bytes -. empty_receipt_open_bytes < 16. *. 1024. *. 1024.) "opening decoded historical receipt payloads"; T.require (peak_bytes < 4 * 1024 * 1024) @@ -3421,19 +4454,21 @@ let receipt_history_open_is_bounded () = ~finally:(fun () -> Database.close opened |> T.require_ok ~behavior) (fun () -> if table = "overlay_mutation_receipts" - then + then ( match Database.inspect_local_mutation opened (Save_block - { mutation_id = T.uuid "10000000-0000-4000-8000-000000025000" + { mutation_id = + T.uuid "10000000-0000-4000-8000-000000025000" ; block = T.authoritative_block_uuid ; title = "Must not replay a corrupt receipt" }) with | Error (Local_commit_persistence_failed _) -> () - | Error _ -> Alcotest.fail "receipt lookup lost the storage cause" - | Ok _ -> Alcotest.fail "corrupt receipt was treated as missing"); + | Error _ -> + Alcotest.fail "receipt lookup lost the storage cause" + | Ok _ -> Alcotest.fail "corrupt receipt was treated as missing")); (match Database.collect_garbage (inspect_mirror support behavior) with | Error _ -> () | Ok _ -> Alcotest.fail "garbage collection accepted invalid receipt"); @@ -3458,23 +4493,33 @@ let receipt_open_errors_are_results () = let behavior = "receipt open failure is a result" in T.with_temp_directory "receipt-open-failure-" (fun support -> let path = T.seed_mirror support in - Eio_main.run (fun _ -> Eio.Switch.run (fun sw -> - let dependencies, _, inspection = location_and_inspection support behavior in - let database = Database.open_ ~sw dependencies inspection ~graph_name:"receipt-open" - |> T.require_ok ~behavior in - let moved = path ^ ".moved" in - Fun.protect - ~finally:(fun () -> - Unix.rename moved path; - Database.close database |> T.require_ok ~behavior) - (fun () -> - Unix.rename path moved; - match Database.inspect_local_mutation database - (Save_block {mutation_id=T.uuid "10000000-0000-4000-8000-000000099999"; - block=T.authoritative_block_uuid;title="Must not replay"}) with - | Error (Local_commit_persistence_failed _) -> () - | Error _ -> Alcotest.fail "receipt open failure lost persistence classification" - | Ok _ -> Alcotest.fail "receipt open failure became absent receipt")))) + Eio_main.run (fun _ -> + Eio.Switch.run (fun sw -> + let dependencies, _, inspection = location_and_inspection support behavior in + let database = + Database.open_ ~sw dependencies inspection ~graph_name:"receipt-open" + |> T.require_ok ~behavior + in + let moved = path ^ ".moved" in + Fun.protect + ~finally:(fun () -> + Unix.rename moved path; + Database.close database |> T.require_ok ~behavior) + (fun () -> + Unix.rename path moved; + match + Database.inspect_local_mutation + database + (Save_block + { mutation_id = T.uuid "10000000-0000-4000-8000-000000099999" + ; block = T.authoritative_block_uuid + ; title = "Must not replay" + }) + with + | Error (Local_commit_persistence_failed _) -> () + | Error _ -> + Alcotest.fail "receipt open failure lost persistence classification" + | Ok _ -> Alcotest.fail "receipt open failure became absent receipt")))) ;; let receipt_lookup_schema_errors_are_results () = @@ -3485,10 +4530,13 @@ let receipt_lookup_schema_errors_are_results () = ~finally:(fun () -> ignore (Sqlite3.db_close sqlite)) (fun () -> Sqlite3.Rc.check - (Sqlite3.exec sqlite ("CREATE TABLE " ^ table ^ " (" ^ key_column ^ " TEXT PRIMARY KEY)")); + (Sqlite3.exec + sqlite + ("CREATE TABLE " ^ table ^ " (" ^ key_column ^ " TEXT PRIMARY KEY)")); match read sqlite (String.make 64 'a') with | Error message -> - Alcotest.(check string) "missing receipt column cause is preserved" + Alcotest.(check string) + "missing receipt column cause is preserved" "Sqlite3.Error(\"Sqlite3.prepare: no such column: receipt\")" message | Ok _ -> Alcotest.fail "malformed receipt table was treated as an absent key") @@ -3533,7 +4581,6 @@ let sqlite_full_int64_checkpoint () = "SQLite update preserves the complete cursor") ;; - let storage_corruption_after_open operation () = let module Storage = Logseq_db_storage.Logseq_sqlite_storage in let module D = Logseq_db_types.Failure_diagnostic in @@ -3542,19 +4589,30 @@ let storage_corruption_after_open operation () = T.copy_storage_fixture path; let connection = Storage.open_database path |> Result.get_ok in let callbacks = Storage.connection_callbacks connection in - Fun.protect ~finally:(fun () -> ignore (Storage.close callbacks)) (fun () -> - let sqlite = Sqlite3.db_open path in - Sqlite3.exec sqlite "UPDATE kvs SET content = '{invalid' WHERE addr = 0" |> Sqlite3.Rc.check; - T.require (Sqlite3.db_close sqlite) "close corrupting writer"; - let result = match operation with - | `Startup -> Result.map (fun _ -> ()) (Storage.startup_metadata connection) - | `Garbage -> Result.map (fun _ -> ()) (Storage.garbage_stats callbacks) in - match result with - | Ok () -> Alcotest.fail "corruption was ignored" - | Error error -> - let diagnostic = Storage.diagnostic_error ~operation:"attachGraph" error |> D.decode |> Option.get in - Alcotest.(check string) "read returns concrete typed cause" - "codecMalformedTransit" (D.reason diagnostic))) + Fun.protect + ~finally:(fun () -> ignore (Storage.close callbacks)) + (fun () -> + let sqlite = Sqlite3.db_open path in + Sqlite3.exec sqlite "UPDATE kvs SET content = '{invalid' WHERE addr = 0" + |> Sqlite3.Rc.check; + T.require (Sqlite3.db_close sqlite) "close corrupting writer"; + let result = + match operation with + | `Startup -> Result.map (fun _ -> ()) (Storage.startup_metadata connection) + | `Garbage -> Result.map (fun _ -> ()) (Storage.garbage_stats callbacks) + in + match result with + | Ok () -> Alcotest.fail "corruption was ignored" + | Error error -> + let diagnostic = + Storage.diagnostic_error ~operation:"attachGraph" error + |> D.decode + |> Option.get + in + Alcotest.(check string) + "read returns concrete typed cause" + "codecMalformedTransit" + (D.reason diagnostic))) ;; (* Commit_batch owns the sequence of persistence stages; no reducer executes @@ -3567,29 +4625,50 @@ let storage_commit_retains_stage stage expected_reason () = T.copy_storage_fixture path; let connection = Storage.open_database path |> Result.get_ok in let base = Storage.connection_callbacks connection in - Fun.protect ~finally:(fun () -> ignore (Storage.close base)) (fun () -> - let fail _ = Error "SQLITE_FULL" in - let callbacks = match stage with - | `Metadata -> { base with upsert_sync_metadata = fail } - | `Outbox -> { base with replace_sync_outbox = fail } - | `Mutation -> { base with upsert_mutation_receipts = fail } - | `Terminal -> { base with upsert_terminal_batch_receipts = fail } in - let metadata = Logseq_db_types.Sync_checkpoint.create - ~graph_id:T.graph_uuid ~schema:Logseq_db_types.Graph_types.{ major=1; minor=0 } - ~applied_server_t:(server_cursor_of_int 0) ~checksum:"0000000000000000" - |> Result.get_ok in - let batch = Storage.{ writes=[]; - sync_metadata=(if stage=`Metadata then Some metadata else None); - sync_outbox=(if stage=`Outbox then Some [] else None); - mutation_receipts=(if stage=`Mutation then Some [] else None); - terminal_batch_receipts=(if stage=`Terminal then Some [] else None) } in - match Storage.commit_batch callbacks batch with - | Ok () -> Alcotest.fail "failed persistence stage was ignored" - | Error error -> - let diagnostic = Storage.diagnostic_error ~operation:"commitLocal" error - |> D.decode |> Option.get in - Alcotest.(check string) "failing commit stage" expected_reason (D.reason diagnostic); - Alcotest.(check string) "underlying SQLite cause" "sqlite-full" (D.source_class diagnostic))) + Fun.protect + ~finally:(fun () -> ignore (Storage.close base)) + (fun () -> + let fail _ = Error "SQLITE_FULL" in + let callbacks = + match stage with + | `Metadata -> { base with upsert_sync_metadata = fail } + | `Outbox -> { base with replace_sync_outbox = fail } + | `Mutation -> { base with upsert_mutation_receipts = fail } + | `Terminal -> { base with upsert_terminal_batch_receipts = fail } + in + let metadata = + Logseq_db_types.Sync_checkpoint.create + ~graph_id:T.graph_uuid + ~schema:Logseq_db_types.Graph_types.{ major = 1; minor = 0 } + ~applied_server_t:(server_cursor_of_int 0) + ~checksum:"0000000000000000" + |> Result.get_ok + in + let batch = + Storage. + { writes = [] + ; sync_metadata = (if stage = `Metadata then Some metadata else None) + ; sync_outbox = (if stage = `Outbox then Some [] else None) + ; mutation_receipts = (if stage = `Mutation then Some [] else None) + ; terminal_batch_receipts = (if stage = `Terminal then Some [] else None) + } + in + match Storage.commit_batch callbacks batch with + | Ok () -> Alcotest.fail "failed persistence stage was ignored" + | Error error -> + let diagnostic = + Storage.diagnostic_error ~operation:"commitLocal" error + |> D.decode + |> Option.get + in + Alcotest.(check string) + "failing commit stage" + expected_reason + (D.reason diagnostic); + Alcotest.(check string) + "underlying SQLite cause" + "sqlite-full" + (D.source_class diagnostic))) ;; let codec_preserves_structural_cause () = @@ -3598,7 +4677,9 @@ let codec_preserves_structural_cause () = | Ok _ -> Alcotest.fail "unknown storage map was accepted" | Error error -> let diagnostic = Codec.diagnostic_error error |> D.decode |> Option.get in - Alcotest.(check string) "concrete structural cause" "storage-payload-map-shape" + Alcotest.(check string) + "concrete structural cause" + "storage-payload-map-shape" (D.source_class diagnostic) ;; @@ -3610,39 +4691,92 @@ let storage_rollback_retains_primary () = T.copy_storage_fixture path; let connection = Storage.open_database path |> Result.get_ok in let base = Storage.connection_callbacks connection in - Fun.protect ~finally:(fun () -> ignore (Storage.close base)) (fun () -> - let callbacks = { base with - upsert = (fun _ -> Error "SQLITE_FULL"); - rollback = (fun () -> base.rollback (); failwith "SQLITE_BUSY") } in - let batch = Storage.{ writes=[{address="0";payload="";addresses=[]}]; - sync_metadata=None;sync_outbox=None;mutation_receipts=None;terminal_batch_receipts=None } in - match Storage.commit_batch callbacks batch with - | Error (Storage.Write_failed _ as error) -> - let diagnostic = Storage.diagnostic_error ~operation:"commitLocal" error |> D.decode |> Option.get in - Alcotest.(check string) "primary write cause" "sqlite-full" (D.source_class diagnostic); - let secondary = D.related diagnostic |> Option.get in - Alcotest.(check string) "rollback stage" "storageRollbackFailed" (D.reason secondary); - Alcotest.(check string) "rollback cause" "sqlite-locked" (D.source_class secondary) - | _ -> Alcotest.fail "rollback replaced the primary typed write error")) + Fun.protect + ~finally:(fun () -> ignore (Storage.close base)) + (fun () -> + let callbacks = + { base with + upsert = (fun _ -> Error "SQLITE_FULL") + ; rollback = + (fun () -> + base.rollback (); + failwith "SQLITE_BUSY") + } + in + let batch = + Storage. + { writes = [ { address = "0"; payload = ""; addresses = [] } ] + ; sync_metadata = None + ; sync_outbox = None + ; mutation_receipts = None + ; terminal_batch_receipts = None + } + in + match Storage.commit_batch callbacks batch with + | Error (Storage.Write_failed _ as error) -> + let diagnostic = + Storage.diagnostic_error ~operation:"commitLocal" error + |> D.decode + |> Option.get + in + Alcotest.(check string) + "primary write cause" + "sqlite-full" + (D.source_class diagnostic); + let secondary = D.related diagnostic |> Option.get in + Alcotest.(check string) + "rollback stage" + "storageRollbackFailed" + (D.reason secondary); + Alcotest.(check string) + "rollback cause" + "sqlite-locked" + (D.source_class secondary) + | _ -> Alcotest.fail "rollback replaced the primary typed write error")) ;; let related_diagnostic_validation () = let module D = Logseq_db_types.Failure_diagnostic in - let primary = D.create ~operation:"closeDatabase" ~reason:"storageCheckpointFailed" - ~source:"SQLITE_FULL" ~cursor:Int64.max_int ~http_status:599 () in - let secondary = D.create ~operation:"closeDatabase" ~reason:"storageCloseFailed" - ~source:"SQLITE_BUSY" ~cursor:Int64.max_int ~http_status:599 () in + let primary = + D.create + ~operation:"closeDatabase" + ~reason:"storageCheckpointFailed" + ~source:"SQLITE_FULL" + ~cursor:Int64.max_int + ~http_status:599 + () + in + let secondary = + D.create + ~operation:"closeDatabase" + ~reason:"storageCloseFailed" + ~source:"SQLITE_BUSY" + ~cursor:Int64.max_int + ~http_status:599 + () + in let encoded = D.with_related primary secondary |> D.encode in T.require (String.length encoded <= 1024) "dual diagnostic exceeds wire budget"; - T.require (String.length (D.message (D.with_related primary secondary)) <= 512) + T.require + (String.length (D.message (D.with_related primary secondary)) <= 512) "dual diagnostic exceeds display budget"; let prefix = "journal-failure:v1:" in - let fields = match Yojson.Safe.from_string - (String.sub encoded (String.length prefix) (String.length encoded - String.length prefix)) with - | `Assoc fields -> fields | _ -> assert false in + let fields = + match + Yojson.Safe.from_string + (String.sub + encoded + (String.length prefix) + (String.length encoded - String.length prefix)) + with + | `Assoc fields -> fields + | _ -> assert false + in let reject fields = - T.require (D.decode (prefix ^ Yojson.Safe.to_string (`Assoc fields)) = None) - "unsafe related diagnostic was accepted" in + T.require + (D.decode (prefix ^ Yojson.Safe.to_string (`Assoc fields)) = None) + "unsafe related diagnostic was accepted" + in reject (("related", `String (D.encode secondary)) :: fields); reject (("unexpected", `String "SECRET") :: fields); reject (("related", `String "SECRET") :: List.remove_assoc "related" fields); @@ -3651,8 +4785,12 @@ let related_diagnostic_validation () = let diagnostic_unix_cause error expected () = let module D = Logseq_db_types.Failure_diagnostic in - let source = Printexc.to_string (Unix.Unix_error (error, "SECRET-operation", "SECRET-path")) in - let diagnostic = D.of_source ~operation:"workerOperation" ~reason:"persistenceFailed" source in + let source = + Printexc.to_string (Unix.Unix_error (error, "SECRET-operation", "SECRET-path")) + in + let diagnostic = + D.of_source ~operation:"workerOperation" ~reason:"persistenceFailed" source + in Alcotest.(check string) "concrete errno" expected (D.source_class diagnostic); let encoded = D.encode diagnostic in Alcotest.(check bool) "no exception argument" false (String.contains encoded 'S') @@ -3669,28 +4807,48 @@ let storage_close_retains_both_causes ~checkpoint_raises () = T.copy_storage_fixture path; let connection = Storage.open_database path |> Result.get_ok in let base = Storage.connection_callbacks connection in - let callbacks = { base with - checkpoint = (fun () -> if checkpoint_raises then failwith "SQLITE_FULL" else Error "SQLITE_FULL"); - close = (fun () -> ignore (Storage.close base); Error "SQLITE_BUSY") } in + let callbacks = + { base with + checkpoint = + (fun () -> + if checkpoint_raises then failwith "SQLITE_FULL" else Error "SQLITE_FULL") + ; close = + (fun () -> + ignore (Storage.close base); + Error "SQLITE_BUSY") + } + in let session = Session.create ~tail:[] ~callbacks in match Session.close session with | Error (Session.Persistence_failed encoded) -> - let diagnostic = match D.decode encoded with + let diagnostic = + match D.decode encoded with | Some value -> value - | None -> Alcotest.fail "dual failure destroyed the structured carrier" in - Alcotest.(check string) "checkpoint stage" "storageCheckpointFailed" (D.reason diagnostic); + | None -> Alcotest.fail "dual failure destroyed the structured carrier" + in + Alcotest.(check string) + "checkpoint stage" + "storageCheckpointFailed" + (D.reason diagnostic); Alcotest.(check string) "checkpoint cause" "sqlite-full" (D.source_class diagnostic); let secondary = D.related diagnostic |> Option.get in Alcotest.(check string) "close stage" "storageCloseFailed" (D.reason secondary); Alcotest.(check string) "close cause" "sqlite-locked" (D.source_class secondary); - let wrapped = D.of_source ~operation:"attachGraph" ~reason:"resourceFailed" encoded in - Alcotest.(check string) "upper wrapper retains secondary" "storageCloseFailed" + let wrapped = + D.of_source ~operation:"attachGraph" ~reason:"resourceFailed" encoded + in + Alcotest.(check string) + "upper wrapper retains secondary" + "storageCloseFailed" (D.reason (D.related wrapped |> Option.get)); let message = D.message diagnostic in let contains text = - let rec loop n = n + String.length text <= String.length message - && (String.sub message n (String.length text) = text || loop (n + 1)) in - loop 0 in + let rec loop n = + n + String.length text <= String.length message + && (String.sub message n (String.length text) = text || loop (n + 1)) + in + loop 0 + in T.require (contains "close" && contains "locked") "close cause was discarded" | _ -> Alcotest.fail "close errors were lost") ;; @@ -3716,44 +4874,85 @@ let storage_open_preserves_codec_cause content expected_reason () = | Error error -> let encoded = Storage.diagnostic_error ~operation:"attachGraph" error in let diagnostic = D.decode encoded |> Option.get in - Alcotest.(check string) "concrete codec cause survives storage boundary" - expected_reason (D.reason diagnostic); + Alcotest.(check string) + "concrete codec cause survives storage boundary" + expected_reason + (D.reason diagnostic); let rec contains_secret offset = offset + 6 <= String.length encoded && (String.sub encoded offset 6 = "SECRET" || contains_secret (offset + 1)) in - T.require (not (contains_secret 0)) - "diagnostic must not include synthetic secret") + T.require (not (contains_secret 0)) "diagnostic must not include synthetic secret") ;; let () = Alcotest.run "logseq_overlay_db storage" [ ( "storage error causes" - , [ Alcotest.test_case "codec structural cause" `Quick codec_preserves_structural_cause - ; Alcotest.test_case "rollback retains primary cause" `Quick storage_rollback_retains_primary - ; Alcotest.test_case "related diagnostic validates and bounds input" `Quick related_diagnostic_validation - ; Alcotest.test_case "startup corruption returns result" `Quick + , [ Alcotest.test_case + "codec structural cause" + `Quick + codec_preserves_structural_cause + ; Alcotest.test_case + "rollback retains primary cause" + `Quick + storage_rollback_retains_primary + ; Alcotest.test_case + "related diagnostic validates and bounds input" + `Quick + related_diagnostic_validation + ; Alcotest.test_case + "startup corruption returns result" + `Quick (storage_corruption_after_open `Startup) - ; Alcotest.test_case "garbage traversal corruption returns result" `Quick + ; Alcotest.test_case + "garbage traversal corruption returns result" + `Quick (storage_corruption_after_open `Garbage) - ; Alcotest.test_case "metadata commit stage" `Quick + ; Alcotest.test_case + "metadata commit stage" + `Quick (storage_commit_retains_stage `Metadata "storageMetadataFailed") - ; Alcotest.test_case "outbox commit stage" `Quick + ; Alcotest.test_case + "outbox commit stage" + `Quick (storage_commit_retains_stage `Outbox "storageOutboxFailed") - ; Alcotest.test_case "mutation receipt commit stage" `Quick + ; Alcotest.test_case + "mutation receipt commit stage" + `Quick (storage_commit_retains_stage `Mutation "storageMutationReceiptFailed") - ; Alcotest.test_case "terminal receipt commit stage" `Quick + ; Alcotest.test_case + "terminal receipt commit stage" + `Quick (storage_commit_retains_stage `Terminal "storageTerminalReceiptFailed") - ; Alcotest.test_case "Unix I/O failure" `Quick (diagnostic_unix_cause Unix.EIO "io-error") - ; Alcotest.test_case "Unix connection reset" `Quick (diagnostic_unix_cause Unix.ECONNRESET "network-reset") - ; Alcotest.test_case "close retains both failures" `Quick (storage_close_retains_both_causes ~checkpoint_raises:false) - ; Alcotest.test_case "throwing checkpoint still closes" `Quick + ; Alcotest.test_case + "Unix I/O failure" + `Quick + (diagnostic_unix_cause Unix.EIO "io-error") + ; Alcotest.test_case + "Unix connection reset" + `Quick + (diagnostic_unix_cause Unix.ECONNRESET "network-reset") + ; Alcotest.test_case + "close retains both failures" + `Quick + (storage_close_retains_both_causes ~checkpoint_raises:false) + ; Alcotest.test_case + "throwing checkpoint still closes" + `Quick (storage_close_retains_both_causes ~checkpoint_raises:true) - ; Alcotest.test_case "malformed Transit survives open" `Quick - (storage_open_preserves_codec_cause "{SECRET-synthetic-invalid" "codecMalformedTransit") - ; Alcotest.test_case "unsupported tag survives open" `Quick - (storage_open_preserves_codec_cause {|["~#SECRET-synthetic-tag","SECRET"]|} "codecUnsupportedTag") + ; Alcotest.test_case + "malformed Transit survives open" + `Quick + (storage_open_preserves_codec_cause + "{SECRET-synthetic-invalid" + "codecMalformedTransit") + ; Alcotest.test_case + "unsupported tag survives open" + `Quick + (storage_open_preserves_codec_cause + {|["~#SECRET-synthetic-tag","SECRET"]|} + "codecUnsupportedTag") ] ) ; ( "legacy status compatibility" , [ Alcotest.test_case @@ -3762,11 +4961,26 @@ let () = pre_uuid_status_outbox_preserves_identity ] ) ; ( "receipt history" - , [ Alcotest.test_case "bounded opening and fail-closed validation" `Quick receipt_history_open_is_bounded - ; Alcotest.test_case "writer schema failure is atomic" `Quick receipt_writer_schema_failure_is_atomic - ; Alcotest.test_case "writer schema errors are results" `Quick receipt_writer_schema_errors_are_results - ; Alcotest.test_case "lookup schema errors are results" `Quick receipt_lookup_schema_errors_are_results - ; Alcotest.test_case "open errors are results" `Quick receipt_open_errors_are_results + , [ Alcotest.test_case + "bounded opening and fail-closed validation" + `Quick + receipt_history_open_is_bounded + ; Alcotest.test_case + "writer schema failure is atomic" + `Quick + receipt_writer_schema_failure_is_atomic + ; Alcotest.test_case + "writer schema errors are results" + `Quick + receipt_writer_schema_errors_are_results + ; Alcotest.test_case + "lookup schema errors are results" + `Quick + receipt_lookup_schema_errors_are_results + ; Alcotest.test_case + "open errors are results" + `Quick + receipt_open_errors_are_results ] ) ; ( "cursor checkpoint" , [ Alcotest.test_case @@ -3781,8 +4995,10 @@ let () = ; "snapshot UUID metadata", snapshot_uuid_metadata_cases ; ( "mirror and durability" , cases - @ [ Alcotest.test_case "committed deletion survives unavailable warning output" `Quick - (retired_cleanup_retries_without_touching_a_new_mirror ~closed_stderr:true) + @ [ Alcotest.test_case + "committed deletion survives unavailable warning output" + `Quick + (retired_cleanup_retries_without_touching_a_new_mirror ~closed_stderr:true) ; Alcotest.test_case "legacy post-rename retirement replays" `Quick diff --git a/logseq_overlay_db/test/test_overlay_sync.ml b/logseq_overlay_db/test/test_overlay_sync.ml index 0d5d5365..b1bea3c9 100644 --- a/logseq_overlay_db/test/test_overlay_sync.ml +++ b/logseq_overlay_db/test/test_overlay_sync.ml @@ -104,9 +104,26 @@ let empty_batch_update_wire = let empty_authoritative_preserves_logical_state wire database = let before, projection, updated = empty_batch_state database in + let observed = ref false in let commit = - apply_empty_compatible_batch database (empty_authoritative_batch [ wire ]) + Eio.Fiber.first + (fun () -> + Database.await_sync_change database ~after:(sync_view_token before) + |> T.require_ok ~behavior:"nonlogical sync notification" + |> ignore; + observed := true; + Eio.Fiber.await_cancel ()) + (fun () -> + Eio.Fiber.yield (); + let commit = + apply_empty_compatible_batch database (empty_authoritative_batch [ wire ]) + in + for _turn = 1 to 20 do + Eio.Fiber.yield () + done; + commit) in + T.require !observed "nonlogical checkpoint failed to wake sync observer"; let after, after_projection, after_updated = empty_batch_state database in T.require (Server_cursor.equal (sync_view_checkpoint after) (server_cursor 1)) @@ -581,7 +598,9 @@ let block_precondition database block behavior = ~finally:(fun () -> Database.release_snapshot snapshot) (fun () -> let revision = - match Database.get_outline_blocks snapshot [ block ] |> T.require_ok ~behavior with + match + Database.get_outline_blocks snapshot [ block ] |> T.require_ok ~behavior + with | [ Present_block { revision; _ } ] | [ Missing_block { revision; _ } ] -> revision | _ -> Alcotest.fail "block precondition returned the wrong cardinality" @@ -1231,203 +1250,600 @@ let decrypt_submitted_request request behavior = request, plaintexts ;; -let atomic_append_submits_and_replays_complete_tree ?(remote_day=false) database = - let behavior="atomic append submission and authoritative echo" in +let reordered_echo_rejects_changed_uuid database = + let behavior = "reordered echo rejects changed UUID" in + let local = commit_insert database ~ordinal:7796 ~uuid:T.block_uuid behavior in + let submitted = submit_one database local behavior in + let sync = Database.inspect_sync database |> T.require_ok ~behavior in + let prepared, crypto = + Database.begin_outbox_transition + database + ~expected:(sync_view_token sync) + (Accept_group + { batch_id = submission_batch_id submitted + ; barrier = { through = server_cursor 1; checksum = checksum "0000000000000000" } + }) + |> T.require_ok ~behavior + in + T.require (Option.is_none crypto) "acceptance requested crypto"; + ignore + (Database.apply_outbox_transition database prepared ~encrypted:None + |> T.require_ok ~behavior); + let module J = Transit_core.Json in + let module Codec = Transit_native.Transit.Json in + let wire = + server_transaction ~mutation_id:local.mutation_id ~operation:"insert-blocks" submitted + |> Codec.of_string + |> function + | J.Array operations -> + J.Array + (List.rev_map + (function + | J.Array [ operation; entity; J.Keyword "block/uuid"; _; source ] -> + J.Array + [ operation + ; entity + ; J.Keyword "block/uuid" + ; J.Uuid (Logseq_db_types.Graph_types.Uuid.to_string T.child_uuid) + ; source + ] + | operation -> operation) + operations) + |> Codec.to_string ~mode:Codec.Verbose + | _ -> Alcotest.fail "insert echo is not an array" + in + let batch = + authoritative_batch + ~maximum_count:16 + ~maximum_bytes:65536 + ~transactions: + [ authoritative_transaction + ~cursor:(server_cursor 1) + ~transaction:(encoded ~maximum_bytes:65536 wire) + ] + ~through:(server_cursor 1) + ~checksum:None + |> T.require_ok ~behavior + in + let before = Database.inspect_sync database |> T.require_ok ~behavior in + let prepared, crypto = + Database.begin_authoritative database ~expected:(sync_view_token before) batch + |> T.require_ok ~behavior + in + let decrypted = + Option.map (fun request -> decrypt_submitted_request request behavior) crypto + in + (match Database.apply_authoritative database prepared ~decrypted with + | Error (Authoritative_integrity_failure _) -> () + | Error _ -> Alcotest.fail "changed UUID did not reach origin integrity validation" + | Ok _ -> Alcotest.fail "reordered echo accepted a changed UUID"); + let after = Database.inspect_sync database |> T.require_ok ~behavior in + T.require + (sync_view_checkpoint before = sync_view_checkpoint after + && sync_view_submissions before = sync_view_submissions after) + "identity rejection changed checkpoint or retired accepted evidence" +;; + +let atomic_append_submits_and_replays_complete_tree + ?(remote_day = false) + ?(reverse_operations = false) + database + = + let behavior = "atomic append submission and authoritative echo" in T.seed_task_statuses database; - let page=T.mutation_uuid 7790 and root_uuid=T.mutation_uuid 7791 and child_uuid=T.mutation_uuid 7792 in - let root:Types.append_node={uuid=root_uuid;title="Atomic wire root";task_status=Some(T.status_uuid 2); - children=Rrbvec.singleton{uuid=child_uuid;title="Atomic wire child";task_status=Some(T.status_uuid 3);children=Rrbvec.empty}} in - let mutation=Types.Append_journal{mutation_id=T.mutation_uuid 7793;page;title="2026-11-02";journal_day=20261102;root} in - let expected=Database.write_precondition ~blocks:[] ~pages:[] ~scopes:[] |> T.require_ok ~behavior in - let intent=Database.persist_local_intent database ~namespace:(String.make 64 'f') ~payload:"complete append wire" - ~expected mutation |> T.require_ok ~behavior in - let proof=Database.commit_local_intent database intent |> T.require_ok ~behavior in - let local=Database.intent_commit_outcome proof in - let resolved_page=if not remote_day then page else ( - let module J=Transit_core.Json in - let web=T.mutation_uuid 7794 in - let page_entity=J.String "remote-web-day" and child_entity=J.String "remote-first-entry" in - let add_page attribute value=T.status_add page_entity attribute value in - let add_child attribute value=T.status_add child_entity attribute value in - T.apply_test_transaction database ~cursor:2 - [add_page "block/uuid" (J.Uuid(Logseq_db_types.Graph_types.Uuid.to_string web)); - add_page "block/title" (J.String "Web day");add_page "block/name" (J.String "web-remote-day"); - add_page "block/journal-day" (J.Int 20261102); - add_child "block/uuid" (J.Uuid(Logseq_db_types.Graph_types.Uuid.to_string(T.mutation_uuid 7795))); - add_child "block/title" (J.String "Remote first entry");add_child "block/parent" page_entity; - add_child "block/page" page_entity;add_child "block/order" (J.String "a9")]; - let snapshot=Database.current_snapshot database |> T.require_ok ~behavior in - Fun.protect ~finally:(fun()->Database.release_snapshot snapshot)(fun()-> - T.require(match Database.get_outline_blocks snapshot [root_uuid] |> T.require_ok ~behavior with - |[Present_block{value;_}]->value.block.parent=web && String.compare value.block.order "a9">0 - |_->false) "queued append did not follow the Web day's existing tail"); - web) in - let echo_cursor=if remote_day then 3 else 2 in - let page_value ()= - let snapshot=Database.current_snapshot database |> T.require_ok ~behavior in - Fun.protect ~finally:(fun()->Database.release_snapshot snapshot)(fun()-> - match Database.get_pages snapshot [resolved_page] |> T.require_ok ~behavior with - |[Present_page{value;_}]->value|_->Alcotest.fail "atomic page disappeared") in - let before_submit=page_value() in - let submitted=submit_one database local behavior in - T.require(page_value()=before_submit) "transport submission changed complete append page DTO"; - T.require(List.length(submission_batch_wires submitted)=1) "append split one atomic action into separate outbox transactions"; - let wire=server_transaction ~mutation_id:local.mutation_id ~operation:"append-journal" submitted in - let barrier={through=server_cursor echo_cursor;checksum=checksum "1111111111111111"} in - let sync=Database.inspect_sync database |> T.require_ok ~behavior in - let prepared,crypto=Database.begin_outbox_transition database ~expected:(sync_view_token sync) - (Accept_group{batch_id=submission_batch_id submitted;barrier}) |> T.require_ok ~behavior in - T.require(crypto=None) "append acceptance requested extra crypto"; - ignore(Database.apply_outbox_transition database prepared ~encrypted:None |> T.require_ok ~behavior); - let batch=authoritative_batch ~maximum_count:16 ~maximum_bytes:65536 - ~transactions:[authoritative_transaction ~cursor:(server_cursor echo_cursor) ~transaction:(encoded ~maximum_bytes:65536 wire)] - ~through:(server_cursor echo_cursor) ~checksum:(Some barrier.checksum) |> T.require_ok ~behavior in - let sync=Database.inspect_sync database |> T.require_ok ~behavior in - let prepared,crypto=Database.begin_authoritative database ~expected:(sync_view_token sync)batch |> T.require_ok ~behavior in - let decrypted=Option.map(fun request->decrypt_submitted_request request behavior)crypto in - (match Database.apply_authoritative database prepared ~decrypted |> T.require_ok ~behavior with - | Authoritative_applied _->()|_->Alcotest.fail "atomic append echo was deferred"); - let snapshot=Database.current_snapshot database |> T.require_ok ~behavior in - Fun.protect ~finally:(fun()->Database.release_snapshot snapshot)(fun()-> - (match Database.get_pages snapshot [resolved_page] |> T.require_ok ~behavior with - | [Present_page{value;_}]->T.require(value.page.kind=Logseq_db_types.Graph_types.Journal_page{journal_day=20261102}) "echo lost day" - | _->Alcotest.fail "echo lost append page"); - match Database.get_outline_blocks snapshot [root_uuid;child_uuid] |> T.require_ok ~behavior with - | [Present_block{value=root;_};Present_block{value=child;_}]-> - T.require(root.block.parent=resolved_page && child.block.parent=root_uuid && child.block.page=resolved_page) - "echo changed atomic tree ownership"; - T.require(root.task_status=Some(Status_uuid(T.status_uuid 2)) && child.task_status=Some(Status_uuid(T.status_uuid 3))) - "echo lost atomic statuses" - | _->Alcotest.fail "echo lost atomic blocks"); - let sync=Database.inspect_sync database |> T.require_ok ~behavior in - T.require(sync_view_submissions sync=[]) "authoritative append did not retire its accepted outbox record"; - T.require(Database.intent_commit_outcome(Database.commit_local_intent database intent |> T.require_ok ~behavior)=local) + let page = T.mutation_uuid 7790 + and root_uuid = T.mutation_uuid 7791 + and child_uuid = T.mutation_uuid 7792 in + let root : Types.append_node = + { uuid = root_uuid + ; title = "Atomic wire root" + ; task_status = Some (T.status_uuid 2) + ; children = + Rrbvec.singleton + { uuid = child_uuid + ; title = "Atomic wire child" + ; task_status = Some (T.status_uuid 3) + ; children = Rrbvec.empty + } + } + in + let mutation = + Types.Append_journal + { mutation_id = T.mutation_uuid 7793 + ; page + ; title = "2026-11-02" + ; journal_day = 20261102 + ; root + } + in + let expected = + Database.write_precondition ~blocks:[] ~pages:[] ~scopes:[] |> T.require_ok ~behavior + in + let intent = + Database.persist_local_intent + database + ~namespace:(String.make 64 'f') + ~payload:"complete append wire" + ~expected + mutation + |> T.require_ok ~behavior + in + let proof = Database.commit_local_intent database intent |> T.require_ok ~behavior in + let local = Database.intent_commit_outcome proof in + let resolved_page = + if not remote_day + then page + else + let module J = Transit_core.Json in + let web = T.mutation_uuid 7794 in + let page_entity = J.String "remote-web-day" + and child_entity = J.String "remote-first-entry" in + let add_page attribute value = T.status_add page_entity attribute value in + let add_child attribute value = T.status_add child_entity attribute value in + T.apply_test_transaction + database + ~cursor:2 + [ add_page "block/uuid" (J.Uuid (Logseq_db_types.Graph_types.Uuid.to_string web)) + ; add_page "block/title" (J.String "Web day") + ; add_page "block/name" (J.String "web-remote-day") + ; add_page "block/journal-day" (J.Int 20261102) + ; add_child + "block/uuid" + (J.Uuid (Logseq_db_types.Graph_types.Uuid.to_string (T.mutation_uuid 7795))) + ; add_child "block/title" (J.String "Remote first entry") + ; add_child "block/parent" page_entity + ; add_child "block/page" page_entity + ; add_child "block/order" (J.String "a9") + ]; + let snapshot = Database.current_snapshot database |> T.require_ok ~behavior in + Fun.protect + ~finally:(fun () -> Database.release_snapshot snapshot) + (fun () -> + T.require + (match + Database.get_outline_blocks snapshot [ root_uuid ] + |> T.require_ok ~behavior + with + | [ Present_block { value; _ } ] -> + value.block.parent = web && String.compare value.block.order "a9" > 0 + | _ -> false) + "queued append did not follow the Web day's existing tail"); + web + in + let echo_cursor = if remote_day then 3 else 2 in + let page_value () = + let snapshot = Database.current_snapshot database |> T.require_ok ~behavior in + Fun.protect + ~finally:(fun () -> Database.release_snapshot snapshot) + (fun () -> + match + Database.get_pages snapshot [ resolved_page ] |> T.require_ok ~behavior + with + | [ Present_page { value; _ } ] -> value + | _ -> Alcotest.fail "atomic page disappeared") + in + let before_submit = page_value () in + let submitted = submit_one database local behavior in + T.require + (page_value () = before_submit) + "transport submission changed complete append page DTO"; + T.require + (List.length (submission_batch_wires submitted) = 1) + "append split one atomic action into separate outbox transactions"; + let wire = + server_transaction + ~mutation_id:local.mutation_id + ~operation:"append-journal" + submitted + in + let wire = + if reverse_operations + then + let module Codec = Transit_native.Transit.Json in + match Codec.of_string wire with + | Transit_core.Json.Array operations -> + Codec.to_string + ~mode:Codec.Verbose + (Transit_core.Json.Array (List.rev operations)) + | _ -> Alcotest.fail "append echo is not an array" + else wire + in + let barrier = + { through = server_cursor echo_cursor + ; checksum = checksum (if remote_day then "7c0e6c83a247118b" else "1afbe793c7bf779b") + } + in + let sync = Database.inspect_sync database |> T.require_ok ~behavior in + let prepared, crypto = + Database.begin_outbox_transition + database + ~expected:(sync_view_token sync) + (Accept_group { batch_id = submission_batch_id submitted; barrier }) + |> T.require_ok ~behavior + in + T.require (crypto = None) "append acceptance requested extra crypto"; + ignore + (Database.apply_outbox_transition database prepared ~encrypted:None + |> T.require_ok ~behavior); + let batch = + authoritative_batch + ~maximum_count:16 + ~maximum_bytes:65536 + ~transactions: + [ authoritative_transaction + ~cursor:(server_cursor echo_cursor) + ~transaction:(encoded ~maximum_bytes:65536 wire) + ] + ~through:(server_cursor echo_cursor) + ~checksum:(Some barrier.checksum) + |> T.require_ok ~behavior + in + let sync = Database.inspect_sync database |> T.require_ok ~behavior in + let prepared, crypto = + Database.begin_authoritative database ~expected:(sync_view_token sync) batch + |> T.require_ok ~behavior + in + let decrypted = + Option.map (fun request -> decrypt_submitted_request request behavior) crypto + in + (match + Database.apply_authoritative database prepared ~decrypted |> T.require_ok ~behavior + with + | Authoritative_applied _ -> () + | _ -> Alcotest.fail "atomic append echo was deferred"); + let snapshot = Database.current_snapshot database |> T.require_ok ~behavior in + Fun.protect + ~finally:(fun () -> Database.release_snapshot snapshot) + (fun () -> + (match Database.get_pages snapshot [ resolved_page ] |> T.require_ok ~behavior with + | [ Present_page { value; _ } ] -> + T.require + (value.page.kind + = Logseq_db_types.Graph_types.Journal_page { journal_day = 20261102 }) + "echo lost day" + | _ -> Alcotest.fail "echo lost append page"); + match + Database.get_outline_blocks snapshot [ root_uuid; child_uuid ] + |> T.require_ok ~behavior + with + | [ Present_block { value = root; _ }; Present_block { value = child; _ } ] -> + T.require + (root.block.parent = resolved_page + && child.block.parent = root_uuid + && child.block.page = resolved_page) + "echo changed atomic tree ownership"; + T.require + (root.task_status = Some (Status_uuid (T.status_uuid 2)) + && child.task_status = Some (Status_uuid (T.status_uuid 3))) + "echo lost atomic statuses" + | _ -> Alcotest.fail "echo lost atomic blocks"); + let sync = Database.inspect_sync database |> T.require_ok ~behavior in + T.require + (sync_view_submissions sync = []) + "authoritative append did not retire its accepted outbox record"; + T.require + (Database.intent_commit_outcome + (Database.commit_local_intent database intent |> T.require_ok ~behavior) + = local) "authoritative echo replaced original append proof" ;; let atomic_append_queue_replans_actual_owners database = - let behavior="queued append prefix preserves day owners and per-day tail" in - let expected=Database.write_precondition ~blocks:[] ~pages:[] ~scopes:[] |> T.require_ok ~behavior in - let roots=Rrbvec.init 128(fun index->T.mutation_uuid(80000+index)) in - Rrbvec.iteri(fun index uuid-> - let slot=index mod 3 in - let root:Types.append_node={uuid;title=Printf.sprintf "Queued root %d" index;task_status=None;children=Rrbvec.empty} in - let mutation=Types.Append_journal{mutation_id=T.mutation_uuid(90000+index);page=T.mutation_uuid(79000+slot); - title=Printf.sprintf "Day %d" slot;journal_day=20261101+slot;root} in - ignore(T.commit_mutation database ~expected mutation ~behavior)) roots; - let web=T.mutation_uuid 78000 in - let module J=Transit_core.Json in - let owner=J.String "prefix-web-day" and first=J.String "prefix-web-first" in - let add entity attribute value=T.status_add entity attribute value in - T.apply_test_transaction database ~cursor:1 - [add owner "block/uuid" (J.Uuid(Logseq_db_types.Graph_types.Uuid.to_string web)); - add owner "block/title" (J.String "Web middle day");add owner "block/name" (J.String "prefix-web-day"); - add owner "block/journal-day" (J.Int 20261102); - add first "block/uuid" (J.Uuid(Logseq_db_types.Graph_types.Uuid.to_string(T.mutation_uuid 78001))); - add first "block/title" (J.String "Remote first");add first "block/parent" owner; - add first "block/page" owner;add first "block/order" (J.String "a9")]; - let snapshot=Database.current_snapshot database |> T.require_ok ~behavior in - Fun.protect ~finally:(fun()->Database.release_snapshot snapshot)(fun()-> - let previous=Array.make 3 None in previous.(1)<-Some "a9"; - let blocks=Database.get_outline_blocks snapshot(Rrbvec.to_list roots) |> T.require_ok ~behavior in - List.iteri(fun index->function - | Present_block{value;_}-> - let slot=index mod 3 in - let page=if slot=1 then web else T.mutation_uuid(79000+slot) in - T.require(value.block.page=page && value.block.parent=page - && value.block.title=Printf.sprintf "Queued root %d" index) "prefix changed a root owner/source"; - Option.iter(fun previous->T.require(String.compare previous value.block.order<0) - "prefix order did not advance within the resolved day")previous.(slot); - previous.(slot)<-Some value.block.order - | _->Alcotest.fail "prefix lost a pending root")blocks; - T.require(match Database.get_pages snapshot [T.mutation_uuid 79001] |> T.require_ok ~behavior with - |[Missing_page _]->true|_->false) "replan retained the unused candidate day"); - let view=Database.inspect_sync database |> T.require_ok ~behavior in - T.require(List.length(sync_view_submissions view)=128) "prefix replan dropped an unsubmitted intent" + let behavior = "queued append prefix preserves day owners and per-day tail" in + let expected = + Database.write_precondition ~blocks:[] ~pages:[] ~scopes:[] |> T.require_ok ~behavior + in + let roots = Rrbvec.init 128 (fun index -> T.mutation_uuid (80000 + index)) in + Rrbvec.iteri + (fun index uuid -> + let slot = index mod 3 in + let root : Types.append_node = + { uuid + ; title = Printf.sprintf "Queued root %d" index + ; task_status = None + ; children = Rrbvec.empty + } + in + let mutation = + Types.Append_journal + { mutation_id = T.mutation_uuid (90000 + index) + ; page = T.mutation_uuid (79000 + slot) + ; title = Printf.sprintf "Day %d" slot + ; journal_day = 20261101 + slot + ; root + } + in + ignore (T.commit_mutation database ~expected mutation ~behavior)) + roots; + let web = T.mutation_uuid 78000 in + let module J = Transit_core.Json in + let owner = J.String "prefix-web-day" + and first = J.String "prefix-web-first" in + let add entity attribute value = T.status_add entity attribute value in + T.apply_test_transaction + database + ~cursor:1 + [ add owner "block/uuid" (J.Uuid (Logseq_db_types.Graph_types.Uuid.to_string web)) + ; add owner "block/title" (J.String "Web middle day") + ; add owner "block/name" (J.String "prefix-web-day") + ; add owner "block/journal-day" (J.Int 20261102) + ; add + first + "block/uuid" + (J.Uuid (Logseq_db_types.Graph_types.Uuid.to_string (T.mutation_uuid 78001))) + ; add first "block/title" (J.String "Remote first") + ; add first "block/parent" owner + ; add first "block/page" owner + ; add first "block/order" (J.String "a9") + ]; + let snapshot = Database.current_snapshot database |> T.require_ok ~behavior in + Fun.protect + ~finally:(fun () -> Database.release_snapshot snapshot) + (fun () -> + let previous = Array.make 3 None in + previous.(1) <- Some "a9"; + let blocks = + Database.get_outline_blocks snapshot (Rrbvec.to_list roots) + |> T.require_ok ~behavior + in + List.iteri + (fun index -> function + | Present_block { value; _ } -> + let slot = index mod 3 in + let page = if slot = 1 then web else T.mutation_uuid (79000 + slot) in + T.require + (value.block.page = page + && value.block.parent = page + && value.block.title = Printf.sprintf "Queued root %d" index) + "prefix changed a root owner/source"; + Option.iter + (fun previous -> + T.require + (String.compare previous value.block.order < 0) + "prefix order did not advance within the resolved day") + previous.(slot); + previous.(slot) <- Some value.block.order + | _ -> Alcotest.fail "prefix lost a pending root") + blocks; + T.require + (match + Database.get_pages snapshot [ T.mutation_uuid 79001 ] + |> T.require_ok ~behavior + with + | [ Missing_page _ ] -> true + | _ -> false) + "replan retained the unused candidate day"); + let view = Database.inspect_sync database |> T.require_ok ~behavior in + T.require + (List.length (sync_view_submissions view) = 128) + "prefix replan dropped an unsubmitted intent" ;; let atomic_append_tail_includes_later_submitted_shadow database = - let behavior="append tail includes a later submitted dependency shadow" in - let module J=Transit_core.Json in let module Codec=Transit_native.Transit.Json in - let low=T.mutation_uuid 92000 in - let entity=J.String "shadow-low-sibling" in - T.apply_test_transaction database ~cursor:1 - [T.status_add(T.status_lookup T.page_uuid) "block/journal-day" (J.Int 20261102); - T.status_add(T.status_lookup T.authoritative_block_uuid) "block/order" (J.String "a9"); - T.status_add entity "block/uuid" (J.Uuid(Logseq_db_types.Graph_types.Uuid.to_string low)); - T.status_add entity "block/title" (J.String "Low sibling"); - T.status_add entity "block/parent" (T.status_lookup T.page_uuid); - T.status_add entity "block/page" (T.status_lookup T.page_uuid);T.status_add entity "block/order" (J.String "a0")]; + let behavior = "append tail includes a later submitted dependency shadow" in + let module J = Transit_core.Json in + let module Codec = Transit_native.Transit.Json in + let low = T.mutation_uuid 92000 in + let entity = J.String "shadow-low-sibling" in + T.apply_test_transaction + database + ~cursor:1 + [ T.status_add (T.status_lookup T.page_uuid) "block/journal-day" (J.Int 20261102) + ; T.status_add + (T.status_lookup T.authoritative_block_uuid) + "block/order" + (J.String "a9") + ; T.status_add + entity + "block/uuid" + (J.Uuid (Logseq_db_types.Graph_types.Uuid.to_string low)) + ; T.status_add entity "block/title" (J.String "Low sibling") + ; T.status_add entity "block/parent" (T.status_lookup T.page_uuid) + ; T.status_add entity "block/page" (T.status_lookup T.page_uuid) + ; T.status_add entity "block/order" (J.String "a0") + ]; let append index = - let expected=Database.write_precondition ~blocks:[] ~pages:[] ~scopes:[] |> T.require_ok ~behavior in - ignore(T.commit_mutation database ~expected ~behavior - (Types.Append_journal{mutation_id=T.mutation_uuid(92100+index);page=T.mutation_uuid 92004; - title="Unused candidate";journal_day=20261102; - root={uuid=T.mutation_uuid(92001+index);title="Queued append";task_status=None;children=Rrbvec.empty}})) in + let expected = + Database.write_precondition ~blocks:[] ~pages:[] ~scopes:[] + |> T.require_ok ~behavior + in + ignore + (T.commit_mutation + database + ~expected + ~behavior + (Types.Append_journal + { mutation_id = T.mutation_uuid (92100 + index) + ; page = T.mutation_uuid 92004 + ; title = "Unused candidate" + ; journal_day = 20261102 + ; root = + { uuid = T.mutation_uuid (92001 + index) + ; title = "Queued append" + ; task_status = None + ; children = Rrbvec.empty + } + })) + in append 0; - let saved=match T.commit_mutation database ~expected:(T.delete_precondition database - ~block:T.authoritative_block_uuid ~behavior) ~behavior - (Types.Save_block{mutation_id=T.mutation_uuid 92003;block=T.authoritative_block_uuid;title="Submitted save"}) with - | Local_committed value->value|_->Alcotest.fail "save already existed" in - let submitted=submit_one database saved behavior in + let saved = + match + T.commit_mutation + database + ~expected: + (T.delete_precondition database ~block:T.authoritative_block_uuid ~behavior) + ~behavior + (Types.Save_block + { mutation_id = T.mutation_uuid 92003 + ; block = T.authoritative_block_uuid + ; title = "Submitted save" + }) + with + | Local_committed value -> value + | _ -> Alcotest.fail "save already existed" + in + let submitted = submit_one database saved behavior in append 1; (* A genuine own echo at the reserved cursor followed by a remote deletion; the Save's transport ACK has not arrived, so its dependency remains owned. *) - let own=server_transaction ~mutation_id:saved.mutation_id ~operation:"save-block" submitted in - let deletion=Codec.to_string ~mode:Codec.Verbose(J.Array[ - J.Array[J.Keyword "db/retractEntity";T.status_lookup T.authoritative_block_uuid]]) in - let batch=authoritative_batch ~maximum_count:16 ~maximum_bytes:65536 - ~transactions:[authoritative_transaction ~cursor:(server_cursor 2) ~transaction:(encoded ~maximum_bytes:65536 own); - authoritative_transaction ~cursor:(server_cursor 3) ~transaction:(encoded deletion)] - ~through:(server_cursor 3) ~checksum:None |> T.require_ok ~behavior in - let view=Database.inspect_sync database |> T.require_ok ~behavior in - let prepared,crypto=Database.begin_authoritative database ~expected:(sync_view_token view)batch |> T.require_ok ~behavior in - let decrypted=Option.map(fun request->decrypt_submitted_request request behavior)crypto in - (match Database.apply_authoritative database prepared ~decrypted |> T.require_ok ~behavior with - | Authoritative_applied _->()|_->Alcotest.fail "legal delayed-ACK shadow fixture was deferred"); - let snapshot=Database.current_snapshot database |> T.require_ok ~behavior in - Fun.protect ~finally:(fun()->Database.release_snapshot snapshot)(fun()-> - match Database.get_outline_blocks snapshot [T.authoritative_block_uuid;T.mutation_uuid 92001;T.mutation_uuid 92002] - |> T.require_ok ~behavior with - | [Present_block{value=shadow;_};Present_block{value=first;_};Present_block{value=last;_}]-> - T.require(shadow.block.order="a9") "fixture did not retain the active submitted shadow"; - T.require(String.compare last.block.order shadow.block.order>0) - "next Append was ordered before a still-owned submitted sibling"; - T.require(String.compare first.block.order shadow.block.order>0 && String.compare last.block.order first.block.order>0) - "replan lost the original queued Append's logical tail" - | _->Alcotest.fail "fixture lost the shadow or queued append") + let own = + server_transaction ~mutation_id:saved.mutation_id ~operation:"save-block" submitted + in + let deletion = + Codec.to_string + ~mode:Codec.Verbose + (J.Array + [ J.Array + [ J.Keyword "db/retractEntity"; T.status_lookup T.authoritative_block_uuid ] + ]) + in + let batch = + authoritative_batch + ~maximum_count:16 + ~maximum_bytes:65536 + ~transactions: + [ authoritative_transaction + ~cursor:(server_cursor 2) + ~transaction:(encoded ~maximum_bytes:65536 own) + ; authoritative_transaction + ~cursor:(server_cursor 3) + ~transaction:(encoded deletion) + ] + ~through:(server_cursor 3) + ~checksum:None + |> T.require_ok ~behavior + in + let view = Database.inspect_sync database |> T.require_ok ~behavior in + let prepared, crypto = + Database.begin_authoritative database ~expected:(sync_view_token view) batch + |> T.require_ok ~behavior + in + let decrypted = + Option.map (fun request -> decrypt_submitted_request request behavior) crypto + in + (match + Database.apply_authoritative database prepared ~decrypted |> T.require_ok ~behavior + with + | Authoritative_applied _ -> () + | _ -> Alcotest.fail "legal delayed-ACK shadow fixture was deferred"); + let snapshot = Database.current_snapshot database |> T.require_ok ~behavior in + Fun.protect + ~finally:(fun () -> Database.release_snapshot snapshot) + (fun () -> + match + Database.get_outline_blocks + snapshot + [ T.authoritative_block_uuid; T.mutation_uuid 92001; T.mutation_uuid 92002 ] + |> T.require_ok ~behavior + with + | [ Present_block { value = shadow; _ } + ; Present_block { value = first; _ } + ; Present_block { value = last; _ } + ] -> + T.require + (shadow.block.order = "a9") + "fixture did not retain the active submitted shadow"; + T.require + (String.compare last.block.order shadow.block.order > 0) + "next Append was ordered before a still-owned submitted sibling"; + T.require + (String.compare first.block.order shadow.block.order > 0 + && String.compare last.block.order first.block.order > 0) + "replan lost the original queued Append's logical tail" + | _ -> Alcotest.fail "fixture lost the shadow or queued append") ;; let submitted_append_page_shadow_stays_in_journal_index database = - let behavior="journal catalog hydrates submitted Append's retained page shadow" in - let module J=Transit_core.Json in let module Codec=Transit_native.Transit.Json in - T.apply_test_transaction database ~cursor:1 - [T.status_add(T.status_lookup T.page_uuid) "block/journal-day" (J.Int 20261102)]; - let expected=Database.write_precondition ~blocks:[] ~pages:[] ~scopes:[] |> T.require_ok ~behavior in - let record=match T.commit_mutation database ~expected ~behavior - (Types.Append_journal{mutation_id=T.mutation_uuid 92900;page=T.mutation_uuid 92901; - title="Unused candidate";journal_day=20261102; - root={uuid=T.mutation_uuid 92902;title="Submitted root";task_status=None;children=Rrbvec.empty}}) with - |Local_committed record->record|_->Alcotest.fail "Append already existed" in - let submitted=submit_one database record behavior in - let own=server_transaction ~mutation_id:record.mutation_id ~operation:"append-journal" submitted in - let deletion=Codec.to_string ~mode:Codec.Verbose(J.Array[ - J.Array[J.Keyword "db/retractEntity";T.status_lookup T.page_uuid]]) in - let batch=authoritative_batch ~maximum_count:16 ~maximum_bytes:65536 - ~transactions:[authoritative_transaction ~cursor:(server_cursor 2) ~transaction:(encoded ~maximum_bytes:65536 own); - authoritative_transaction ~cursor:(server_cursor 3) ~transaction:(encoded deletion)] - ~through:(server_cursor 3) ~checksum:None |> T.require_ok ~behavior in - let view=Database.inspect_sync database |> T.require_ok ~behavior in - let prepared,crypto=Database.begin_authoritative database ~expected:(sync_view_token view)batch |> T.require_ok ~behavior in - let decrypted=Option.map(fun request->decrypt_submitted_request request behavior)crypto in - (match Database.apply_authoritative database prepared ~decrypted |> T.require_ok ~behavior with - |Authoritative_applied _->()|_->Alcotest.fail "valid delayed-ACK page shadow was deferred"); - let snapshot=Database.current_snapshot database |> T.require_ok ~behavior in - Fun.protect ~finally:(fun()->Database.release_snapshot snapshot)(fun()-> - let journals=Database.get_journals snapshot ~from_day:20261102 ~through_day:20261102 ~limit:1 ~cursor:None - |> T.require_ok ~behavior in - T.require(List.length journals.items=1 && (List.hd journals.items).page.page.uuid=T.page_uuid) - "journal catalog discarded the still-owned submitted page shadow") + let behavior = "journal catalog hydrates submitted Append's retained page shadow" in + let module J = Transit_core.Json in + let module Codec = Transit_native.Transit.Json in + T.apply_test_transaction + database + ~cursor:1 + [ T.status_add (T.status_lookup T.page_uuid) "block/journal-day" (J.Int 20261102) ]; + let expected = + Database.write_precondition ~blocks:[] ~pages:[] ~scopes:[] |> T.require_ok ~behavior + in + let record = + match + T.commit_mutation + database + ~expected + ~behavior + (Types.Append_journal + { mutation_id = T.mutation_uuid 92900 + ; page = T.mutation_uuid 92901 + ; title = "Unused candidate" + ; journal_day = 20261102 + ; root = + { uuid = T.mutation_uuid 92902 + ; title = "Submitted root" + ; task_status = None + ; children = Rrbvec.empty + } + }) + with + | Local_committed record -> record + | _ -> Alcotest.fail "Append already existed" + in + let submitted = submit_one database record behavior in + let own = + server_transaction + ~mutation_id:record.mutation_id + ~operation:"append-journal" + submitted + in + let deletion = + Codec.to_string + ~mode:Codec.Verbose + (J.Array [ J.Array [ J.Keyword "db/retractEntity"; T.status_lookup T.page_uuid ] ]) + in + let batch = + authoritative_batch + ~maximum_count:16 + ~maximum_bytes:65536 + ~transactions: + [ authoritative_transaction + ~cursor:(server_cursor 2) + ~transaction:(encoded ~maximum_bytes:65536 own) + ; authoritative_transaction + ~cursor:(server_cursor 3) + ~transaction:(encoded deletion) + ] + ~through:(server_cursor 3) + ~checksum:None + |> T.require_ok ~behavior + in + let view = Database.inspect_sync database |> T.require_ok ~behavior in + let prepared, crypto = + Database.begin_authoritative database ~expected:(sync_view_token view) batch + |> T.require_ok ~behavior + in + let decrypted = + Option.map (fun request -> decrypt_submitted_request request behavior) crypto + in + (match + Database.apply_authoritative database prepared ~decrypted |> T.require_ok ~behavior + with + | Authoritative_applied _ -> () + | _ -> Alcotest.fail "valid delayed-ACK page shadow was deferred"); + let snapshot = Database.current_snapshot database |> T.require_ok ~behavior in + Fun.protect + ~finally:(fun () -> Database.release_snapshot snapshot) + (fun () -> + let journals = + Database.get_journals + snapshot + ~from_day:20261102 + ~through_day:20261102 + ~limit:1 + ~cursor:None + |> T.require_ok ~behavior + in + T.require + (List.length journals.items = 1 + && (List.hd journals.items).page.page.uuid = T.page_uuid) + "journal catalog discarded the still-owned submitted page shadow") ;; let acceptance_is_transport_only database = @@ -1571,7 +1987,9 @@ let definitive_rejection_rolls_back_once database = | No_logical_change | Logical_resync_required _ -> Alcotest.fail "single rejected insertion did not return an exact rollback"); let snapshot = Database.current_snapshot database |> T.require_ok ~behavior in - (match Database.get_outline_blocks snapshot [ T.block_uuid ] |> T.require_ok ~behavior with + (match + Database.get_outline_blocks snapshot [ T.block_uuid ] |> T.require_ok ~behavior + with | [ Missing_block _ ] -> () | _ -> Alcotest.fail "rejected insertion remained visible"); Database.release_snapshot snapshot; @@ -1692,7 +2110,8 @@ let authoritative_batch_updates_the_logical_snapshot database = "visible authoritative update did not advance projection"; let after = Database.current_snapshot database |> T.require_ok ~behavior in (match - Database.get_outline_blocks after [ T.authoritative_block_uuid ] |> T.require_ok ~behavior + Database.get_outline_blocks after [ T.authoritative_block_uuid ] + |> T.require_ok ~behavior with | [ Present_block { value; _ } ] -> T.require @@ -1751,7 +2170,8 @@ let authoritative_rebase_replans_queued_ordinary_mutation database = "queued ordinary mutation was not reported as replanned"; let snapshot = Database.current_snapshot database |> T.require_ok ~behavior in (match - Database.get_outline_blocks snapshot [ T.authoritative_block_uuid ] |> T.require_ok ~behavior + Database.get_outline_blocks snapshot [ T.authoritative_block_uuid ] + |> T.require_ok ~behavior with | [ Present_block { value; _ } ] -> T.require @@ -2057,7 +2477,8 @@ let authoritative_crypto_is_correlated_and_applied database = | Ok _ -> Alcotest.fail "stale authoritative crypto was accepted"); let snapshot = Database.current_snapshot database |> T.require_ok ~behavior in (match - Database.get_outline_blocks snapshot [ T.authoritative_block_uuid ] |> T.require_ok ~behavior + Database.get_outline_blocks snapshot [ T.authoritative_block_uuid ] + |> T.require_ok ~behavior with | [ Present_block { value; _ } ] -> T.require @@ -2091,7 +2512,7 @@ let submitted_delete_defers_authoritative_batch_until_transport_outcome database |> encoded in let cursor = server_cursor 1 in - let checksum = checksum "dededededededede" in + let checksum = checksum "aa5ce131b706db6b" in let batch = authoritative_batch ~maximum_count:16 @@ -2242,7 +2663,8 @@ let pull_first_delete_conflict_proves_non_execution database = | _ -> Alcotest.fail "Pull-first conflict omitted its retained terminal receipt"); let snapshot = Database.current_snapshot database |> T.require_ok ~behavior in (match - Database.get_outline_blocks snapshot [ T.authoritative_block_uuid ] |> T.require_ok ~behavior + Database.get_outline_blocks snapshot [ T.authoritative_block_uuid ] + |> T.require_ok ~behavior with | [ Present_block { value; _ } ] -> T.require @@ -2517,7 +2939,7 @@ let accepted_member_terminalizes_at_its_authoritative_barrier_with | Local_existing _ -> Alcotest.fail "fresh accepted fixture already existed" in let submitted = submit_one database local behavior in - let barrier = { through = server_cursor 1; checksum = checksum "1111111111111111" } in + let barrier = { through = server_cursor 1; checksum = checksum "87d0b4e319dbf287" } in let sync = Database.inspect_sync database |> T.require_ok ~behavior in let accepted, _crypto = Database.begin_outbox_transition @@ -2541,7 +2963,7 @@ let accepted_member_terminalizes_at_its_authoritative_barrier_with ~transactions: [ authoritative_transaction ~cursor:(server_cursor 1) ~transaction:wire ] ~through:(server_cursor 1) - ~checksum:(Some (checksum "1111111111111111")) + ~checksum:(Some (checksum "87d0b4e319dbf287")) |> T.require_ok ~behavior in let sync = Database.inspect_sync database |> T.require_ok ~behavior in @@ -2619,7 +3041,7 @@ let accepted_member_rejects_wrong_payload database = | Local_existing _ -> Alcotest.fail "fresh wrong-digest fixture already existed" in let submitted = submit_one database local behavior in - let barrier = { through = server_cursor 1; checksum = checksum "1111111111111111" } in + let barrier = { through = server_cursor 1; checksum = checksum "f244e62fe240079f" } in let sync = Database.inspect_sync database |> T.require_ok ~behavior in let accepted, _ = Database.begin_outbox_transition @@ -2667,7 +3089,7 @@ let accepted_member_rejects_wrong_payload database = ~transactions: [ authoritative_transaction ~cursor:(server_cursor 1) ~transaction:wire ] ~through:(server_cursor 1) - ~checksum:(Some (checksum "1111111111111111")) + ~checksum:(Some (checksum "f244e62fe240079f")) |> T.require_ok ~behavior in let sync = Database.inspect_sync database |> T.require_ok ~behavior in @@ -2719,7 +3141,7 @@ let accepted_member_rejects_extra_touched_fact database = | Local_existing _ -> Alcotest.fail "fresh touched-footprint fixture already existed" in let submitted = submit_one database local behavior in - let barrier = { through = server_cursor 1; checksum = checksum "1111111111111111" } in + let barrier = { through = server_cursor 1; checksum = checksum "0ad260ea46ced2ec" } in let sync = Database.inspect_sync database |> T.require_ok ~behavior in let accepted, _ = Database.begin_outbox_transition @@ -2764,7 +3186,7 @@ let accepted_member_rejects_extra_touched_fact database = ~transactions: [ authoritative_transaction ~cursor:(server_cursor 1) ~transaction:wire ] ~through:(server_cursor 1) - ~checksum:(Some (checksum "1111111111111111")) + ~checksum:(Some (checksum "0ad260ea46ced2ec")) |> T.require_ok ~behavior in let sync = Database.inspect_sync database |> T.require_ok ~behavior in @@ -2904,7 +3326,8 @@ let accepted_member_uses_intermediate_barrier_root database = | _ -> Alcotest.fail "intermediate barrier did not earn Applied receipt"); let snapshot = Database.current_snapshot database |> T.require_ok ~behavior in (match - Database.get_outline_blocks snapshot [ T.authoritative_block_uuid ] |> T.require_ok ~behavior + Database.get_outline_blocks snapshot [ T.authoritative_block_uuid ] + |> T.require_ok ~behavior with | [ Present_block { value; _ } ] -> T.require @@ -2949,7 +3372,7 @@ let covered_acceptance_uses_combined_group_requirements database = |> fun commit -> Option.get commit.submission_batch in let wires = submission_batch_wires submitted in - let group_checksum = checksum "4141414141414141" in + let group_checksum = checksum "84be9968a909845a" in let batch = authoritative_batch ~maximum_count:16 @@ -3038,7 +3461,7 @@ let accepted_delete_rejects_missing_own_incorporation database = | Local_existing _ -> Alcotest.fail "fresh mismatch fixture already existed" in let submitted = submit_one database local behavior in - let barrier = { through = server_cursor 1; checksum = checksum "2222222222222222" } in + let barrier = { through = server_cursor 1; checksum = checksum "03feca6686fda40c" } in let sync = Database.inspect_sync database |> T.require_ok ~behavior in let accepted, _crypto = Database.begin_outbox_transition @@ -3074,7 +3497,7 @@ let accepted_delete_rejects_missing_own_incorporation database = ~transactions: [ authoritative_transaction ~cursor:(server_cursor 1) ~transaction:wire ] ~through:(server_cursor 1) - ~checksum:(Some (checksum "2222222222222222")) + ~checksum:(Some (checksum "03feca6686fda40c")) |> T.require_ok ~behavior in let sync = Database.inspect_sync database |> T.require_ok ~behavior in @@ -3088,7 +3511,8 @@ let accepted_delete_rejects_missing_own_incorporation database = | Ok _ -> Alcotest.fail "accepted delete tolerated missing own incorporation"); let snapshot = Database.current_snapshot database |> T.require_ok ~behavior in (match - Database.get_outline_blocks snapshot [ T.authoritative_block_uuid ] |> T.require_ok ~behavior + Database.get_outline_blocks snapshot [ T.authoritative_block_uuid ] + |> T.require_ok ~behavior with | [ Missing_block _ ] -> () | _ -> Alcotest.fail "failed integrity preparation changed the logical delete"); @@ -3121,7 +3545,7 @@ let accepted_delete_allows_server_normalized_order database = | Local_existing _ -> Alcotest.fail "fresh normalized delete fixture already existed" in let submitted = submit_one database local behavior in - let barrier = { through = server_cursor 1; checksum = checksum "2323232323232323" } in + let barrier = { through = server_cursor 1; checksum = checksum "aa5ce131b706db6b" } in let sync = Database.inspect_sync database |> T.require_ok ~behavior in let accepted, _crypto = Database.begin_outbox_transition @@ -3261,7 +3685,8 @@ let queued_delete_yields_to_remote_change_before_submission database = | _ -> Alcotest.fail "queued delete conflict omitted Remote_won receipt"); let snapshot = Database.current_snapshot database |> T.require_ok ~behavior in (match - Database.get_outline_blocks snapshot [ T.authoritative_block_uuid ] |> T.require_ok ~behavior + Database.get_outline_blocks snapshot [ T.authoritative_block_uuid ] + |> T.require_ok ~behavior with | [ Present_block { value; _ } ] -> T.require @@ -3700,7 +4125,9 @@ let partial_rejection_blocks_suffix_dependent_on_failed_insert database = (states = [ Blocked; Blocked ]) "failed member and dependent suffix did not become atomically Blocked"; let snapshot = Database.current_snapshot database |> T.require_ok ~behavior in - (match Database.get_outline_blocks snapshot [ T.block_uuid ] |> T.require_ok ~behavior with + (match + Database.get_outline_blocks snapshot [ T.block_uuid ] |> T.require_ok ~behavior + with | [ Missing_block _ ] -> () | _ -> Alcotest.fail "dependent suffix remained in the logical projection"); Database.release_snapshot snapshot @@ -3773,7 +4200,7 @@ let reopened_submission_does_not_reuse_terminal_batch () = { batch_id = old_id ; barrier = { through = server_cursor 1 - ; checksum = checksum "1111111111111111" + ; checksum = checksum "87d0b4e319dbf287" } }) |> T.require_ok ~behavior @@ -3860,7 +4287,9 @@ let stale_remote_wire uuid attribute value = let stale_commit database mutation behavior = let expected = match mutation with - | Append_journal _ -> Database.write_precondition ~blocks:[] ~pages:[] ~scopes:[] |> T.require_ok ~behavior + | Append_journal _ -> + Database.write_precondition ~blocks:[] ~pages:[] ~scopes:[] + |> T.require_ok ~behavior | Insert_blocks { parent; _ } -> T.insert_precondition database ~parent ~behavior | Create_journal_page { page; _ } -> let snapshot = Database.current_snapshot database |> T.require_ok ~behavior in @@ -4196,7 +4625,9 @@ let stale_insert_collision_blocks_intent database = = [ Blocked ]) "replanning a conflicting insertion overwrote the authoritative UUID"; let snapshot = Database.current_snapshot database |> T.require_ok ~behavior in - (match Database.get_outline_blocks snapshot [ T.block_uuid ] |> T.require_ok ~behavior with + (match + Database.get_outline_blocks snapshot [ T.block_uuid ] |> T.require_ok ~behavior + with | [ Present_block { value; _ } ] -> T.require (value.block.title = "Peer owns this UUID") @@ -4697,102 +5128,253 @@ let database_cases = (* Sync's pure reducer carries opaque transactions and cannot execute this retirement/rebase decision. Exercise its owning public Database boundary. *) -let queued_delete_survives_own_append_echo ?(existing_block=false) ?(remote_later=false) ~remote_change db = - let behavior="queued delete follows own append echo" in - let root=if existing_block then T.authoritative_block_uuid else T.mutation_uuid 9001 in - let append=Types.Append_journal {mutation_id=T.mutation_uuid 9002;page=T.mutation_uuid 9003; - title="2026-11-07";journal_day=20261107; - root={uuid=root;title="Synthetic capture";task_status=None;children=Rrbvec.empty}} in - let mutation = if existing_block then - Types.Save_block {mutation_id=T.mutation_uuid 9002;block=root;title="Own saved title"} - else append in - let expected = if existing_block then T.delete_precondition db ~block:root ~behavior - else T.empty_precondition ~behavior in - let intent=Database.persist_local_intent db ~namespace:(String.make 64 'a') ~payload:"append" - ~expected mutation |> T.require_ok ~behavior in - let proof=Database.commit_local_intent db intent |> T.require_ok ~behavior in - let local=Database.intent_commit_outcome proof in - let deletion=Types.Delete_blocks{mutation_id=T.mutation_uuid 9004;root} in - ignore(T.commit_mutation db ~expected:(T.delete_precondition db ~block:root ~behavior) deletion ~behavior); - let submitted=submit_one db local behavior in - let wire=server_transaction ~mutation_id:local.mutation_id ~operation:"append-journal" submitted in - let barrier={through=server_cursor 1;checksum=checksum "1111111111111111"} in - let sync=Database.inspect_sync db |> T.require_ok ~behavior in - let prepared,_=Database.begin_outbox_transition db ~expected:(sync_view_token sync) - (Accept_group{batch_id=submission_batch_id submitted;barrier}) |> T.require_ok ~behavior in - ignore(Database.apply_outbox_transition db prepared ~encrypted:None |> T.require_ok ~behavior); - let transactions = [authoritative_transaction ~cursor:(server_cursor 1) - ~transaction:(encoded ~maximum_bytes:65536 wire)] in - let transactions = if not remote_change then transactions else - transactions @ [authoritative_transaction ~cursor:(server_cursor 2) - ~transaction:(encoded (Printf.sprintf - "[[\"~:db/add\",[\"~:block/uuid\",\"~u%s\"],\"~:block/updated-at\",1704067200999]]" - (Graph.Uuid.to_string root)))] in - let batch=authoritative_batch ~maximum_count:16 ~maximum_bytes:65536 - ~transactions - ~through:(server_cursor (if remote_change then 2 else 1)) ~checksum:(Some barrier.checksum) |> T.require_ok ~behavior in - let sync=Database.inspect_sync db |> T.require_ok ~behavior in - let prepared,crypto=Database.begin_authoritative db ~expected:(sync_view_token sync) batch |> T.require_ok ~behavior in - let decrypted=Option.map(fun req->decrypt_submitted_request req behavior)crypto in - let commit=match Database.apply_authoritative db prepared ~decrypted |> T.require_ok ~behavior with - |Authoritative_applied c->c|_->failwith "deferred" in - let remote_won = List.exists (fun terminal -> match terminal.receipt with - | Remote_won_receipt r -> Graph.Uuid.equal r.mutation_id (T.mutation_uuid 9004) - | _ -> false) commit.terminal_receipts in - T.require (remote_won = remote_change) "own echo was mistaken for a remote delete conflict"; - let sync=Database.inspect_sync db |> T.require_ok ~behavior in - T.require (List.length (sync_view_submissions sync) = if remote_change then 0 else 1) - "queued delete was lost or genuine remote conflict was ignored"; - let snapshot=Database.current_snapshot db |> T.require_ok ~behavior in - let present=match Database.get_outline_blocks snapshot [root] |> T.require_ok ~behavior with - | [Present_block _] -> true | _ -> false in - Database.release_snapshot snapshot; - T.require (present = remote_change) "own echo resurrected locally deleted block"; - if remote_later then ( - let remote = Printf.sprintf - "[[\"~:db/add\",[\"~:block/uuid\",\"~u%s\"],\"~:block/updated-at\",1704067200999]]" - (Graph.Uuid.to_string root) in - let later = commit_plain_authoritative db ~cursor:(server_cursor 2) remote behavior in - T.require (List.exists (fun terminal -> match terminal.receipt with - | Remote_won_receipt r -> Graph.Uuid.equal r.mutation_id (T.mutation_uuid 9004) - | _ -> false) later.terminal_receipts) - "real remote change in a later batch did not win") - else if not remote_change then ( - let prepared, request, _ = prepare_submit db [T.mutation_uuid 9004] behavior in - let protected = protect request behavior in - let sent = Database.apply_outbox_transition db prepared ~encrypted:(Some protected) - |> T.require_ok ~behavior in - T.require (Option.is_some sent.submission_batch) "preserved delete cannot submit") +let queued_delete_survives_own_append_echo + ?(existing_block = false) + ?(remote_later = false) + ~remote_change + db + = + let behavior = "queued delete follows own append echo" in + let root = + if existing_block then T.authoritative_block_uuid else T.mutation_uuid 9001 + in + let append = + Types.Append_journal + { mutation_id = T.mutation_uuid 9002 + ; page = T.mutation_uuid 9003 + ; title = "2026-11-07" + ; journal_day = 20261107 + ; root = + { uuid = root + ; title = "Synthetic capture" + ; task_status = None + ; children = Rrbvec.empty + } + } + in + let mutation = + if existing_block + then + Types.Save_block + { mutation_id = T.mutation_uuid 9002; block = root; title = "Own saved title" } + else append + in + let expected = + if existing_block + then T.delete_precondition db ~block:root ~behavior + else T.empty_precondition ~behavior + in + let intent = + Database.persist_local_intent + db + ~namespace:(String.make 64 'a') + ~payload:"append" + ~expected + mutation + |> T.require_ok ~behavior + in + let proof = Database.commit_local_intent db intent |> T.require_ok ~behavior in + let local = Database.intent_commit_outcome proof in + let deletion = Types.Delete_blocks { mutation_id = T.mutation_uuid 9004; root } in + ignore + (T.commit_mutation + db + ~expected:(T.delete_precondition db ~block:root ~behavior) + deletion + ~behavior); + let submitted = submit_one db local behavior in + let wire = + server_transaction + ~mutation_id:local.mutation_id + ~operation:"append-journal" + submitted + in + let barrier = + { through = server_cursor 1 + ; checksum = + checksum (if existing_block then "873bc63b0d8eb9a1" else "d47bafa0dd760364") + } + in + let sync = Database.inspect_sync db |> T.require_ok ~behavior in + let prepared, _ = + Database.begin_outbox_transition + db + ~expected:(sync_view_token sync) + (Accept_group { batch_id = submission_batch_id submitted; barrier }) + |> T.require_ok ~behavior + in + ignore + (Database.apply_outbox_transition db prepared ~encrypted:None + |> T.require_ok ~behavior); + let transactions = + [ authoritative_transaction + ~cursor:(server_cursor 1) + ~transaction:(encoded ~maximum_bytes:65536 wire) + ] + in + let transactions = + if not remote_change + then transactions + else + transactions + @ [ authoritative_transaction + ~cursor:(server_cursor 2) + ~transaction: + (encoded + (Printf.sprintf + "[[\"~:db/add\",[\"~:block/uuid\",\"~u%s\"],\"~:block/updated-at\",1704067200999]]" + (Graph.Uuid.to_string root))) + ] + in + let batch = + authoritative_batch + ~maximum_count:16 + ~maximum_bytes:65536 + ~transactions + ~through:(server_cursor (if remote_change then 2 else 1)) + ~checksum:(Some barrier.checksum) + |> T.require_ok ~behavior + in + let sync = Database.inspect_sync db |> T.require_ok ~behavior in + let prepared, crypto = + Database.begin_authoritative db ~expected:(sync_view_token sync) batch + |> T.require_ok ~behavior + in + let decrypted = Option.map (fun req -> decrypt_submitted_request req behavior) crypto in + let commit = + match + Database.apply_authoritative db prepared ~decrypted |> T.require_ok ~behavior + with + | Authoritative_applied c -> c + | _ -> failwith "deferred" + in + let remote_won = + List.exists + (fun terminal -> + match terminal.receipt with + | Remote_won_receipt r -> Graph.Uuid.equal r.mutation_id (T.mutation_uuid 9004) + | _ -> false) + commit.terminal_receipts + in + T.require + (remote_won = remote_change) + "own echo was mistaken for a remote delete conflict"; + let sync = Database.inspect_sync db |> T.require_ok ~behavior in + T.require + (List.length (sync_view_submissions sync) = if remote_change then 0 else 1) + "queued delete was lost or genuine remote conflict was ignored"; + let snapshot = Database.current_snapshot db |> T.require_ok ~behavior in + let present = + match Database.get_outline_blocks snapshot [ root ] |> T.require_ok ~behavior with + | [ Present_block _ ] -> true + | _ -> false + in + Database.release_snapshot snapshot; + T.require (present = remote_change) "own echo resurrected locally deleted block"; + if remote_later + then ( + let remote = + Printf.sprintf + "[[\"~:db/add\",[\"~:block/uuid\",\"~u%s\"],\"~:block/updated-at\",1704067200999]]" + (Graph.Uuid.to_string root) + in + let later = commit_plain_authoritative db ~cursor:(server_cursor 2) remote behavior in + T.require + (List.exists + (fun terminal -> + match terminal.receipt with + | Remote_won_receipt r -> + Graph.Uuid.equal r.mutation_id (T.mutation_uuid 9004) + | _ -> false) + later.terminal_receipts) + "real remote change in a later batch did not win") + else if not remote_change + then ( + let prepared, request, _ = prepare_submit db [ T.mutation_uuid 9004 ] behavior in + let protected = protect request behavior in + let sent = + Database.apply_outbox_transition db prepared ~encrypted:(Some protected) + |> T.require_ok ~behavior + in + T.require (Option.is_some sent.submission_batch) "preserved delete cannot submit") ;; let () = Alcotest.run "logseq_overlay_db sync" - [ "atomic append",[T.database_case "tail survives later submitted shadow" atomic_append_tail_includes_later_submitted_shadow; - T.database_case "mixed-day queued prefix uses incremental owners" atomic_append_queue_replans_actual_owners; - T.database_case "complete protected transaction echoes atomically" (atomic_append_submits_and_replays_complete_tree ~remote_day:false); - T.database_case "queued append adopts remote day and follows its tail" (atomic_append_submits_and_replays_complete_tree ~remote_day:true); - T.database_case "journal catalog keeps submitted page shadow" submitted_append_page_shadow_stays_in_journal_index] - ; "empty authoritative compatibility", - [ T.database_case "empty array advances only sync state" (empty_authoritative_preserves_logical_state "[]") - ; T.database_case "empty Transit list advances only sync state" (empty_authoritative_preserves_logical_state - (Transit_native.Transit.Json.to_string (Transit_core.Json.List []))) - ; T.database_case "mixed batch preserves empty cursor slots" empty_authoritative_mixed_batch - ; Alcotest.test_case "empty cursor and revision survive reopen" `Quick empty_authoritative_survives_reopen - ; Alcotest.test_case "late decode failure rolls back mixed batch" `Quick (empty_authoritative_failure_is_atomic ~storage:false) - ; Alcotest.test_case "storage failure rolls back mixed batch" `Quick (empty_authoritative_failure_is_atomic ~storage:true) - ; T.database_case "empty payload keeps both token guards" empty_authoritative_keeps_token_guards - ; T.database_case "malformed neighbours and empty outer batch reject" empty_authoritative_keeps_invalid_input_rejection ] - ; "queued delete own echoes", [ - T.database_case "queued delete survives own append echo" - (queued_delete_survives_own_append_echo ~remote_change:false); - T.database_case "queued delete still yields to remote edit after own echo" - (queued_delete_survives_own_append_echo ~remote_change:true); - T.database_case "queued delete yields to later remote batch after own echo" - (queued_delete_survives_own_append_echo ~remote_later:true ~remote_change:false); - T.database_case "queued delete survives earlier local save echo" - (queued_delete_survives_own_append_echo ~existing_block:true ~remote_change:false)] + [ ( "atomic append" + , [ T.database_case + "tail survives later submitted shadow" + atomic_append_tail_includes_later_submitted_shadow + ; T.database_case + "mixed-day queued prefix uses incremental owners" + atomic_append_queue_replans_actual_owners + ; T.database_case + "complete protected transaction echoes atomically" + (atomic_append_submits_and_replays_complete_tree ~remote_day:false) + ; T.database_case + "server-reordered append keeps UUID identities" + (atomic_append_submits_and_replays_complete_tree ~reverse_operations:true) + ; T.database_case + "reordered echo still rejects changed UUID" + reordered_echo_rejects_changed_uuid + ; T.database_case + "queued append adopts remote day and follows its tail" + (atomic_append_submits_and_replays_complete_tree ~remote_day:true) + ; T.database_case + "journal catalog keeps submitted page shadow" + submitted_append_page_shadow_stays_in_journal_index + ] ) + ; ( "empty authoritative compatibility" + , [ T.database_case + "empty array advances only sync state" + (empty_authoritative_preserves_logical_state "[]") + ; T.database_case + "empty Transit list advances only sync state" + (empty_authoritative_preserves_logical_state + (Transit_native.Transit.Json.to_string (Transit_core.Json.List []))) + ; T.database_case + "mixed batch preserves empty cursor slots" + empty_authoritative_mixed_batch + ; Alcotest.test_case + "empty cursor and revision survive reopen" + `Quick + empty_authoritative_survives_reopen + ; Alcotest.test_case + "late decode failure rolls back mixed batch" + `Quick + (empty_authoritative_failure_is_atomic ~storage:false) + ; Alcotest.test_case + "storage failure rolls back mixed batch" + `Quick + (empty_authoritative_failure_is_atomic ~storage:true) + ; T.database_case + "empty payload keeps both token guards" + empty_authoritative_keeps_token_guards + ; T.database_case + "malformed neighbours and empty outer batch reject" + empty_authoritative_keeps_invalid_input_rejection + ] ) + ; ( "queued delete own echoes" + , [ T.database_case + "queued delete survives own append echo" + (queued_delete_survives_own_append_echo ~remote_change:false) + ; T.database_case + "queued delete still yields to remote edit after own echo" + (queued_delete_survives_own_append_echo ~remote_change:true) + ; T.database_case + "queued delete yields to later remote batch after own echo" + (queued_delete_survives_own_append_echo + ~remote_later:true + ~remote_change:false) + ; T.database_case + "queued delete survives earlier local save echo" + (queued_delete_survives_own_append_echo + ~existing_block:true + ~remote_change:false) + ] ) ; "validated wire inputs", pure_cases ; "protected same-value rewrites", protected_retraction_cases - ; "durable sync view", database_cases ] + ; "durable sync view", database_cases + ] ;; diff --git a/logseq_sync/lib/pure_reducer/asset_transfer.ml b/logseq_sync/lib/asset_transfer/core.ml similarity index 99% rename from logseq_sync/lib/pure_reducer/asset_transfer.ml rename to logseq_sync/lib/asset_transfer/core.ml index 06d5720d..30c917be 100644 --- a/logseq_sync/lib/pure_reducer/asset_transfer.ml +++ b/logseq_sync/lib/asset_transfer/core.ml @@ -39,14 +39,14 @@ type lookup_source = type lookup_ticket = { id : int - ; scope : Core.graph_scope + ; scope : Logseq_sync_domain.Core.graph_scope ; asset : Logseq_db_types.Graph_types.Uuid.t ; source : lookup_source } type fetch_ticket = { id : int - ; scope : Core.graph_scope + ; scope : Logseq_sync_domain.Core.graph_scope ; asset : Logseq_db_types.Graph_types.Uuid.t ; version : Asset.version } @@ -117,7 +117,7 @@ type entry = type t = { config : config - ; scope : Core.graph_scope + ; scope : Logseq_sync_domain.Core.graph_scope ; online : bool ; unlocked : bool ; entries : entry Assets.t diff --git a/logseq_sync/spec/pure_reducer/asset_transfer.mli b/logseq_sync/lib/asset_transfer/core.mli similarity index 50% rename from logseq_sync/spec/pure_reducer/asset_transfer.mli rename to logseq_sync/lib/asset_transfer/core.mli index 16dcf254..5af2f6a7 100644 --- a/logseq_sync/spec/pure_reducer/asset_transfer.mli +++ b/logseq_sync/lib/asset_transfer/core.mli @@ -1,20 +1,3 @@ -(** {1 TLDR} - - Coordinate bounded cache lookup and download demand for explicit asset - versions. The caller owns {!t}; {!step} performs no I/O or persistence and - returns the next state plus ordered {!instruction}s for a runner. It never - discovers graph assets. {!availability} reports per-consumer progress; - successful file handles still require release through emitted instructions. - - The dependent event and instruction variants appear below; these are excerpts - of the actual entry points, not additional declarations: - {[ - val create : config -> scope:Core.graph_scope -> online:bool -> unlocked:bool -> t - val step : t -> event -> t * instruction list - val availability : t -> consumer:string - -> (Logseq_db_types.Graph_types.Uuid.t * availability) list - ]} *) - module Asset = Logseq_db_types.Asset_descriptor type config @@ -28,26 +11,11 @@ val config type t -val create : config -> scope:Core.graph_scope -> online:bool -> unlocked:bool -> t +val create : config -> scope:Logseq_sync_domain.Core.graph_scope -> online:bool -> unlocked:bool -> t + -(** {1 Minimal usage example} - The caller supplies validated [config], current [scope], connectivity flags, - an explicit [consumer] and [assets], plus [interpret]: - {[ - let state = ref (create config ~scope ~online ~unlocked) in - let next, instructions = - step !state (Replace { consumer; priority = Foreground; assets }) - in - state := next; - List.iter interpret instructions; - availability !state ~consumer - ]} - [interpret] queues cache/download/timer completions for later serialized turns. - On retirement, feed [Release consumer] (or [Shutdown] for the whole owner), - install its next state and interpret every cleanup instruction the same way. *) -(** {1 Capability details} *) type priority = | Foreground @@ -85,24 +53,18 @@ type lookup_source = ; remote_version : Asset.version } -(** {1 Lifecycle and recovery} - Tickets are created by this reducer for one graph scope, asset and lookup - source/version. The runner returns the exact ticket; {!ticket_current} checks - current ownership before I/O. Replacing demand, version changes and shutdown - can retire work. Stale successful completions may emit [Release_handle] and - must still be interpreted. Tickets are not persisted across restart. *) type lookup_ticket = private { id : int - ; scope : Core.graph_scope + ; scope : Logseq_sync_domain.Core.graph_scope ; asset : Logseq_db_types.Graph_types.Uuid.t ; source : lookup_source } type fetch_ticket = private { id : int - ; scope : Core.graph_scope + ; scope : Logseq_sync_domain.Core.graph_scope ; asset : Logseq_db_types.Graph_types.Uuid.t ; version : Asset.version } @@ -145,7 +107,7 @@ type event = | Unlock_changed of bool | Shutdown -(** {1 Observation and helpers} *) + val availability : t diff --git a/logseq_sync/lib/asset_transfer/dune b/logseq_sync/lib/asset_transfer/dune new file mode 100644 index 00000000..2fbece54 --- /dev/null +++ b/logseq_sync/lib/asset_transfer/dune @@ -0,0 +1,5 @@ +(library + (name logseq_sync_asset_transfer) + (public_name logseq_sync.asset_transfer) + (modules core) + (libraries logseq_sync.domain logseq_db_types)) diff --git a/logseq_sync/lib/assets/cache.ml b/logseq_sync/lib/assets/cache.ml new file mode 100644 index 00000000..d0a1d4e4 --- /dev/null +++ b/logseq_sync/lib/assets/cache.ml @@ -0,0 +1,1101 @@ +module Failure_diagnostic = Logseq_db_types.Failure_diagnostic +module Asset_codec = Logseq_sync_native_crypto.Asset_codec +module Core = Namespace +module Asset = Logseq_db_types.Asset_descriptor +module Uuid = Logseq_db_types.Graph_types.Uuid + +type handle = string + +type error = + | Invalid of string + | Io of string + | Full + | Stale + | Checksum_mismatch + +type manifest = + { asset : Uuid.t + ; file_type : string + ; checksum : string + ; size : int64 + } +[@@deriving yojson] + +type record = + { name : string + ; file_type : string + ; checksum : string + ; size : int64 + ; mutable touched : float + ; mutable pins : int + } + +type staged_record = + { staged_name : string + ; staged_location : string + ; mutable staged_pins : int + ; mutable retired : bool + } + +type t = + { directory : string + ; budget : int64 + ; maximum_file_bytes : int + ; records : (string, record) Hashtbl.t + ; handles : (handle, record) Hashtbl.t + ; staged_records : (string, staged_record) Hashtbl.t + ; staged_handles : (handle, staged_record) Hashtbl.t + ; mutable closed : bool + } + +let next_handle = Atomic.make 0 +let ( let* ) = Result.bind + +let protect f = + try f () with + | Failure source as error -> + let diagnostic = + Failure_diagnostic.of_source + ~operation:"assetCache" + ~reason:"persistenceFailed" + source + in + let diagnostic = + if + Failure_diagnostic.source_class diagnostic = "unknown" + && Option.is_none (Failure_diagnostic.decode source) + then + Failure_diagnostic.of_source + ~operation:"assetCache" + ~reason:"persistenceFailed" + (Printexc.to_string error) + else diagnostic + in + Error (Io (Failure_diagnostic.encode diagnostic)) + | (Sys_error _ | Unix.Unix_error _) as error -> + Error + (Io + Failure_diagnostic.( + of_source + ~operation:"assetCache" + ~reason:"persistenceFailed" + (Printexc.to_string error) + |> encode)) + | Yojson.Json_error _ | Yojson.Safe.Util.Type_error _ -> + Error (Invalid "Asset cache metadata shape is invalid") +;; + +let rec mkdir path = + if not (Sys.file_exists path) + then ( + mkdir (Filename.dirname path); + Unix.mkdir path 0o700; + let parent = Unix.openfile (Filename.dirname path) [ Unix.O_RDONLY ] 0 in + Fun.protect ~finally:(fun () -> Unix.close parent) (fun () -> Unix.fsync parent)) +;; + +let unlink path = + try Unix.unlink path with + | Unix.Unix_error (Unix.ENOENT, _, _) -> () +;; + +(* Data files carry the validated attachment type so native presentation + (for example Quick Look) classifies them correctly. *) +let data_path t record = Filename.concat t.directory (record.name ^ "." ^ record.file_type) +let manifest_path t name = Filename.concat t.directory (name ^ ".json") + +let name asset (version : Asset.version) = + Asset_codec.checksum + (Uuid.to_string asset ^ ":" ^ version.file_type ^ ":" ^ version.checksum) +;; + +let sync_directory t = + let fd = Unix.openfile t.directory [ Unix.O_RDONLY ] 0 in + Fun.protect ~finally:(fun () -> Unix.close fd) (fun () -> Unix.fsync fd) +;; + +let write path bytes = + let fd = Unix.openfile path [ Unix.O_WRONLY; Unix.O_CREAT; Unix.O_TRUNC ] 0o600 in + Fun.protect + ~finally:(fun () -> Unix.close fd) + (fun () -> + let rec loop offset = + if offset < String.length bytes + then ( + let count = + Unix.write_substring fd bytes offset (String.length bytes - offset) + in + if count = 0 + then raise (Sys_error "Asset file write made no progress") + else loop (offset + count)) + in + loop 0; + Unix.fsync fd) +;; + +let checksum_file path = + let channel = open_in_bin path in + Fun.protect + ~finally:(fun () -> close_in channel) + (fun () -> + let buffer = Bytes.create 65536 in + let rec read context = + match input channel buffer 0 (Bytes.length buffer) with + | 0 -> Digestif.SHA256.(to_hex (get context)) + | length -> read (Digestif.SHA256.feed_bytes context ~off:0 ~len:length buffer) + in + read Digestif.SHA256.empty) +;; + +let remove_record t record = + unlink (manifest_path t record.name); + unlink (data_path t record); + Hashtbl.remove t.records record.name +;; + +let pin t record = + record.pins <- record.pins + 1; + record.touched <- Unix.gettimeofday (); + Unix.utimes (manifest_path t record.name) record.touched record.touched; + let handle = Printf.sprintf "asset:%d" (Atomic.fetch_and_add next_handle 1) in + Hashtbl.add t.handles handle record; + handle +;; + +let read_manifest directory file = + let path = Filename.concat directory file in + let invalid message = Error (Invalid message) in + try + if (Unix.stat path).st_size > 1024 + then invalid "Asset cache manifest exceeds size limit" + else ( + match + Logseq_db_types.Json_codec.decode manifest_of_yojson (Yojson.Safe.from_file path) + with + | Error _ -> invalid "Asset cache manifest shape is invalid" + | Ok manifest -> + (match + Asset.version ~checksum:manifest.checksum ~file_type:manifest.file_type + with + | Error _ -> invalid "Asset cache manifest asset version is invalid" + | Ok _ when manifest.size < 0L -> invalid "Asset cache manifest size is invalid" + | Ok version when name manifest.asset version ^ ".json" <> file -> + invalid "Asset cache manifest filename does not match its identity" + | Ok version -> + Ok + { name = name manifest.asset version + ; file_type = version.file_type + ; checksum = version.checksum + ; size = manifest.size + ; touched = (Unix.stat path).st_mtime + ; pins = 0 + })) + with + | Yojson.Json_error _ -> invalid "Asset cache manifest JSON is invalid" + | Failure _ | Invalid_argument _ -> invalid "Asset cache manifest shape is invalid" +;; + +type account_identity = string * string [@@deriving to_yojson] + +let account_directory ~root (account : Core.account_scope) = + let name = + Asset_codec.checksum + (account_identity_to_yojson + (Uri.to_string account.managed_sync_origin, account.user_id) + |> Yojson.Safe.to_string) + in + Filename.concat root name +;; + +let lookup t ~asset ~version = + protect (fun () -> + if t.closed + then Error Stale + else ( + match Hashtbl.find_opt t.records (name asset version) with + | None -> Ok None + | Some record -> + let path = data_path t record in + let valid = + try + let stat = Unix.stat path in + stat.st_kind = Unix.S_REG + && Int64.of_int stat.st_size = record.size + && stat.st_size <= t.maximum_file_bytes + && checksum_file path = record.checksum + with + | Sys_error _ | Unix.Unix_error _ -> false + in + if valid + then Ok (Some (pin t record)) + else ( + if record.pins = 0 then remove_record t record; + Ok None))) +;; + +let report t = + protect (fun () -> + let resident_bytes = + Hashtbl.fold (fun _ record total -> Int64.add total record.size) t.records 0L + in + let retained_cache = + Hashtbl.fold + (fun _ record total -> + if record.pins > 0 then Int64.add total record.size else total) + t.records + 0L + in + let directory = Filename.concat t.directory "pending" in + let pending_bytes = + if Sys.file_exists directory + then + Array.fold_left + (fun total file -> + let metadata = Unix.lstat (Filename.concat directory file) in + if metadata.st_kind = Unix.S_REG + then Int64.add total (Int64.of_int metadata.st_size) + else total) + 0L + (Sys.readdir directory) + else 0L + in + let retained_staging = + Hashtbl.fold + (fun _ record total -> + if record.staged_pins > 0 + then Int64.add total (Int64.of_int (Unix.stat record.staged_location).st_size) + else total) + t.staged_records + 0L + in + let cleanup_pending = + Hashtbl.fold + (fun _ record count -> if record.retired then count + 1 else count) + t.staged_records + 0 + in + Ok + { Sync_dsl.Core.resident_bytes + ; pending_bytes + ; retained_bytes = Int64.add retained_cache retained_staging + ; cleanup_pending + ; over_budget = resident_bytes > t.budget + }) +;; + +let reserve t needed = + let usage = + Hashtbl.fold (fun _ record total -> Int64.add total record.size) t.records 0L + in + let candidates = + Hashtbl.fold + (fun _ record acc -> if record.pins = 0 then record :: acc else acc) + t.records + [] + |> List.sort (fun a b -> Float.compare a.touched b.touched) + in + let rec evict usage = function + | _ when needed <= t.budget && usage <= Int64.sub t.budget needed -> Ok () + | [] -> Error Full + | record :: rest -> + remove_record t record; + evict (Int64.sub usage record.size) rest + in + if needed > t.budget then Error Full else evict usage candidates +;; + +let prune t = + let* () = protect (fun () -> reserve t 0L) in + report t +;; + +let publish t ~asset ~current ~verified = + let version = Asset_codec.verified_version verified in + let plaintext = Asset_codec.verified_bytes verified in + protect (fun () -> + if t.closed || not (current ()) + then Error Stale + else if String.length plaintext > t.maximum_file_bytes + then Error (Invalid "Asset exceeds cache file limit") + else if Asset_codec.checksum plaintext <> version.checksum + then Error Checksum_mismatch + else + let* existing = lookup t ~asset ~version in + match existing with + | Some handle -> Ok handle + | None -> + let size = Int64.of_int (String.length plaintext) in + let* () = reserve t size in + let name = name asset version in + (* A corrupt pinned record cannot be replaced beneath a renderer. *) + if Hashtbl.mem t.records name + then Error Full + else ( + let record = + { name + ; file_type = version.file_type + ; checksum = version.checksum + ; size + ; touched = Unix.gettimeofday () + ; pins = 0 + } + in + let data = data_path t record + and manifest = manifest_path t name in + let temporary = data ^ ".part" + and temporary_manifest = manifest ^ ".part" in + Fun.protect + ~finally:(fun () -> + unlink temporary; + unlink temporary_manifest) + (fun () -> + write temporary plaintext; + write + temporary_manifest + (manifest_to_yojson + { asset + ; file_type = version.file_type + ; checksum = version.checksum + ; size + } + |> Yojson.Safe.to_string); + if t.closed || not (current ()) + then Error Stale + else ( + Unix.rename temporary data; + Unix.rename temporary_manifest manifest; + sync_directory t; + Hashtbl.replace t.records name record; + Ok (pin t record))))) +;; + +let pin_staged t record = + record.staged_pins <- record.staged_pins + 1; + let handle = Printf.sprintf "staged:%d" (Atomic.fetch_and_add next_handle 1) in + Hashtbl.add t.staged_handles handle record; + handle +;; + +let report_cleanup ?(code = "collectionPending") operation error = + let source = + match error with + | Invalid source | Io source -> source + | Full -> "Asset staging storage is full" + | Stale -> "Asset staging scope is stale" + | Checksum_mismatch -> "Asset staging checksum mismatch" + in + let diagnostic = + Failure_diagnostic.of_source ~operation:"assetRelease" ~reason:"cleanupFailed" source + in + Printf.eprintf + "journal: phase=cleanup operation=%s code=%s source=%s\n%!" + operation + code + (Failure_diagnostic.source_class diagnostic) +;; + +let report_cleanup_exception ?code operation error = + report_cleanup ?code operation (Io (Printexc.to_string error)) +;; + +let clean_retired_staging t record = + let result = + if record.retired && record.staged_pins = 0 + then + protect (fun () -> + unlink record.staged_location; + let directory = Filename.dirname record.staged_location in + if Sys.file_exists directory + then ( + let fd = Unix.openfile directory [ Unix.O_RDONLY ] 0 in + Fun.protect ~finally:(fun () -> Unix.close fd) (fun () -> Unix.fsync fd)); + Hashtbl.remove t.staged_records record.staged_name; + Ok ()) + else Ok () + in + (match result with + | Ok () -> () + | Error error -> report_cleanup ~code:"cleanupPending" "releaseRetiredStaging" error); + result +;; + +let path t handle = + if t.closed + then None + else ( + match Hashtbl.find_opt t.handles handle with + | Some record -> Some (data_path t record) + | None -> + Option.map + (fun record -> record.staged_location) + (Hashtbl.find_opt t.staged_handles handle)) +;; + +let retain t handle = + if t.closed + then None + else ( + match Hashtbl.find_opt t.handles handle with + | Some record -> Some (pin t record) + | None -> Option.map (pin_staged t) (Hashtbl.find_opt t.staged_handles handle)) +;; + +let release_checked t handle = + match Hashtbl.find_opt t.handles handle with + | Some record -> + record.pins <- record.pins - 1; + Hashtbl.remove t.handles handle; + Ok () + | None -> + (match Hashtbl.find_opt t.staged_handles handle with + | None -> Ok () + | Some record -> + record.staged_pins <- record.staged_pins - 1; + let result = if record.retired then clean_retired_staging t record else Ok () in + (match result with + | Error _ -> + (* The exact handle still owns this failed release. Restoring its pin + permits a same-handle retry without a second decrement. *) + record.staged_pins <- record.staged_pins + 1; + result + | Ok () -> + Hashtbl.remove t.staged_handles handle; + if (not record.retired) && record.staged_pins = 0 + then Hashtbl.remove t.staged_records record.staged_name; + Ok ())) +;; + +let release t handle = ignore (release_checked t handle) + +let close t = + t.closed <- true; + Hashtbl.clear t.handles; + Hashtbl.iter (fun _ record -> record.pins <- 0) t.records; + Hashtbl.clear t.staged_handles; + Hashtbl.fold (fun _ record records -> record :: records) t.staged_records [] + |> List.iter (fun record -> + record.staged_pins <- 0; + ignore (clean_retired_staging t record)); + Hashtbl.clear t.staged_records +;; + +let close_checked t = + if Hashtbl.length t.handles > 0 || Hashtbl.length t.staged_handles > 0 + then Error (Invalid "Attachment resources are still retained.") + else ( + let retired = + Hashtbl.fold + (fun _ record records -> Rrbvec.push_back records record) + t.staged_records + Rrbvec.empty + in + let cleaned = + Rrbvec.fold_left + (fun result record -> + Result.bind result (fun () -> clean_retired_staging t record)) + (Ok ()) + retired + in + Result.map (fun () -> close t) cleaned) +;; + +let remove_tree path = + let rec remove path = + match Unix.lstat path with + | { Unix.st_kind = Unix.S_DIR; _ } -> + Array.iter (fun name -> remove (Filename.concat path name)) (Sys.readdir path); + Unix.rmdir path + | _ -> Unix.unlink path + | exception Unix.Unix_error (Unix.ENOENT, _, _) -> () + in + protect (fun () -> + remove path; + Ok ()) +;; + +external rename_exclusive : string -> string -> int = "logseq_journal_rename_exclusive" + +let graph_deletion_paths ~root (scope : Core.graph_scope) retirement = + let account = account_directory ~root scope.account in + let graph = Uuid.to_string scope.graph_id in + let directory = + Filename.concat + (Filename.concat + (Filename.concat root ".asset-retirements") + (Filename.basename account)) + graph + in + ( Filename.concat account graph + , directory + , Filename.concat + (Filename.concat directory "states") + (Core.retirement_id_to_string retirement ^ ".json") + , Filename.concat + (Filename.concat directory "pending") + (Core.retirement_id_to_string retirement ^ ".data") ) +;; + +let sync_path directory = + let fd = Unix.openfile directory [ Unix.O_RDONLY ] 0 in + Fun.protect ~finally:(fun () -> Unix.close fd) (fun () -> Unix.fsync fd) +;; + +let write_retirement path json = + let temporary = path ^ ".part" in + write temporary (Yojson.Safe.to_string json); + Unix.rename temporary path; + sync_path (Filename.dirname path) +;; + +let identity path = + match Unix.lstat path with + | { Unix.st_kind = Unix.S_DIR; st_dev; st_ino; _ } -> + Printf.sprintf "%d:%d" st_dev st_ino + | _ -> failwith "asset namespace is not a directory" +;; + +let retirement_state manifest_path = + let open Yojson.Safe.Util in + let json = Yojson.Safe.from_file manifest_path in + if json |> member "version" |> to_int <> 1 then failwith "unsupported asset retirement"; + json |> member "identity" |> to_string, json |> member "state" |> to_string +;; + +let retirement_json captured state = + `Assoc [ "version", `Int 1; "identity", `String captured; "state", `String state ] +;; + +let graph_deletion_terminal ~root ~scope ~retirement = + protect (fun () -> + let _, _, manifest, _ = graph_deletion_paths ~root scope retirement in + if not (Sys.file_exists manifest) + then Ok None + else ( + let _, state = retirement_state manifest in + Ok + (match state with + | "complete" | "committed" -> + let fd = Unix.openfile manifest [ Unix.O_RDONLY ] 0 in + Fun.protect ~finally:(fun () -> Unix.close fd) (fun () -> Unix.fsync fd); + sync_path (Filename.dirname manifest); + Some `Committed + | "aborted" | "account-deleted" -> Some `Aborted + | _ -> None))) +;; + +let prepare_graph_deletion ~root ~(scope : Core.graph_scope) ~retirement = + protect (fun () -> + let canonical, directory, manifest, data = + graph_deletion_paths ~root scope retirement + in + mkdir directory; + mkdir (Filename.dirname manifest); + mkdir (Filename.dirname data); + let captured, state = + if Sys.file_exists manifest + then retirement_state manifest + else ( + let captured = + if Sys.file_exists canonical then identity canonical else "absent" + in + write_retirement manifest (retirement_json captured "preparing"); + captured, "preparing") + in + if state <> "preparing" && state <> "prepared" + then Error Stale + else ( + if Sys.file_exists data + then (if identity data <> captured then failwith "retired asset identity changed") + else if captured <> "absent" + then ( + if (not (Sys.file_exists canonical)) || identity canonical <> captured + then failwith "canonical asset identity changed"; + let error = rename_exclusive canonical data in + if error <> 0 + then ( + Printf.eprintf + "journal: phase=cleanup operation=prepareAssetRetirement code=renameFailed \ + errno=%d\n\ + %!" + error; + failwith "asset retirement rename failed"); + sync_path (Filename.dirname canonical); + sync_path (Filename.dirname data)); + write_retirement manifest (retirement_json captured "prepared"); + Ok ())) +;; + +let finish_graph_deletion ~root ~deletion ~commit = + protect (fun () -> + let scope = Core.asset_deletion_scope deletion in + let canonical, _directory, manifest, data = + graph_deletion_paths ~root scope (Core.asset_deletion_retirement deletion) + in + if not (Sys.file_exists manifest) + then + if Sys.file_exists data + then Error Stale + else if commit + then Error Stale + else ( + mkdir (Filename.dirname manifest); + write_retirement manifest (retirement_json "absent" "aborted"); + Ok ()) + else ( + let captured, state = retirement_state manifest in + if state = "account-deleted" + then ( + (match remove_tree data with + | Ok () -> () + | Error error -> report_cleanup "collectDeletedAccountAssets" error); + Ok ()) + else if state = "aborted" || state = "complete" + then Ok () + else if commit + then ( + if Sys.file_exists data && identity data <> captured + then failwith "retired asset identity changed"; + write_retirement manifest (retirement_json captured "committed"); + (* The producer owns deferred collection. A purge failure must not + block a replacement cache or undo the durable commit decision. *) + (match remove_tree data with + | Error error -> report_cleanup "collectRetiredAssets" error + | Ok () -> + (try + sync_path (Filename.dirname data); + write_retirement manifest (retirement_json captured "complete") + with + | error -> + report_cleanup_exception ~code:"cleanupPending" "completeRetirement" error)); + Ok ()) + else if state = "committed" + then Ok () + else ( + if Sys.file_exists data + then ( + if identity data <> captured then failwith "retired asset identity changed"; + mkdir (Filename.dirname canonical); + let error = rename_exclusive data canonical in + if error <> 0 + then ( + Printf.eprintf + "journal: phase=cleanup operation=restoreRetiredAssets code=renameFailed \ + errno=%d\n\ + %!" + error; + failwith "asset restore conflict; retirement retained"); + sync_path (Filename.dirname canonical); + sync_path (Filename.dirname data)) + else if + captured <> "absent" + && ((not (Sys.file_exists canonical)) || identity canonical <> captured) + then failwith "asset restore is incomplete; retirement retained"; + if Sys.file_exists (Filename.dirname canonical) + then sync_path (Filename.dirname canonical); + if Sys.file_exists (Filename.dirname data) then sync_path (Filename.dirname data); + write_retirement manifest (retirement_json captured "aborted"); + Ok ()))) +;; + +let collect_retired_assets ~root ~(scope : Core.graph_scope) = + let _, directory, _, _ = + graph_deletion_paths + ~root + scope + (Core.retirement_id_of_string "00000000-0000-4000-8000-000000000000" + |> Result.get_ok) + in + let pending = Filename.concat directory "pending" in + if Sys.file_exists pending + then + Array.iter + (fun name -> + if Filename.check_suffix name ".data" + then ( + try + let retirement = + Filename.chop_suffix name ".data" + |> Core.retirement_id_of_string + |> Result.get_ok + in + let _, _, manifest, _ = graph_deletion_paths ~root scope retirement in + let _, state = retirement_state manifest in + if state = "committed" || state = "complete" || state = "account-deleted" + then ( + match + finish_graph_deletion + ~root + ~deletion:(Core.asset_deletion_reference ~scope ~retirement) + ~commit:true + with + | Ok () -> () + | Error error -> report_cleanup "finishRetiredAssets" error) + with + | error -> report_cleanup_exception "collectRetiredAssets" error)) + (Sys.readdir pending) +;; + +let delete_account ~root ~account = + protect (fun () -> + let directory = account_directory ~root account in + let retired = + Filename.concat + (Filename.concat root ".asset-retirements") + (Filename.basename directory) + in + if Sys.file_exists retired + then + Array.iter + (fun graph -> + match Uuid.of_string graph with + | Error _ -> () + | Ok _ -> + let folder = Filename.concat retired graph in + let states = Filename.concat folder "states" in + if Sys.file_exists states + then + Array.iter + (fun file -> + if Filename.check_suffix file ".json" + then ( + let manifest = Filename.concat states file in + let captured, _ = retirement_state manifest in + write_retirement + manifest + (retirement_json captured "account-deleted"))) + (Sys.readdir states); + (match remove_tree (Filename.concat folder "pending") with + | Ok () -> () + | Error error -> + report_cleanup "collectDeletedAccountAssets" error; + let source = + match error with + | Invalid source | Io source -> source + | Full -> "Asset staging storage is full" + | Stale -> "Asset staging scope is stale" + | Checksum_mismatch -> "Asset staging checksum mismatch" + in + failwith + Failure_diagnostic.( + of_source ~operation:"assetRelease" ~reason:"cleanupFailed" source + |> encode))) + (Sys.readdir retired); + remove_tree directory) +;; + +let create ~root ~(scope : Core.graph_scope) ~budget_bytes ~maximum_file_bytes = + (try collect_retired_assets ~root ~scope with + | error -> report_cleanup_exception "collectRetiredAssets" error); + protect (fun () -> + if + budget_bytes < 0L + || maximum_file_bytes < 0 + || maximum_file_bytes > 100 * 1024 * 1024 + then Error (Invalid "Invalid cache size limits") + else ( + let directory = + Filename.concat + (account_directory ~root scope.account) + (Uuid.to_string scope.graph_id) + in + mkdir directory; + (* Validate all persisted JSON before cleaning anything. Unsupported + internal formats require an explicit caller-visible failure. *) + let rec load_manifests records = function + | [] -> Ok (List.rev records) + | file :: rest when Filename.check_suffix file ".json" -> + let* record = read_manifest directory file in + load_manifests ((file, record) :: records) rest + | _ :: rest -> load_manifests records rest + in + let* manifests = load_manifests [] (Array.to_list (Sys.readdir directory)) in + let pending = Filename.concat directory "pending" in + if Sys.file_exists pending + then + Array.iter + (fun file -> + if Filename.check_suffix file ".part" + then unlink (Filename.concat pending file)) + (Sys.readdir pending); + let t = + { directory + ; budget = budget_bytes + ; maximum_file_bytes + ; records = Hashtbl.create 64 + ; handles = Hashtbl.create 64 + ; staged_records = Hashtbl.create 8 + ; staged_handles = Hashtbl.create 8 + ; closed = false + } + in + List.iter + (fun (file, (record : record)) -> + if + record.size <= Int64.of_int maximum_file_bytes + && Sys.file_exists (data_path t record) + then Hashtbl.replace t.records record.name record + else unlink (Filename.concat directory file)) + manifests; + Array.iter + (fun file -> + if Filename.check_suffix file ".part" + then unlink (Filename.concat directory file)) + (Sys.readdir directory); + let kept = Hashtbl.create (2 * Hashtbl.length t.records) in + Hashtbl.iter + (fun _ record -> + Hashtbl.replace kept (record.name ^ ".json") (); + Hashtbl.replace kept (record.name ^ "." ^ record.file_type) ()) + t.records; + Array.iter + (fun file -> + let path = Filename.concat directory file in + if (not (Hashtbl.mem kept file)) && not (Sys.is_directory path) + then unlink path) + (Sys.readdir directory); + Ok t)) +;; + +type staged = + { file : string + ; checksum : string + ; size : int64 + } + +let pending_directory t = Filename.concat t.directory "pending" + +let valid_staged_type file_type = + Result.is_ok (Asset.version ~checksum:(String.make 64 '0') ~file_type) +;; + +let valid_staged_nonce nonce = + String.length nonce = 64 + && String.for_all + (function + | '0' .. '9' | 'a' .. 'f' -> true + | _ -> false) + nonce +;; + +let valid_staged_file file = + Filename.basename file = file + && + match String.split_on_char '.' file with + | [ operation; file_type ] -> + Result.is_ok (Uuid.of_string operation) && valid_staged_type file_type + | [ operation; nonce; file_type ] -> + Result.is_ok (Uuid.of_string operation) + && valid_staged_nonce nonce + && valid_staged_type file_type + | _ -> false +;; + +let staged_path t ~file = + if t.closed || not (valid_staged_file file) + then None + else ( + let path = Filename.concat (pending_directory t) file in + match Unix.lstat path with + | { Unix.st_kind = Unix.S_REG; _ } -> Some path + | _ -> None + | exception Unix.Unix_error _ -> None) +;; + +let retain_staged t ~file = + Option.bind (staged_path t ~file) (fun location -> + match Hashtbl.find_opt t.staged_records file with + | Some record when record.retired -> None + | Some record -> Some (pin_staged t record) + | None -> + let record = + { staged_name = file + ; staged_location = location + ; staged_pins = 0 + ; retired = false + } + in + Hashtbl.add t.staged_records file record; + Some (pin_staged t record)) +;; + +let release_staged t ~file = + if t.closed + then Error Stale + else if not (valid_staged_file file) + then Error (Invalid "Invalid staged file") + else ( + let record = + match Hashtbl.find_opt t.staged_records file with + | Some record -> record + | None -> + { staged_name = file + ; staged_location = Filename.concat (pending_directory t) file + ; staged_pins = 0 + ; retired = true + } + in + record.retired <- true; + clean_retired_staging t record) +;; + +let release_staged_if_unused t ~file = + match Hashtbl.find_opt t.staged_records file with + | Some record when record.staged_pins > 0 -> Error Full + | Some _ | None -> release_staged t ~file +;; + +let stage t ~operation ~file_type ~source_file ~pending_budget_bytes = + if t.closed + then Error Stale + else if not (valid_staged_type file_type) + then Error (Invalid "Invalid staged file type") + else if pending_budget_bytes < 0L + then Error (Invalid "Invalid staging budget") + else + protect (fun () -> + let directory = pending_directory t in + mkdir directory; + (* A delayed completion must name this physical instance, even if its + business operation is staged again after pruning or recovery. *) + let nonce = Asset_codec.checksum (Mirage_crypto_rng_unix.getrandom 16) in + let file = Uuid.to_string operation ^ "." ^ nonce ^ "." ^ file_type in + let target = Filename.concat directory file in + let partial = target ^ ".part" in + if Sys.file_exists target + then Error (Invalid "Staged import already exists") + else ( + let source = + Unix.openfile source_file [ Unix.O_RDONLY; Unix.O_NONBLOCK; Unix.O_CLOEXEC ] 0 + in + Fun.protect + ~finally:(fun () -> Unix.close source) + (fun () -> + let metadata = Unix.fstat source in + if metadata.st_kind <> Unix.S_REG + then Error (Invalid "Import source must be a regular file") + else if metadata.st_size > t.maximum_file_bytes + then Error (Invalid "Import exceeds file size limit") + else ( + let files = Sys.readdir directory in + let used = + Array.fold_left + (fun total name -> + let size = (Unix.lstat (Filename.concat directory name)).st_size in + Int64.add total (Int64.of_int (max 1 size))) + 0L + files + in + let allocation = Int64.of_int (max 1 metadata.st_size) in + if + Array.length files >= 4096 + || used > Int64.sub pending_budget_bytes allocation + then Error Full + else ( + let output = + Unix.openfile + partial + [ Unix.O_WRONLY; Unix.O_CREAT; Unix.O_EXCL; Unix.O_CLOEXEC ] + 0o600 + in + Fun.protect + ~finally:(fun () -> + Unix.close output; + unlink partial) + (fun () -> + let buffer = Bytes.create 65536 in + let rec copy count digest = + match Unix.read source buffer 0 (Bytes.length buffer) with + | 0 -> Ok (count, digest) + | length -> + if count + length > metadata.st_size + then Error (Invalid "Import source changed during staging") + else ( + let rec write offset = + if offset < length + then ( + let written = + Unix.write output buffer offset (length - offset) + in + if written = 0 + then raise (Sys_error "Staging write made no progress") + else write (offset + written)) + in + write 0; + copy + (count + length) + (Digestif.SHA256.feed_bytes + digest + ~off:0 + ~len:length + buffer)) + in + let* size, digest = copy 0 Digestif.SHA256.empty in + let after = Unix.fstat source in + if + size <> metadata.st_size + || after.st_mtime <> metadata.st_mtime + || after.st_size <> metadata.st_size + then Error (Invalid "Import source changed during staging") + else ( + Unix.fsync output; + (* Publish without replacing another physical instance. *) + Unix.link partial target; + unlink partial; + let fd = Unix.openfile directory [ Unix.O_RDONLY ] 0 in + Fun.protect + ~finally:(fun () -> Unix.close fd) + (fun () -> Unix.fsync fd); + Ok + { file + ; checksum = Digestif.SHA256.to_hex (Digestif.SHA256.get digest) + ; size = Int64.of_int size + }))))))) +;; + +let prune_staged t ~keep = + if t.closed + then Error Stale + else + protect (fun () -> + let directory = pending_directory t in + if not (Sys.file_exists directory) + then Ok 0 + else ( + let stream = Unix.opendir directory in + let rec inspect count orphaned = + match Unix.readdir stream with + | "." | ".." -> inspect count orphaned + | _ when count >= 4096 -> Error Full + | file -> + let path = Filename.concat directory file in + if + (not (valid_staged_file file)) + || + match Hashtbl.find_opt t.staged_records file with + | Some record -> record.staged_pins > 0 + | None -> false + then inspect (count + 1) orphaned + else ( + match Unix.lstat path with + | { Unix.st_kind = Unix.S_REG; _ } -> + (match keep file with + | Error message -> Error (Io message) + | Ok true -> inspect (count + 1) orphaned + | Ok false -> inspect (count + 1) (file :: orphaned)) + | _ -> inspect (count + 1) orphaned + | exception Unix.Unix_error (Unix.ENOENT, _, _) -> + inspect (count + 1) orphaned) + | exception End_of_file -> Ok orphaned + in + match + Fun.protect ~finally:(fun () -> Unix.closedir stream) (fun () -> inspect 0 []) + with + | Error _ as error -> error + | Ok orphaned -> + List.iter (fun file -> unlink (Filename.concat directory file)) orphaned; + if orphaned <> [] + then ( + let fd = Unix.openfile directory [ Unix.O_RDONLY ] 0 in + Fun.protect ~finally:(fun () -> Unix.close fd) (fun () -> Unix.fsync fd)); + Ok (List.length orphaned))) +;; diff --git a/logseq_sync/lib/assets/cache.mli b/logseq_sync/lib/assets/cache.mli new file mode 100644 index 00000000..6f760825 --- /dev/null +++ b/logseq_sync/lib/assets/cache.mli @@ -0,0 +1,98 @@ +(** The owner serializes calls. Handles pin bytes until checked release. *) +type t + +type handle = string + +type error = + | Invalid of string + | Io of string + | Full + | Stale + | Checksum_mismatch + +val create + : root:string + -> scope:Namespace.graph_scope + -> budget_bytes:int64 + -> maximum_file_bytes:int + -> (t, error) result + +val report : t -> (Sync_dsl.Core.cache_report, error) result +val prune : t -> (Sync_dsl.Core.cache_report, error) result + +val lookup + : t + -> asset:Logseq_db_types.Graph_types.Uuid.t + -> version:Logseq_db_types.Asset_descriptor.version + -> (handle option, error) result + +val publish + : t + -> asset:Logseq_db_types.Graph_types.Uuid.t + -> current:(unit -> bool) + -> verified:Logseq_sync_native_crypto.Asset_codec.verified + -> (handle, error) result + +val path : t -> handle -> string option +val retain : t -> handle -> handle option +val release : t -> handle -> unit +val release_checked : t -> handle -> (unit, error) result + +(** Preserve all retained handles and failed cleanup ownership on error. *) +val close_checked : t -> (unit, error) result + +val close : t -> unit + +(** Remove durable files for one origin/account after its live caches are closed. *) +val delete_account + : root:string + -> account:Namespace.account_scope + -> (unit, error) result + +val graph_deletion_terminal + : root:string + -> scope:Namespace.graph_scope + -> retirement:Namespace.retirement_id + -> ([ `Committed | `Aborted ] option, error) result + +val prepare_graph_deletion + : root:string + -> scope:Namespace.graph_scope + -> retirement:Namespace.retirement_id + -> (unit, error) result + +val finish_graph_deletion + : root:string + -> deletion:Namespace.asset_deletion + -> commit:bool + -> (unit, error) result + +type staged = private + { file : string + ; checksum : string + ; size : int64 + } + +(** Copy an explicit import into the durable, non-evictable pending namespace. + Each copy has a distinct physical file identity, including copies of the + same operation. Legacy operation-only filenames remain readable. + The caller persists the returned identity before graph mutation or upload. *) +val stage + : t + -> operation:Logseq_db_types.Graph_types.Uuid.t + -> file_type:string + -> source_file:string + -> pending_budget_bytes:int64 + -> (staged, error) result + +(** Retain a staged file for a local preview. Completion cleanup waits for all leases. *) +val retain_staged : t -> file:string -> handle option + +val staged_path : t -> file:string -> string option +val release_staged : t -> file:string -> (unit, error) result +val release_staged_if_unused : t -> file:string -> (unit, error) result + +(** Remove recognized orphan staging files only after all ownership checks succeed. + Ownership checks receive the exact physical file identity, including legacy + filenames. The caller must serialize this with staging and intent persistence. *) +val prune_staged : t -> keep:(string -> (bool, string) result) -> (int, error) result diff --git a/logseq_sync/lib/assets/core.ml b/logseq_sync/lib/assets/core.ml new file mode 100644 index 00000000..596b7eba --- /dev/null +++ b/logseq_sync/lib/assets/core.ml @@ -0,0 +1,563 @@ +module Dsl = Sync_dsl.Core +module Intent = Logseq_db_types.Asset_upload_intent +module Asset = Logseq_db_types.Asset_descriptor +module Codec = Logseq_sync_native_crypto.Asset_codec +module Crypto = Logseq_sync_crypto.Core +module Native = Logseq_sync_native_crypto.Platform_crypto + +let ( let* ) = Result.bind + +type network = + { get : + identity:Dsl.identity + -> asset:Dsl.asset + -> version:Asset.version + -> maximum_wire_bytes:int + -> (string option, Dsl.error) result + ; put : + identity:Dsl.identity + -> asset:Logseq_db_types.Graph_types.Uuid.t + -> version:Asset.version + -> wire:string + -> (unit, Dsl.error) result + } + +type key = + check_current:(unit -> (unit, Dsl.error) result) -> (string option, Dsl.error) result + +let native_key ~identity ~crypto ~check_current = + Crypto.asset_key crypto ~check_current ~identity +;; + +module Resources = Ephemeron.K1.Make (struct + type t = Dsl.resource + + let equal left right = left == right + let hash resource = Hashtbl.hash resource.Dsl.token + end) + +type held = + { handle : Cache.handle + ; reference : Dsl.resource_reference + ; mutable released : bool + } + +type t = + { identity : Dsl.identity + ; limits : Dsl.limits + ; crypto : Crypto.t + ; key : key + ; network : network + ; cache : Cache.t + ; lock : Eio.Mutex.t + ; resources : held Resources.t + ; mutable live_resources : int + ; mutable reserved_resources : int + ; mutable active_transfers : int + ; mutable next_resource : int64 + ; mutable fenced : bool + } + +let identity t = t.identity +let limits t = t.limits +let cache t = t.cache +let crypto t = t.crypto + +let cache_error = function + | Cache.Invalid message -> Dsl.Invalid_input message + | Io message -> Dsl.Storage_failure message + | Full -> Dsl.Capacity_exceeded "Asset storage" + | Stale -> Dsl.Stale_context + | Checksum_mismatch -> Dsl.Protocol_failure "Asset checksum mismatch" +;; + +let storage result = Result.map_error cache_error result + +let create ~root ~identity ~limits ~crypto ~key ~network = + let* scope = + Namespace.of_identity identity + |> Result.map_error (fun message -> Dsl.Invalid_input message) + in + let* cache = + storage + (Cache.create + ~root + ~scope + ~budget_bytes:limits.Dsl.asset_cache_budget_bytes + ~maximum_file_bytes:limits.asset_maximum_file_bytes) + in + Ok + { identity + ; limits + ; crypto + ; key + ; network + ; cache + ; lock = Eio.Mutex.create () + ; resources = Resources.create 16 + ; live_resources = 0 + ; reserved_resources = 0 + ; active_transfers = 0 + ; next_resource = 0L + ; fenced = false + } +;; + +let locked t callback = Eio.Mutex.use_rw ~protect:true t.lock callback + +let same_owner t identity = + if identity = t.identity then Ok () else Error (Dsl.Identity_mismatch "Asset owner") +;; + +let check t check_current = + let* () = check_current () in + if t.fenced then Error Dsl.Stale_context else Ok () +;; + +let protected t = + if Crypto.status t.crypto = Dsl.Key_locked then Error Dsl.Locked else Ok () +;; + +let resource_of_reference = function + | Dsl.File_reference file -> file.file + | Lease_reference lease -> lease.lease + | Snapshot_reference snapshot -> snapshot.resource + | Connection_reference _ -> invalid_arg "Not an asset reference" +;; + +let held t resource = + match Resources.find_opt t.resources resource with + | Some held -> Ok held + | None -> Error (Dsl.Identity_mismatch "Unregistered asset resource") +;; + +let validate_resource t reference = + match reference with + | Dsl.File_reference file -> + let* owned = held t file.file in + (match owned.reference with + | Dsl.File_reference original when original == file -> Ok () + | _ -> Error (Dsl.Identity_mismatch "Asset file metadata")) + | Lease_reference lease -> + let* owned = held t lease.lease in + (match owned.reference with + | Dsl.Lease_reference original when original == lease -> Ok () + | _ -> Error (Dsl.Identity_mismatch "Asset lease metadata")) + | Snapshot_reference _ | Connection_reference _ -> + Error (Dsl.Identity_mismatch "Asset resource kind") +;; + +let reserve t = + if t.live_resources + t.reserved_resources >= t.limits.maximum_resources + then Error (Dsl.Capacity_exceeded "Asset resources") + else Ok () +;; + +let with_transfer t ~resource callback = + let* () = + locked t (fun () -> + if t.active_transfers >= t.limits.maximum_active_assets + then Error (Dsl.Capacity_exceeded "Asset transfers") + else + let* () = if resource then reserve t else Ok () in + t.active_transfers <- t.active_transfers + 1; + if resource then t.reserved_resources <- t.reserved_resources + 1; + Ok ()) + in + Fun.protect + ~finally:(fun () -> + locked t (fun () -> + t.active_transfers <- t.active_transfers - 1; + if resource then t.reserved_resources <- t.reserved_resources - 1)) + callback +;; + +let fresh t = + if t.next_resource = Int64.max_int + then Error (Dsl.Capacity_exceeded "Asset resource generation") + else ( + t.next_resource <- Int64.succ t.next_resource; + Dsl.resource + ~owner:t.identity + ~token:("asset-owner:" ^ Int64.to_string t.next_resource)) +;; + +let register t handle reference = + let resource = resource_of_reference reference in + Resources.add t.resources resource { handle; reference; released = false }; + t.live_resources <- t.live_resources + 1 +;; + +let acquire_file ?(reserved = false) t asset acquire = + let* () = if reserved then Ok () else reserve t in + let* resource = fresh t in + let* file = Dsl.asset_file resource asset in + let* handle = acquire () in + match handle with + | None -> Ok None + | Some handle -> + register t handle (Dsl.File_reference file); + Ok (Some file) +;; + +let release t identity reference = + let* () = same_owner t identity in + locked t (fun () -> + let* () = validate_resource t reference in + let* owned = held t (resource_of_reference reference) in + if owned.released + then Ok () + else + let* () = storage (Cache.release_checked t.cache owned.handle) in + owned.released <- true; + t.live_resources <- t.live_resources - 1; + Ok ()) +;; + +let release_file t ~identity (file : Dsl.asset_file) = + release t identity (Dsl.File_reference file) +;; + +let release_lease t ~identity (lease : Dsl.asset_lease) = + release t identity (Dsl.Lease_reference lease) +;; + +let file_path t (file : Dsl.asset_file) = + locked t (fun () -> + let* () = validate_resource t (Dsl.File_reference file) in + let* owned = held t file.file in + if owned.released + then Error Dsl.Stale_context + else ( + match Cache.path t.cache owned.handle with + | Some path -> Ok path + | None -> Error (Dsl.Storage_failure "Asset file disappeared"))) +;; + +let retain t (file : Dsl.asset_file) = + locked t (fun () -> + let* () = validate_resource t (Dsl.File_reference file) in + let* owned = held t file.file in + if owned.released + then Error Dsl.Stale_context + else + let* () = reserve t in + let* resource = fresh t in + let* lease = Dsl.asset_lease resource file in + match Cache.retain t.cache owned.handle with + | None -> Error Dsl.Stale_context + | Some handle -> + register t handle (Dsl.Lease_reference lease); + Ok lease) +;; + +let read_source t path = + try + let channel = open_in_bin path in + Fun.protect + ~finally:(fun () -> close_in_noerr channel) + (fun () -> + let size = in_channel_length channel in + if size > t.limits.asset_maximum_file_bytes + then Error (Dsl.Capacity_exceeded "Asset file bytes") + else ( + let bytes = really_input_string channel size in + match input_char channel with + | _ -> Error (Dsl.Storage_failure "Asset file grew") + | exception End_of_file -> Ok bytes)) + with + | End_of_file -> Error (Dsl.Storage_failure "Asset file shrank") + | Sys_error message -> Error (Dsl.Storage_failure message) +;; + +let codec_crypto = + { Codec.encrypt = Native.crypto.encrypt_aes_gcm + ; decrypt = Native.crypto.decrypt_aes_gcm + } +;; + +let verified t version bytes = + Codec.decode + ~maximum_plaintext_bytes:t.limits.asset_maximum_file_bytes + ~crypto:codec_crypto + ~key:None + ~expected:version + bytes + |> Result.map_error (fun _ -> Dsl.Protocol_failure "Invalid local asset bytes") +;; + +let with_staged t (intent : Intent.t) callback = + let* handle = + locked t (fun () -> + match Cache.retain_staged t.cache ~file:intent.staged_file with + | Some handle -> Ok handle + | None -> Error (Dsl.Storage_failure "Original import staging is missing")) + in + let result = + try + match Cache.path t.cache handle with + | None -> Error (Dsl.Storage_failure "Original import staging is missing") + | Some path -> callback path + with + | exception_ -> + let cleanup = locked t (fun () -> storage (Cache.release_checked t.cache handle)) in + (match cleanup with + | Ok () -> raise exception_ + | Error error -> + Error + (Dsl.Cleanup_failure + { primary = Some (Dsl.Unexpected_failure (Printexc.to_string exception_)) + ; failures = Rrbvec.singleton error + })) + in + let cleanup = locked t (fun () -> storage (Cache.release_checked t.cache handle)) in + match cleanup, result with + | Ok (), result -> result + | Error error, result -> + Error + (Dsl.Cleanup_failure + { primary = + (match result with + | Error error -> Some error + | Ok _ -> None) + ; failures = Rrbvec.singleton error + }) +;; + +let publish_preview t ~check_current intent = + let* () = check t check_current in + let* () = protected t in + with_staged t intent (fun path -> + let* bytes = read_source t path in + let* verified = verified t intent.version bytes in + let* () = check t check_current in + locked t (fun () -> + let* handle = + storage + (Cache.publish + t.cache + ~asset:intent.asset + ~current:(fun () -> Result.is_ok (check t check_current)) + ~verified) + in + storage (Cache.release_checked t.cache handle))) +;; + +let stage t ~check_current (input : Logseq_db_types.Asset_import.t) = + let* () = check t check_current in + let* () = protected t in + locked t (fun () -> + let* () = check t check_current in + storage + (Cache.stage + t.cache + ~operation:input.operation + ~file_type:input.file_type + ~source_file:input.source_file + ~pending_budget_bytes:t.limits.asset_pending_budget_bytes)) +;; + +let put_staged t ~check_current ~(network : Execute.Core.worker_network) intent = + let* () = check t check_current in + let* () = protected t in + with_transfer t ~resource:false (fun () -> + with_staged t intent (fun path -> + let* bytes = read_source t path in + let* _ = verified t intent.version bytes in + let* key = t.key ~check_current:(fun () -> check t check_current) in + let* wire = + Codec.encode + ~maximum_plaintext_bytes:t.limits.asset_maximum_file_bytes + ~crypto:codec_crypto + ~key + bytes + |> Result.map_error (fun message -> Dsl.Crypto_failure message) + in + network.with_admission (fun () -> + let* () = check t check_current in + t.network.put + ~identity:t.identity + ~asset:intent.asset + ~version:intent.version + ~wire))) +;; + +let lookup t ~original asset = + let* original = original asset in + let staged = + match original with + | Some intent + when asset.Asset.current_checksum = Some intent.Intent.version.checksum + && intent.phase <> Intent.Cancelled -> + acquire_file t asset (fun () -> + Ok (Cache.retain_staged t.cache ~file:intent.staged_file)) + | Some _ | None -> Ok None + in + let* staged = staged in + match staged with + | Some _ -> Ok staged + | None -> + let rec find = function + | [] -> Ok None + | version :: rest -> + let* file = + acquire_file t asset (fun () -> + storage (Cache.lookup t.cache ~asset:asset.uuid ~version)) + in + (match file with + | Some _ -> Ok file + | None -> find rest) + in + find (Selection.lookup_versions asset) +;; + +let run + (type value) + t + ~check_current + ~admit_network + ~original + ~(upload : Dsl.asset -> (unit, Dsl.error) result) + ~identity + (operation : value Dsl.assets) + : (value, Dsl.error) result + = + let* () = same_owner t identity in + let precheck : type value. value Dsl.assets -> (unit, Dsl.error) result = function + | Dsl.Release_file _ | Release_lease _ -> Ok () + | Prune_cache | Reconcile_staging | Lookup _ | Retain _ | Upload _ | Download _ -> check t check_current + in + let* () = precheck operation in + let perform : type value. value Dsl.assets -> (value, Dsl.error) result = function + | Dsl.Release_file file -> release_file t ~identity file + | Release_lease lease -> release_lease t ~identity lease + | Dsl.Prune_cache -> locked t (fun () -> storage (Cache.prune t.cache)) + | Reconcile_staging -> Error (Dsl.Invalid_input "Staging reconciliation requires worker owner") + | Lookup asset -> + let* () = protected t in + locked t (fun () -> lookup t ~original asset) + | Retain file -> + let* () = protected t in + retain t file + | Upload asset -> + let* () = protected t in + upload asset + | Download asset -> + let* () = protected t in + (match Selection.download_version asset with + | None -> + Ok + (match asset.source with + | Asset.External _ -> Dsl.Missing_local + | Managed _ -> Waiting_remote) + | Some version -> + with_transfer t ~resource:true (fun () -> + let* key = t.key ~check_current:(fun () -> check t check_current) in + let maximum_wire_bytes = + match key with + | None -> t.limits.asset_maximum_file_bytes + | Some _ -> (4 * ((t.limits.asset_maximum_file_bytes + 18) / 3)) + 128 + in + let* () = admit_network () in + let* wire = t.network.get ~identity ~asset ~version ~maximum_wire_bytes in + let* () = check t check_current in + match wire with + | None -> Ok Dsl.Waiting_remote + | Some wire -> + if Option.is_some key && not (Codec.envelope_shape wire) + then + Error + (Dsl.Protocol_failure + "Encrypted asset requires an authenticated envelope") + else + let* verified = + Codec.decode + ~maximum_plaintext_bytes:t.limits.asset_maximum_file_bytes + ~crypto:codec_crypto + ~key + ~expected:version + wire + |> Result.map_error (fun _ -> + Dsl.Protocol_failure + "Asset authentication or checksum verification failed") + in + locked t (fun () -> + let* () = check t check_current in + let* file = + acquire_file ~reserved:true t asset (fun () -> + storage + (Cache.publish + t.cache + ~asset:asset.uuid + ~current:(fun () -> Result.is_ok (check t check_current)) + ~verified) + |> Result.map Option.some) + in + match file with + | Some file -> Ok (Dsl.Available file) + | None -> + Error (Dsl.Storage_failure "Asset publication returned no file")))) + in + perform operation +;; + +let dispatch t ~original ~upload ~scope ~identity operation = + let* result = + run + t + ~check_current:(fun () -> Execute.Core.check_current scope) + ~admit_network:(fun () -> Execute.Core.admit_network scope) + ~original + ~upload + ~identity + operation + in + let references + : type value. value Dsl.assets -> value -> Dsl.resource_reference Rrbvec.t + = + fun operation result -> + match operation with + | Dsl.Lookup _ -> + (match result with + | Some file -> Rrbvec.singleton (Dsl.File_reference file) + | None -> Rrbvec.empty) + | Download _ -> + (match result with + | Available file -> Rrbvec.singleton (Dsl.File_reference file) + | Waiting_remote | Missing_local -> Rrbvec.empty) + | Retain _ -> Rrbvec.singleton (Dsl.Lease_reference result) + | Prune_cache | Reconcile_staging | Upload _ | Release_file _ | Release_lease _ -> Rrbvec.empty + in + let refs = references operation result in + match Execute.Core.handoff scope refs with + | Ok () -> Ok result + | Error error -> + let failures = + Rrbvec.fold_left + (fun failures reference -> + let released = + match reference with + | Dsl.File_reference file -> release_file t ~identity file + | Lease_reference lease -> release_lease t ~identity lease + | Snapshot_reference _ | Connection_reference _ -> Ok () + in + match released with + | Ok () -> failures + | Error error -> Rrbvec.push_back failures error) + Rrbvec.empty + refs + in + if Rrbvec.length failures = 0 + then Error error + else Error (Dsl.Cleanup_failure { primary = Some error; failures }) +;; + +let fence t = t.fenced <- true + +let close t = + fence t; + locked t (fun () -> + if t.live_resources <> 0 || t.active_transfers <> 0 + then Error (Dsl.Capacity_exceeded "Asset consumer resources remain") + else storage (Cache.close_checked t.cache)) +;; diff --git a/logseq_sync/lib/assets/core.mli b/logseq_sync/lib/assets/core.mli new file mode 100644 index 00000000..c8600c8a --- /dev/null +++ b/logseq_sync/lib/assets/core.mli @@ -0,0 +1,32 @@ +module Dsl = Sync_dsl.Core +module Intent = Logseq_db_types.Asset_upload_intent +module Codec = Logseq_sync_native_crypto.Asset_codec + +type network = + { get : identity:Dsl.identity -> asset:Dsl.asset -> version:Logseq_db_types.Asset_descriptor.version -> maximum_wire_bytes:int -> (string option, Dsl.error) result + ; put : identity:Dsl.identity -> asset:Logseq_db_types.Graph_types.Uuid.t -> version:Logseq_db_types.Asset_descriptor.version -> wire:string -> (unit, Dsl.error) result + } + +type key = check_current:(unit -> (unit, Dsl.error) result) -> (string option, Dsl.error) result +val native_key : identity:Dsl.identity -> crypto:Logseq_sync_crypto.Core.t -> key + +type t +val create : root:string -> identity:Dsl.identity -> limits:Dsl.limits -> crypto:Logseq_sync_crypto.Core.t -> key:key -> network:network -> (t, Dsl.error) result +val identity : t -> Dsl.identity +val limits : t -> Dsl.limits +val cache : t -> Cache.t +val crypto : t -> Logseq_sync_crypto.Core.t + +(** Callbacks return only after their physical work has joined. Upload returns verified publication, not a PUT acknowledgement. *) +val run : t -> check_current:(unit -> (unit, Dsl.error) result) -> admit_network:(unit -> (unit, Dsl.error) result) -> original:(Dsl.asset -> (Intent.t option, Dsl.error) result) -> upload:(Dsl.asset -> (unit, Dsl.error) result) -> identity:Dsl.identity -> 'a Dsl.assets -> ('a, Dsl.error) result +val dispatch : t -> original:(Dsl.asset -> (Intent.t option, Dsl.error) result) -> upload:(Dsl.asset -> (unit, Dsl.error) result) -> scope:Execute.Core.effect_scope -> identity:Dsl.identity -> 'a Dsl.assets -> ('a, Dsl.error) result +val validate_resource : t -> Dsl.resource_reference -> (unit, Dsl.error) result +val release_file : t -> identity:Dsl.identity -> Dsl.asset_file -> (unit, Dsl.error) result +val release_lease : t -> identity:Dsl.identity -> Dsl.asset_lease -> (unit, Dsl.error) result +val file_path : t -> Dsl.asset_file -> (string, Dsl.error) result +val stage : t -> check_current:(unit -> (unit, Dsl.error) result) -> Logseq_db_types.Asset_import.t -> (Cache.staged, Dsl.error) result +val publish_preview : t -> check_current:(unit -> (unit, Dsl.error) result) -> Intent.t -> (unit, Dsl.error) result +val put_staged : t -> check_current:(unit -> (unit, Dsl.error) result) -> network:Execute.Core.worker_network -> Intent.t -> (unit, Dsl.error) result +val cache_error : Cache.error -> Dsl.error +val fence : t -> unit +val close : t -> (unit, Dsl.error) result diff --git a/logseq_sync/lib/assets/dune b/logseq_sync/lib/assets/dune new file mode 100644 index 00000000..d6d9ee54 --- /dev/null +++ b/logseq_sync/lib/assets/dune @@ -0,0 +1,6 @@ +(library + (name logseq_sync_assets) + (public_name logseq_sync.assets) + (modules namespace cache selection core) + (libraries eio execute sync_dsl logseq_sync.crypto logseq_sync.native_crypto logseq_db_types digestif mirage-crypto-rng.unix rrbvec unix uri yojson ppx_deriving_yojson.runtime) + (preprocess (pps ppx_deriving_yojson))) diff --git a/logseq_sync/lib/assets/namespace.ml b/logseq_sync/lib/assets/namespace.ml new file mode 100644 index 00000000..ac97c428 --- /dev/null +++ b/logseq_sync/lib/assets/namespace.ml @@ -0,0 +1,15 @@ +module Uuid = Logseq_db_types.Graph_types.Uuid + +type account_scope = { managed_sync_origin : Uri.t; user_id : string } +type graph_scope = { account : account_scope; graph_id : Uuid.t } +type retirement_id = Logseq_db_types.Graph_types.Retirement_id.t +type asset_deletion = { scope : graph_scope; retirement : retirement_id } +let retirement_id_to_string = Logseq_db_types.Graph_types.Retirement_id.to_string +let retirement_id_of_string = Logseq_db_types.Graph_types.Retirement_id.of_string +let asset_deletion_scope deletion = deletion.scope +let asset_deletion_retirement deletion = deletion.retirement +let asset_deletion_reference ~scope ~retirement = { scope; retirement } +let of_identity (identity : Sync_dsl.Core.identity) = + Result.map (fun graph_id -> + { account = { managed_sync_origin = Uri.of_string identity.origin; user_id = identity.account }; graph_id }) + (Uuid.of_string identity.graph) diff --git a/logseq_sync/lib/assets/namespace.mli b/logseq_sync/lib/assets/namespace.mli new file mode 100644 index 00000000..00622b8d --- /dev/null +++ b/logseq_sync/lib/assets/namespace.mli @@ -0,0 +1,12 @@ +module Uuid = Logseq_db_types.Graph_types.Uuid + +type account_scope = { managed_sync_origin : Uri.t; user_id : string } +type graph_scope = { account : account_scope; graph_id : Uuid.t } +type retirement_id = Logseq_db_types.Graph_types.Retirement_id.t +type asset_deletion +val retirement_id_to_string : retirement_id -> string +val retirement_id_of_string : string -> (retirement_id, string) result +val asset_deletion_scope : asset_deletion -> graph_scope +val asset_deletion_retirement : asset_deletion -> retirement_id +val asset_deletion_reference : scope:graph_scope -> retirement:retirement_id -> asset_deletion +val of_identity : Sync_dsl.Core.identity -> (graph_scope, string) result diff --git a/logseq_sync/lib/assets/selection.ml b/logseq_sync/lib/assets/selection.ml new file mode 100644 index 00000000..a8a43c7c --- /dev/null +++ b/logseq_sync/lib/assets/selection.ml @@ -0,0 +1,14 @@ +module Asset = Logseq_db_types.Asset_descriptor +let download_version (asset : Asset.t) = + match asset.source with Managed version -> version | External _ -> None +let lookup_versions (asset : Asset.t) = + match asset.source with + | External _ | Managed None -> [] + | Managed (Some remote) -> + match asset.current_checksum with + | None -> [remote] + | Some checksum when checksum = remote.checksum -> [remote] + | Some checksum -> + match Asset.version ~checksum ~file_type:remote.file_type with + | Ok current -> [current; remote] + | Error _ -> [remote] diff --git a/logseq_sync/lib/assets/selection.mli b/logseq_sync/lib/assets/selection.mli new file mode 100644 index 00000000..075a2f1f --- /dev/null +++ b/logseq_sync/lib/assets/selection.mli @@ -0,0 +1,2 @@ +val lookup_versions : Logseq_db_types.Asset_descriptor.t -> Logseq_db_types.Asset_descriptor.version list +val download_version : Logseq_db_types.Asset_descriptor.t -> Logseq_db_types.Asset_descriptor.version option diff --git a/logseq_sync/lib/crypto/core.ml b/logseq_sync/lib/crypto/core.ml new file mode 100644 index 00000000..fb574cbf --- /dev/null +++ b/logseq_sync/lib/crypto/core.ml @@ -0,0 +1,254 @@ +module Dsl = Sync_dsl.Core +module Native = Logseq_sync_native_crypto.Platform_crypto + +type backend = + { crypto : Native.crypto + ; unlock : + identity:Dsl.identity -> wrapped_graph_key:string -> (string, Dsl.error) result + } + +let apple_backend = + { crypto = Native.crypto + ; unlock = + (fun ~identity ~wrapped_graph_key -> + Native.unlock_graph_key + ~managed_sync_origin:(Uri.of_string identity.Dsl.origin) + ~user_id:identity.account + ~encrypted_graph_key:wrapped_graph_key + |> Result.map_error (fun message -> Dsl.Crypto_failure message)) + } +;; + +type t = + { identity : Dsl.identity + ; limits : Dsl.limits + ; encrypted : bool + ; backend : backend + ; mutable key : bytes option + ; mutable retired : bool + ; lock : Eio.Mutex.t + } + +let create ~identity ~limits ~encrypted ~backend = + { identity + ; limits + ; encrypted + ; backend + ; key = None + ; retired = false + ; lock = Eio.Mutex.create () + } +;; + +let status t = + if t.retired || (t.encrypted && Option.is_none t.key) + then Dsl.Key_locked + else if t.encrypted + then Dsl.Key_available + else Dsl.Unencrypted +;; + +let retire t = + t.retired <- true; + Option.iter (fun key -> Bytes.fill key 0 (Bytes.length key) '\000') t.key; + t.key <- None +;; + +let ( let* ) = Result.bind + +let asset_key t ~check_current ~identity = + let current () = + let* () = check_current () in + if t.retired + then Error Dsl.Stale_context + else if identity <> t.identity + then Error (Dsl.Identity_mismatch "Crypto owner") + else Ok () + in + let* () = current () in + Eio.Mutex.use_rw ~protect:false t.lock (fun () -> + let* () = current () in + match t.encrypted, t.key with + | false, _ -> Ok None + | true, None -> Error Dsl.Locked + | true, Some key -> Ok (Some (Bytes.to_string key))) +;; + +let run (type a) t ~check_current ~identity (operation : a Dsl.crypto) + : (a, Dsl.error) result + = + let current () = + let* () = check_current () in + if t.retired + then Error Dsl.Stale_context + else if identity <> t.identity + then Error (Dsl.Identity_mismatch "Crypto owner") + else Ok () + in + let* () = current () in + Eio.Mutex.use_rw ~protect:false t.lock (fun () : (a, Dsl.error) result -> + let* () = current () in + let* () = + match operation with + | Dsl.Unlock _ -> Ok () + | Dsl.Encrypt_values _ | Dsl.Decrypt_value _ | Dsl.Protect _ | Dsl.Unprotect _ -> + if t.encrypted && Option.is_none t.key then Error Dsl.Locked else Ok () + in + let bounded_lengths values = + if List.length values > t.limits.maximum_items + then Error (Dsl.Capacity_exceeded "Crypto items") + else + List.fold_left + (fun total value -> + let* total = total in + if value > t.limits.maximum_bytes - total + then Error (Dsl.Capacity_exceeded "Crypto bytes") + else Ok (total + value)) + (Ok 0) + values + |> Result.map (fun _ -> ()) + in + let bounded values = bounded_lengths (List.map String.length values) in + let native callback = + let* () = current () in + let result = callback () in + let* () = current () in + Result.map_error (fun message -> Dsl.Crypto_failure message) result + in + let graph_key () = + match t.key with + | Some key -> Ok (Bytes.to_string key) + | None -> Error Dsl.Locked + in + let decrypt source = + let* () = bounded [ source ] in + let* value = + if t.encrypted + then + let* key = graph_key () in + native (fun () -> + Logseq_sync_native_crypto.E2ee.decrypt_value + ~crypto:{ decrypt_aes_gcm = t.backend.crypto.decrypt_aes_gcm } + ~graph_key:key + source) + else ( + try Ok (Transit_native.Transit.Json.of_string source) with + | Transit_core.Json.Decode_error _ + | Yojson.Json_error _ + | Failure _ + | Invalid_argument _ -> Ok (Transit_core.Json.String source)) + in + match value with + | Transit_core.Json.String plaintext -> + let* () = bounded [ plaintext ] in + Ok plaintext + | _ -> Error (Dsl.Crypto_failure "Protected value must be a string") + in + let encrypt plaintext = + let* key = graph_key () in + let* iv, ciphertext = + native (fun () -> t.backend.crypto.encrypt_aes_gcm ~key ~plaintext) + in + if + String.length iv <> 12 || String.length ciphertext <> String.length plaintext + 16 + then Error (Dsl.Crypto_failure "Invalid AES-GCM output") + else Ok (iv, ciphertext) + in + let rec traverse callback acc = function + | [] -> Ok (List.rev acc) + | value :: rest -> + let* value = callback value in + traverse callback (value :: acc) rest + in + let perform : type value. value Dsl.crypto -> (value, Dsl.error) result = function + | Dsl.Unlock input -> + if not t.encrypted + then Ok Dsl.Unencrypted + else ( + let result = + t.backend.unlock ~identity ~wrapped_graph_key:(Dsl.wrapped_graph_key input) + in + let* key = + match result with + | Ok key -> Ok key + | Error error -> + let* () = current () in + Error error + in + let bytes = Bytes.of_string key in + let installed = ref false in + Fun.protect + ~finally:(fun () -> + if not !installed then Bytes.fill bytes 0 (Bytes.length bytes) '\000') + (fun () -> + let* () = current () in + if Bytes.length bytes <> 32 + then Error (Dsl.Crypto_failure "Invalid graph key length") + else ( + Option.iter + (fun previous -> Bytes.fill previous 0 (Bytes.length previous) '\000') + t.key; + t.key <- Some bytes; + installed := true; + Ok Dsl.Key_available))) + | Dsl.Encrypt_values values -> + if not t.encrypted + then Error (Dsl.Crypto_failure "Encryption requires a graph key") + else ( + let values = Rrbvec.to_list values in + let* () = bounded values in + let* values = traverse encrypt [] values in + let* () = + bounded_lengths + (List.map + (fun (iv, ciphertext) -> String.length iv + String.length ciphertext) + values) + in + Ok (Rrbvec.of_list values)) + | Dsl.Decrypt_value source -> decrypt source + | Dsl.Protect request -> + let items = Logseq_overlay_db.Database.protection_plaintexts request in + let* () = bounded (List.map snd items) in + if not t.encrypted + then + traverse + (fun (id, source) -> + let* plaintext = decrypt source in + Ok (id, plaintext)) + [] + items + else + let* encrypted = + traverse + (fun (id, plaintext) -> + let* iv, ciphertext = encrypt plaintext in + let wire = + Transit_native.Transit.Json.to_string + (Transit_core.Json.Array [ Binary iv; Binary ciphertext ]) + in + Ok (id, wire)) + [] + items + in + let* () = bounded (List.map snd encrypted) in + Ok encrypted + | Dsl.Unprotect request -> + let items = Logseq_overlay_db.Database.unprotection_ciphertexts request in + let* () = bounded (List.map snd items) in + let* plaintexts = + traverse + (fun (id, source) -> + let* plaintext = decrypt source in + Ok (id, plaintext)) + [] + items + in + let* () = bounded (List.map snd plaintexts) in + Ok plaintexts + in + perform operation) +;; + +let dispatch t ~scope ~identity operation = + run t ~check_current:(fun () -> Execute.Core.check_current scope) ~identity operation +;; diff --git a/logseq_sync/lib/crypto/core.mli b/logseq_sync/lib/crypto/core.mli new file mode 100644 index 00000000..83899ed9 --- /dev/null +++ b/logseq_sync/lib/crypto/core.mli @@ -0,0 +1,44 @@ +module Dsl = Sync_dsl.Core +module Native = Logseq_sync_native_crypto.Platform_crypto + +type backend = + { crypto : Native.crypto + ; unlock : + identity:Dsl.identity -> wrapped_graph_key:string -> (string, Dsl.error) result + } + +val apple_backend : backend + +type t + +val create + : identity:Dsl.identity + -> limits:Dsl.limits + -> encrypted:bool + -> backend:backend + -> t + +val status : t -> Dsl.key_status +val retire : t -> unit + +(** Ephemeral asset-codec key from this owner's installed generation; never reload native storage. *) +val asset_key + : t + -> check_current:(unit -> (unit, Dsl.error) result) + -> identity:Dsl.identity + -> (string option, Dsl.error) result + +(** The caller's scope is checked before and after each native operation. Retirement erases the owned key. *) +val run + : t + -> check_current:(unit -> (unit, Dsl.error) result) + -> identity:Dsl.identity + -> 'a Dsl.crypto + -> ('a, Dsl.error) result + +val dispatch + : t + -> scope:Execute.Core.effect_scope + -> identity:Dsl.identity + -> 'a Dsl.crypto + -> ('a, Dsl.error) result diff --git a/logseq_sync/lib/crypto/dune b/logseq_sync/lib/crypto/dune new file mode 100644 index 00000000..c2bcb8e4 --- /dev/null +++ b/logseq_sync/lib/crypto/dune @@ -0,0 +1,5 @@ +(library + (name logseq_sync_crypto) + (public_name logseq_sync.crypto) + (modules core) + (libraries eio execute sync_dsl logseq_sync.native_crypto logseq_overlay_db logseq_db_types rrbvec melange-transit-core melange-transit-native uri yojson)) diff --git a/logseq_sync/lib/domain/core.ml b/logseq_sync/lib/domain/core.ml new file mode 100644 index 00000000..954c08e2 --- /dev/null +++ b/logseq_sync/lib/domain/core.ml @@ -0,0 +1,900 @@ +module Overlay = Logseq_overlay_db.Types +module Server_cursor = Overlay.Server_cursor + +type graph_id = Logseq_db_types.Sync_account.graph_id [@@deriving yojson] + +type cache_schema = Logseq_db_types.Sync_account.cache_schema = + { major : int + ; minor : int + ; exact : bool + } +[@@deriving yojson] + +type graph = Logseq_db_types.Sync_account.graph = + { graph_id : graph_id + ; name : string + ; schema : cache_schema + ; encrypted : bool + } +[@@deriving yojson] + +type account_generation = Logseq_db_types.Sync_account.account_generation +type graph_generation = Logseq_db_types.Sync_account.graph_generation +type connection_generation = Logseq_db_types.Sync_account.connection_generation +type presentation_generation = Logseq_db_types.Sync_account.presentation_generation +type lifecycle_generation = Logseq_db_types.Sync_account.lifecycle_generation + +type account_scope = Logseq_db_types.Sync_account.account_scope = + { managed_sync_origin : Uri.t + ; user_id : string + ; account_generation : account_generation + ; presentation_generation : presentation_generation + ; lifecycle_generation : lifecycle_generation + } + +type graph_scope = Logseq_db_types.Sync_account.graph_scope = + { account : account_scope + ; graph_id : graph_id + ; graph_generation : graph_generation + } + +(** Execution admission gate owned by a pending host handoff. It grants no + deletion authority. Release is exact-ID and cannot unfreeze a later round. *) +module Host_cleanup_scopes=Set.Make(struct type t=graph_scope let compare=Stdlib.compare end) +type host_fence_target=One_graph of graph_id|All_graphs +type host_fence={id:Logseq_db_types.Graph_types.Uuid.t;owner:account_scope; + target:host_fence_target;captured_scopes:graph_scope Rrbvec.t;previous:Logseq_db_types.Graph_types.Uuid.t option} + +type connection_scope = + { graph : graph_scope + ; connection_generation : connection_generation + } + +type sync_phase = Logseq_db_types.Sync_account.sync_phase = + | Offline + | Connecting + | Pulling + | Submitting + | Current + | Paused + | Failed + +type startup_failure_stage = Logseq_db_types.Sync_account.startup_failure_stage = + | During_authentication + | During_catalog + | During_local_restore + | During_bootstrap + | During_e2ee + +type local_unavailability = Logseq_db_types.Sync_account.local_unavailability = + | No_local_account + | Local_catalog_unavailable + | Local_mirror_unavailable + | Local_key_unavailable + | Local_restore_failed + +type local_access = Logseq_db_types.Sync_account.local_access = + | Local_pending + | Local_available + | Local_unavailable of local_unavailability + +type startup_network_phase = Logseq_db_types.Sync_account.startup_network_phase = + | Local_only + | Online_after_timeline + | Online_by_user + +type startup_facts = Logseq_db_types.Sync_account.startup_facts = + { authenticated : bool + ; local_access : local_access + ; network_phase : startup_network_phase + ; catalog_loading : bool + ; awaiting_selection : bool + ; restoring_local : bool + ; bootstrapping : bool + ; awaiting_e2ee_password : bool + ; failure : startup_failure_stage option + ; account_generation : account_generation + ; graph_generation : graph_generation + ; presentation_generation : presentation_generation + } + +type local_deletion_stage = Logseq_db_types.Sync_account.local_deletion_stage = + | Closing_graph + | Deleting_mirror + | Clearing_selection + +type local_deletion = Logseq_db_types.Sync_account.local_deletion = + | Deletion_in_progress of local_deletion_stage + | Deletion_failed of local_deletion_stage + +type unlock_challenge = Logseq_db_types.Sync_account.unlock_challenge = + { challenge_scope : graph_scope + ; challenge_id : int + ; encrypted_graph_key : string + ; private_key_package : string + } + +type unlock_stage = Logseq_db_types.Sync_account.unlock_stage = Private_key of int | Graph_key of int | Attachment + +type unlock_submission = Logseq_db_types.Sync_account.unlock_submission = + { challenge : unlock_challenge + ; operation_id : int + ; stage : unlock_stage + } + +type unlock_attached = Logseq_db_types.Sync_account.unlock_attached = { submission : unlock_submission; attached_scope : graph_scope } + +type unlock_recovery = Logseq_db_types.Sync_account.unlock_recovery = + | Retry_password of unlock_challenge * string + | Retry_preparation of string + +type unlock_phase = Logseq_db_types.Sync_account.unlock_phase = + | Unlock_inactive + | Unlock_challenge of unlock_challenge + | Unlock_running of unlock_submission + | Unlock_bootstrapping of unlock_submission + | Unlock_presentable of unlock_attached + | Unlock_failed of unlock_recovery + | Unlock_cancelled of unlock_submission + +let unlock_challenge = function + | Unlock_challenge challenge | Unlock_failed (Retry_password (challenge, _)) -> Some challenge + | _ -> None +;; + +let equal_unlock_challenge a b = + a.challenge_id = b.challenge_id && a.challenge_scope = b.challenge_scope +let unlock_challenge_scope challenge = challenge.challenge_scope +let unlock_submission_challenge submission = submission.challenge +let equal_unlock_submission a b = + a.operation_id = b.operation_id && equal_unlock_challenge a.challenge b.challenge +;; +let unlock_attached_submission receipt = receipt.submission +let unlock_attached_scope receipt = receipt.attached_scope + +(** Presentation only: retry eligibility is still determined by the original outcome. + Unknown and security failures require attention even when transport retries them. *) +type error_attention = Logseq_db_types.Sync_account.error_attention = Transient_connection | Requires_attention + +type snapshot = Logseq_db_types.Sync_account.snapshot = + { sync_phase : sync_phase + ; catalog : graph list + ; selected_graph : graph_id option + ; resident_graphs : graph_id list + ; applied_server_t : Server_cursor.t option + ; timeline_presentation_pending : bool + ; unlock : unlock_phase + ; startup : startup_facts + ; last_error : string option + ; last_error_attention : error_attention option + ; local_deletion : local_deletion option + } + +type diagnostic_group = Logseq_db_types.Sync_account.diagnostic_group = + { title : string + ; entries : (string * string) list + } + +type diagnostics = Logseq_db_types.Sync_account.diagnostics = { groups : diagnostic_group list } + +type state = Logseq_db_types.Sync_account.state = + { snapshot : snapshot + ; diagnostics : diagnostics + } + +type limits = + { maximum_response_bytes : int + ; maximum_artifact_bytes : int + ; submission_batch_size : int + } +[@@warning "-69"] + +type config = + { config_origin : Uri.t + ; limits : limits + } + +type config_error = Invalid_config of string + +let limits ~maximum_response_bytes ~maximum_artifact_bytes ~submission_batch_size = + if + maximum_response_bytes <= 0 + || maximum_response_bytes > Logseq_db_types.Limits.maximum_response_bytes + then Error (Invalid_config "maximum response bytes are outside the supported bound") + else if maximum_artifact_bytes <= 0 + then Error (Invalid_config "maximum artifact bytes must be positive") + else if submission_batch_size <= 0 || submission_batch_size > 4096 + then Error (Invalid_config "submission batch size is outside the supported bound") + else Ok { maximum_response_bytes; maximum_artifact_bytes; submission_batch_size } +;; + +let config ~managed_sync_origin ~limits = + match Uri.scheme managed_sync_origin, Uri.host managed_sync_origin with + | Some "https", Some host when String.length host > 0 -> + Ok { config_origin=managed_sync_origin; limits } + | _ -> Error (Invalid_config "managed sync origin must be an absolute HTTPS URI") +;; + +type bootstrap_progress = + { graph_id : graph_id + ; received_bytes : int64 + ; total_bytes : int64 option + } + +type invalidation = + { basis : int64 + ; changed_uuids : graph_id list + ; changed_uuids_truncated : bool + } + +type effect_scope = + { account_generation : account_generation option + ; graph_generation : graph_generation option + ; connection_generation : connection_generation option + ; presentation_generation : presentation_generation option + ; lifecycle_generation : lifecycle_generation option + } + +type effect_id = int + +type crypto_failure_kind = + | Invalid_key_material + | Crypto_provider_unavailable + +type effect_error = + | Effect_failed of string + | Crypto_failed of crypto_failure_kind * string + +type graph_key_handle = + { handle_id : string + ; handle_scope : graph_scope + } + +type snapshot_download = + { scope : graph_scope + ; uri : Uri.t + ; expected_bytes : int64 option + ; maximum_bytes : int + ; metadata_cursor : Server_cursor.t + ; attempt_id : effect_id + ; download_id : effect_id + } + +type staged_artifact = + { download : snapshot_download + ; artifact_path : string + ; artifact_expected_rows : int + } + +type catalog_cache = Logseq_db_types.Sync_account.catalog_cache = + { cache_user_id : string + ; cache_graphs : graph list + ; cache_selected_graph : graph_id option + ; cache_resident_graphs : graph_id list option [@default None] + } +[@@deriving yojson] + +let graph_key_handle ~id ~scope = { handle_id = id; handle_scope = scope } +let graph_key_handle_id handle = handle.handle_id +let graph_key_handle_scope handle = handle.handle_scope + +let stage_snapshot_download download ~path ~expected_rows = + if path = "" || expected_rows < 0 then Error "Invalid staged snapshot artifact" + else Ok { download; artifact_path = path; artifact_expected_rows = expected_rows } +;; + +let snapshot_download_scope request = request.scope +let snapshot_download_uri request = request.uri +let snapshot_download_expected_bytes request = request.expected_bytes +let snapshot_download_maximum_bytes request = request.maximum_bytes + +let staged_artifact_path artifact = artifact.artifact_path +let staged_artifact_expected_rows artifact = artifact.artifact_expected_rows + +let effect_scope_of_account (account : account_scope) = + { account_generation = Some account.account_generation + ; graph_generation = None + ; connection_generation = None + ; presentation_generation = Some account.presentation_generation + ; lifecycle_generation = Some account.lifecycle_generation + } +;; + +let effect_scope_of_graph (graph : graph_scope) = + { (effect_scope_of_account graph.account) with + graph_generation = Some graph.graph_generation + } +;; + +let effect_scope_of_connection (connection : connection_scope) = + { (effect_scope_of_graph connection.graph) with + connection_generation = Some connection.connection_generation + } +;; + +let catalog_cache ~user_id ~graphs ~selected_graph = + { cache_user_id = user_id + ; cache_graphs = graphs + ; cache_selected_graph = selected_graph + ; cache_resident_graphs = None + } +;; + +let catalog_cache_user_id cache = cache.cache_user_id +let catalog_cache_graphs cache = cache.cache_graphs +let catalog_cache_selected_graph cache = cache.cache_selected_graph +let catalog_cache_resident_graphs cache = + Option.value cache.cache_resident_graphs ~default:(Option.to_list cache.cache_selected_graph) +let encode_catalog_cache cache = catalog_cache_to_yojson cache |> Yojson.Safe.to_string + +let decode_catalog_cache source = + let bounded_text value = + String.length value > 0 + && String.length value <= 512 + && String.is_valid_utf_8 value + && not (String.contains value '\000') + in + try + match + Logseq_db_types.Json_codec.decode + catalog_cache_of_yojson + (Yojson.Safe.from_string source) + with + | Error message -> + let fields = List.concat_map (fun (prefix, names) -> + List.map (fun name -> prefix ^ name, + "field " ^ name ^ " is missing or has an invalid type") names) + [ "Core.catalog_cache.", ["cache_user_id"; "cache_graphs"; "cache_selected_graph"; "cache_resident_graphs"] + ; "Core.graph.", ["graph_id"; "name"; "schema"; "encrypted"] + ; "Core.cache_schema.", ["major"; "minor"; "exact"] ] in + Error ("catalog cache: " ^ Logseq_db_types.Json_codec.error_context ~fields message) + | Ok cache -> + if + (not (bounded_text cache.cache_user_id)) + || (match cache.cache_resident_graphs with + | None -> false + | Some ids -> List.length ids > 3 + || List.length (List.sort_uniq compare ids) <> List.length ids + || List.exists (fun id -> not (List.exists (fun (graph:graph) -> graph.graph_id=id) cache.cache_graphs)) ids + || (match cache.cache_selected_graph with + | Some id -> not (List.mem id ids) + | None -> ids <> [])) + || List.exists + (fun (graph : graph) -> + (not (bounded_text graph.name)) + || graph.schema.major < 0 + || graph.schema.minor < 0) + cache.cache_graphs + then Error "catalog cache contains invalid domain values" + else Ok cache + with + | Yojson.Json_error _ -> Error "catalog cache: invalid JSON syntax" + | Failure _ | Invalid_argument _ -> Error "catalog cache: invalid shape" +;; + +type encryption_batch = + { scope : graph_scope + ; key : graph_key_handle + ; plaintexts : string list + } + +type encrypted_values = (string * string) list + +type decryption_batch = + { scope : graph_scope + ; key : graph_key_handle + ; protected_values : (string * string) list + } + +type decrypted_values = string list +type snapshot_baseline = string +type snapshot_metadata = string + +type graph_key_request = + { scope : graph_scope + ; encrypted_graph_key : string + } + +type private_key_unlock = + { scope : account_scope + ; password : string + ; private_key_package : string + } + +type retirement_id = Logseq_db_types.Graph_types.Retirement_id.t +let retirement_id_of_string = Logseq_db_types.Graph_types.Retirement_id.of_string +let retirement_id_to_string = Logseq_db_types.Graph_types.Retirement_id.to_string + +type asset_deletion = { deletion_scope : graph_scope; retirement : retirement_id } +let asset_deletion_reference ~scope ~retirement = { deletion_scope = scope; retirement } +let asset_deletion_scope deletion = deletion.deletion_scope +let asset_deletion_retirement deletion = deletion.retirement + +type asset_context = + { scope : graph_scope + ; encrypted : bool + ; key : graph_key_handle option + } + +type asset_action = + | Check_asset_cache of graph_id * Logseq_db_types.Asset_descriptor.version + | Fetch_asset of + { asset : graph_id + ; version : Logseq_db_types.Asset_descriptor.version + ; maximum_plaintext_bytes : int + } + | Stage_asset_file of + { operation : graph_id + ; file_type : string + ; source_file : string + } + | Put_asset_file of + { asset : graph_id + ; version : Logseq_db_types.Asset_descriptor.version + ; file : string + ; maximum_plaintext_bytes : int + } + | Publish_staged_asset of + { asset : Logseq_db_types.Graph_types.Uuid.t + ; version : Logseq_db_types.Asset_descriptor.version + ; file : string + } + | Retain_asset_file of string + | Retain_staged_file of string + | Release_asset_file of string + | Release_staged_file of string + | Prune_asset_staging of string list + (** Join already admitted actual asset I/O tasks and their callback posting. + This read-only barrier neither closes caches nor grants a host permit. *) + | Await_asset_operations + | Close_asset_scope + | Prepare_graph_asset_deletion of retirement_id + | Commit_graph_asset_deletion of asset_deletion + | Cancel_graph_asset_deletion of asset_deletion + | Delete_account_assets + | Asset_retry_after of float + | Cancel_asset_operation of string + +type asset_failure = + | Asset_network + | Asset_not_found + | Asset_checksum_mismatch + | Asset_authentication + | Asset_locked + | Asset_storage_full + | Asset_missing_source + | Asset_size_rejected + | Asset_revoked_access + | Asset_cancelled + | Asset_invalid_content of string + | Asset_with_cause of asset_failure * Logseq_db_types.Failure_diagnostic.t + +let rec asset_failure_kind = function + | Asset_with_cause (failure, _) -> asset_failure_kind failure + | failure -> failure +;; + +let asset_failure_cause = function + | Asset_with_cause (_, cause) -> Some cause + | _ -> None +;; + +let asset_failure_with_cause failure cause = + Asset_with_cause (asset_failure_kind failure, cause) +;; + +type asset_value = + | Asset_unit + | Asset_cached of string option + | Asset_downloaded of string + | Asset_staged of + { file : string + ; checksum : string + ; size : int64 + } + | Asset_retained of (string * string) option + | Asset_deletion_prepared of asset_deletion + | Asset_pruned of int + | Asset_retry_elapsed + +type asset_result = (asset_value, asset_failure) result + +type asset_request = + { scope : graph_scope + ; operation : string + ; action : asset_action + } + +type asset_io_request = + { context : asset_context + ; operation : string + ; action : asset_action + } + +type asset_ticket = + { asset_serial : int + ; asset_request : asset_io_request + } + +let asset_ticket_id ticket = "asset:" ^ string_of_int ticket.asset_serial + +type asset_output = + { request : asset_request + ; result : asset_result + } + +type protected_action = + | Encrypt_values of graph_key_handle * string list + | Decrypt_value of graph_key_handle * string + +type protected_request = + { protected_operation : string + ; protected_scope : graph_scope + ; protected_action : protected_action + } + +type protected_value = + | Encrypted_values of (string * string) list + | Decrypted_value of string + +type protected_result = (protected_value, effect_error) result + +type protected_output = + { protected_request : protected_request + ; protected_result : protected_result + } + +type protected_ticket = + { protected_serial : int + ; protected_input : protected_request + } + +let protected_ticket_id ticket = "protected:" ^ string_of_int ticket.protected_serial + +type _ runner_request = + | Load_catalog : account_scope -> catalog_cache option runner_request + | Save_catalog : + { account : account_scope + ; cache : catalog_cache + } + -> unit runner_request + | Fetch_catalog : account_scope -> graph list runner_request + | Fetch_snapshot_baseline : graph_scope -> snapshot_baseline runner_request + | Fetch_snapshot_metadata : graph_scope -> snapshot_metadata runner_request + | Download_snapshot : snapshot_download -> staged_artifact runner_request + | Fetch_e2ee_graph_key : graph_scope -> string runner_request + | Fetch_e2ee_user_keys : account_scope -> string runner_request + | Load_and_unlock_graph_key : graph_scope -> graph_key_handle runner_request + | Fetch_and_unlock_graph_key : graph_key_request -> graph_key_handle runner_request + | Unlock_private_key : private_key_unlock -> unit runner_request + | Delete_account_secrets : account_scope -> unit runner_request + | Encrypt_protected_values : encryption_batch -> encrypted_values runner_request + | Decrypt_protected_values : decryption_batch -> decrypted_values runner_request + +type _ request_kind = + | Load_catalog_kind : catalog_cache option request_kind + | Save_catalog_kind : unit request_kind + | Fetch_catalog_kind : graph list request_kind + | Fetch_snapshot_baseline_kind : snapshot_baseline request_kind + | Fetch_snapshot_metadata_kind : snapshot_metadata request_kind + | Download_snapshot_kind : staged_artifact request_kind + | Fetch_e2ee_graph_key_kind : string request_kind + | Fetch_e2ee_user_keys_kind : string request_kind + | Load_and_unlock_graph_key_kind : graph_key_handle request_kind + | Fetch_and_unlock_graph_key_kind : graph_key_handle request_kind + | Unlock_private_key_kind : unit request_kind + | Delete_account_secrets_kind : unit request_kind + | Encrypt_protected_values_kind : encrypted_values request_kind + | Decrypt_protected_values_kind : decrypted_values request_kind + +type 'a effect_ticket = + { id : effect_id + ; scope : effect_scope + ; kind : 'a request_kind + } + +let effect_ticket_id ticket = ticket.id +let effect_ticket_scope ticket = ticket.scope +let effect_id_to_string = string_of_int + +type websocket_request = + { scope : connection_scope + ; uri : Uri.t + } + +type websocket_send = + { scope : connection_scope + ; message : Logseq_sync_protocol.Sync_protocol.Client.message + } + +type websocket_termination = + | Recoverable_close of { message : string option; attention : error_attention } + | Local_close + | Authentication_failure of string + | Protocol_failure of string + +type timer_id = int + +type timer_request = + { id : timer_id + ; scope : effect_scope + ; delay_seconds : float + } + +type runner_effect = + (* Snapshot actual I/O owners before fork; wait off the serial consumer and + report after their physical task and callback posting finish. No host, + filesystem or data-admission permission is granted. *) + | Await_io of int64 * effect_scope + | Asset_io of asset_ticket * asset_io_request + | Protected_io of protected_ticket * protected_request + | Request : 'a effect_ticket * 'a runner_request -> runner_effect + | Start_websocket of websocket_request + | Send_websocket of websocket_send + | Close_websocket of connection_scope + | Schedule_timer of timer_request + | Cancel_effects of effect_scope + | Cancel_network_effects of effect_scope + +type runner_completion = + | Asset_completion of asset_ticket * asset_result + | Protected_completion of protected_ticket * protected_result + | Completion : 'a effect_ticket * ('a, effect_error) result -> runner_completion + +let scope_of_request : type a. a runner_request -> effect_scope = function + | Load_catalog account | Save_catalog { account; _ } -> effect_scope_of_account account + | Fetch_catalog value -> effect_scope_of_account value + | Fetch_snapshot_baseline value | Fetch_snapshot_metadata value -> + effect_scope_of_graph value + | Download_snapshot value -> effect_scope_of_graph value.scope + | Fetch_e2ee_graph_key value -> effect_scope_of_graph value + | Fetch_e2ee_user_keys value -> effect_scope_of_account value + | Load_and_unlock_graph_key value -> effect_scope_of_graph value + | Fetch_and_unlock_graph_key value -> effect_scope_of_graph value.scope + | Unlock_private_key value -> effect_scope_of_account value.scope + | Delete_account_secrets value -> effect_scope_of_account value + | Encrypt_protected_values value -> effect_scope_of_graph value.scope + | Decrypt_protected_values value -> effect_scope_of_graph value.scope +;; + +let runner_effect_scope = function + | Await_io(_,scope)->scope + | Protected_io (_, request) -> effect_scope_of_graph request.protected_scope + | Asset_io (_, request) -> effect_scope_of_graph request.context.scope + | Request (ticket, _) -> ticket.scope + | Start_websocket value -> effect_scope_of_connection value.scope + | Send_websocket value -> effect_scope_of_connection value.scope + | Close_websocket value -> effect_scope_of_connection value + | Schedule_timer value -> value.scope + | Cancel_effects value | Cancel_network_effects value -> value +;; + +let runner_effect_diagnostic = function + | Await_io(id,_)->Printf.sprintf "io-fence:%Ld" id + | Protected_io (ticket, _) -> protected_ticket_id ticket + | Asset_io (ticket, _) -> asset_ticket_id ticket + | Request (ticket, _) -> Printf.sprintf "request:%d" ticket.id + | Start_websocket _ -> "start-websocket" + | Send_websocket _ -> "send-websocket" + | Close_websocket _ -> "close-websocket" + | Schedule_timer value -> Printf.sprintf "timer:%d" value.id + | Cancel_effects _ -> "cancel-effects" + | Cancel_network_effects _ -> "cancel-network-effects" +;; + +let equal_runner_effect left right = left = right + +type mirror_request = + { graph : graph + ; scope : graph_scope + } + +type snapshot_activation_request = + { artifact : staged_artifact + ; scope : graph_scope + ; applied_server_t : Server_cursor.t + ; key : graph_key_handle option + } + +let activate_staged_snapshot (artifact : staged_artifact) ~key = + let scope = artifact.download.scope in + match key with + | Some key when graph_key_handle_scope key <> scope -> + Error "graph key handle is outside the staged snapshot scope" + | None | Some _ -> + Ok {artifact;scope;applied_server_t=artifact.download.metadata_cursor;key} +;; + +let snapshot_activation_scope request = request.scope +let snapshot_activation_cursor request = request.applied_server_t +let snapshot_activation_artifact request = request.artifact +let snapshot_activation_key request = request.key + +(* Each phase owns all evidence needed by the next phase. Effect ids identify + attempts at runtime; sharing an OCaml type never establishes identity. *) +type bootstrap = + | No_bootstrap + | Preparing_snapshot of graph_scope + | Awaiting_snapshot_metadata of graph_scope * effect_id + | Downloading_snapshot of snapshot_download + | Activating_snapshot of snapshot_activation_request + +let bootstrap_scope = function + | No_bootstrap -> None + | Preparing_snapshot scope | Awaiting_snapshot_metadata (scope, _) -> Some scope + | Downloading_snapshot request -> Some request.scope + | Activating_snapshot request -> Some request.scope +;; + +let bootstrap_downloading = function Downloading_snapshot _ -> true | _ -> false +let bootstrap_activating = function Activating_snapshot _ -> true | _ -> false + +type mirror_deletion = + { account : account_scope + ; graph_id : graph_id + ; scope : effect_scope + } + +type mirror_inspection = + | Mirror_available of mirror_request + | Mirror_absent of graph_scope + +type outbox_transition_request = + { expected : Overlay.sync_token + ; key : graph_key_handle option + ; scope : graph_scope + ; transition : Overlay.outbox_transition + } + +type outbox_transition_failure = + { request : outbox_transition_request + ; message : string + } + +type authoritative_batch = + { input : Overlay.authoritative_batch + ; key : graph_key_handle option + ; scope : connection_scope + ; presentation_generation : presentation_generation + ; lifecycle_generation : lifecycle_generation + } + +type worker_effect = + | Inspect_mirror of mirror_request + | Activate_snapshot of snapshot_activation_request + | Delete_mirror of mirror_deletion + | Attach_graph of mirror_request + | Detach_graph of graph_scope + | Reset_managed_account of account_scope + | Inspect_sync of graph_scope + | Apply_outbox_transition of outbox_transition_request + | Apply_authoritative_batch of authoritative_batch + +type output = + | Asset_finished of asset_output + | Protected_finished of protected_output + | State_changed of state + | Resident_scopes_changed of graph_scope list + | Session_state_changed of graph_scope * state + | Bootstrap_progressed of bootstrap_progress + +type instruction = + | Run of runner_effect + | Delegate of worker_effect + | Publish of output + +let equal_instruction left right = left = right +let equal_instructions left right = left = right + +type scoped_error = + { scope : effect_scope + ; message : string + } + +type graph_attachment = + { scope : graph_scope + ; sync : Overlay.sync_view + } + +type sync_inspection = + { scope : graph_scope + ; sync : Overlay.sync_view + } + +type authoritative_commit_result = + { scope : graph_scope + ; commit : Overlay.authoritative_commit + ; sync : Overlay.sync_view + } + +type authoritative_deferred_result = + { scope : graph_scope + ; defer : Overlay.authoritative_defer + } + +type outbox_transition_result = + { scope : graph_scope + ; commit : Overlay.outbox_commit + ; sync : Overlay.sync_view + } + +type snapshot_activation = snapshot_activation_request +type snapshot_activation_error = { activation : snapshot_activation_request; message : string } +let snapshot_activation_completed (request : snapshot_activation_request) : snapshot_activation = request +let snapshot_activation_failed activation ~message = { activation; message } + +type startup_scope = Logseq_db_types.Sync_account.startup_scope = + { account_generation : account_generation + ; presentation_generation : presentation_generation + } + +type timeline_scope = Logseq_db_types.Sync_account.timeline_scope = + { startup : startup_scope + ; graph_id : graph_id + ; graph_generation : graph_generation + } + +type event = + | Host_fence_started of host_fence + | Host_fence_released of Logseq_db_types.Graph_types.Uuid.t + | IO_fence_requested of int64 * effect_scope + | IO_fence_completed of int64 * effect_scope * (unit,string) result + | Asset_requested of asset_request + | Protected_requested of protected_request + | Restore_local_account of { user_id : string } + | Account_authenticated of { user_id : string option } + | Local_feed_acknowledged + | Local_outbox_changed + | Resident_graphs_changed of graph_id list + | Visible_graph_changed of graph_id + | Local_graph_outbox_changed of graph_scope + | Timeline_presented of timeline_scope + | Graph_selected of graph_id + | Graph_picker_requested + | Sign_in_requested of startup_scope + | Catalog_refresh_requested of startup_scope + | Online_recovery_requested of timeline_scope + | Online_cancelled of startup_scope + | Local_account_unavailable of startup_scope * local_unavailability + | E2ee_password_submitted of unlock_challenge * string + | Local_cache_deletion_requested of graph_id + | Foreground_changed of + { foreground : bool + ; lifecycle_generation : lifecycle_generation + } + | Graph_detached of graph_scope * (unit, string) result + | Mirror_deleted of mirror_deletion * (unit, string) result + | Mirror_inspected of mirror_inspection + | Graph_attached of graph_attachment + | Graph_attachment_failed of scoped_error + | Sync_inspected of sync_inspection + | Outbox_transition_applied of outbox_transition_result + (** An uncommitted preparation lost its token to a newer durable revision. + The exact pending request and a changed token fence recovery; this is not + evidence of a commit or permission to repeat an unchanged failed request. *) + | Outbox_transition_superseded of + { request : outbox_transition_request; sync : Logseq_overlay_db.Types.sync_view } + | Outbox_transition_failed of outbox_transition_failure + | Authoritative_batch_applied of authoritative_commit_result + | Authoritative_batch_deferred of authoritative_deferred_result + | Authoritative_batch_failed of scoped_error + | Snapshot_activated of snapshot_activation + | Snapshot_activation_failed of snapshot_activation_error + | Runner_completed of runner_completion + | Snapshot_download_progress of bootstrap_progress + | Websocket_opened of connection_scope + | Websocket_message of connection_scope * Logseq_sync_protocol.Sync_protocol.Server.message + | Websocket_protocol_error of connection_scope * Logseq_sync_protocol.Sync_protocol.codec_error + | Websocket_closed of connection_scope * websocket_termination + | Websocket_retired of connection_scope + | Timer_elapsed of timer_id + | Shutdown + + +type coordination_state = Logseq_db_types.Sync_account.coordination_state = + { account : account_scope option + ; desired_residents : graph_id list + ; residents : (graph_scope * state) list + ; selected : timeline_scope option + ; deletion : (graph_scope * local_deletion) option + } diff --git a/logseq_sync/lib/domain/core.mli b/logseq_sync/lib/domain/core.mli new file mode 100644 index 00000000..69c7d6c6 --- /dev/null +++ b/logseq_sync/lib/domain/core.mli @@ -0,0 +1,708 @@ +module Overlay = Logseq_overlay_db.Types +module Server_cursor = Overlay.Server_cursor +type graph_id = Logseq_db_types.Sync_account.graph_id +val graph_id_to_yojson : graph_id -> Yojson.Safe.t +val graph_id_of_yojson : + Yojson.Safe.t -> graph_id Ppx_deriving_yojson_runtime.error_or +type cache_schema = + Logseq_db_types.Sync_account.cache_schema = { + major : int; + minor : int; + exact : bool; +} +val cache_schema_to_yojson : cache_schema -> Yojson.Safe.t +val cache_schema_of_yojson : + Yojson.Safe.t -> cache_schema Ppx_deriving_yojson_runtime.error_or +type graph = + Logseq_db_types.Sync_account.graph = { + graph_id : graph_id; + name : string; + schema : cache_schema; + encrypted : bool; +} +val graph_to_yojson : graph -> Yojson.Safe.t +val graph_of_yojson : + Yojson.Safe.t -> graph Ppx_deriving_yojson_runtime.error_or +type account_generation = Logseq_db_types.Sync_account.account_generation +type graph_generation = Logseq_db_types.Sync_account.graph_generation +type connection_generation = + Logseq_db_types.Sync_account.connection_generation +type presentation_generation = + Logseq_db_types.Sync_account.presentation_generation +type lifecycle_generation = Logseq_db_types.Sync_account.lifecycle_generation +type account_scope = + Logseq_db_types.Sync_account.account_scope = { + managed_sync_origin : Uri.t; + user_id : string; + account_generation : account_generation; + presentation_generation : presentation_generation; + lifecycle_generation : lifecycle_generation; +} +type graph_scope = + Logseq_db_types.Sync_account.graph_scope = { + account : account_scope; + graph_id : graph_id; + graph_generation : graph_generation; +} +module Host_cleanup_scopes : + sig + type elt = graph_scope + type t + val empty : t + val add : elt -> t -> t + val singleton : elt -> t + val remove : elt -> t -> t + val union : t -> t -> t + val inter : t -> t -> t + val disjoint : t -> t -> bool + val diff : t -> t -> t + val cardinal : t -> int + val elements : t -> elt list + val min_elt : t -> elt + val min_elt_opt : t -> elt option + val max_elt : t -> elt + val max_elt_opt : t -> elt option + val choose : t -> elt + val choose_opt : t -> elt option + val find : elt -> t -> elt + val find_opt : elt -> t -> elt option + val find_first : (elt -> bool) -> t -> elt + val find_first_opt : (elt -> bool) -> t -> elt option + val find_last : (elt -> bool) -> t -> elt + val find_last_opt : (elt -> bool) -> t -> elt option + val iter : (elt -> unit) -> t -> unit + val fold : (elt -> 'acc -> 'acc) -> t -> 'acc -> 'acc + val map : (elt -> elt) -> t -> t + val filter : (elt -> bool) -> t -> t + val filter_map : (elt -> elt option) -> t -> t + val partition : (elt -> bool) -> t -> t * t + val split : elt -> t -> t * bool * t + val is_empty : t -> bool + val is_singleton : t -> bool + val mem : elt -> t -> bool + val equal : t -> t -> bool + val compare : t -> t -> int + val subset : t -> t -> bool + val for_all : (elt -> bool) -> t -> bool + val exists : (elt -> bool) -> t -> bool + val to_list : t -> elt list + val of_list : elt list -> t + val to_seq_from : elt -> t -> elt Seq.t + val to_seq : t -> elt Seq.t + val to_rev_seq : t -> elt Seq.t + val add_seq : elt Seq.t -> t -> t + val of_seq : elt Seq.t -> t + end +type host_fence_target = One_graph of graph_id | All_graphs +type host_fence = { + id : Logseq_db_types.Graph_types.Uuid.t; + owner : account_scope; + target : host_fence_target; + captured_scopes : graph_scope Rrbvec.t; + previous : Logseq_db_types.Graph_types.Uuid.t option; +} +type connection_scope = { + graph : graph_scope; + connection_generation : connection_generation; +} +type sync_phase = + Logseq_db_types.Sync_account.sync_phase = + Offline + | Connecting + | Pulling + | Submitting + | Current + | Paused + | Failed +type startup_failure_stage = + Logseq_db_types.Sync_account.startup_failure_stage = + During_authentication + | During_catalog + | During_local_restore + | During_bootstrap + | During_e2ee +type local_unavailability = + Logseq_db_types.Sync_account.local_unavailability = + No_local_account + | Local_catalog_unavailable + | Local_mirror_unavailable + | Local_key_unavailable + | Local_restore_failed +type local_access = + Logseq_db_types.Sync_account.local_access = + Local_pending + | Local_available + | Local_unavailable of local_unavailability +type startup_network_phase = + Logseq_db_types.Sync_account.startup_network_phase = + Local_only + | Online_after_timeline + | Online_by_user +type startup_facts = + Logseq_db_types.Sync_account.startup_facts = { + authenticated : bool; + local_access : local_access; + network_phase : startup_network_phase; + catalog_loading : bool; + awaiting_selection : bool; + restoring_local : bool; + bootstrapping : bool; + awaiting_e2ee_password : bool; + failure : startup_failure_stage option; + account_generation : account_generation; + graph_generation : graph_generation; + presentation_generation : presentation_generation; +} +type local_deletion_stage = + Logseq_db_types.Sync_account.local_deletion_stage = + Closing_graph + | Deleting_mirror + | Clearing_selection +type local_deletion = + Logseq_db_types.Sync_account.local_deletion = + Deletion_in_progress of local_deletion_stage + | Deletion_failed of local_deletion_stage +type unlock_challenge = + Logseq_db_types.Sync_account.unlock_challenge = { + challenge_scope : graph_scope; + challenge_id : int; + encrypted_graph_key : string; + private_key_package : string; +} +type unlock_stage = + Logseq_db_types.Sync_account.unlock_stage = + Private_key of int + | Graph_key of int + | Attachment +type unlock_submission = + Logseq_db_types.Sync_account.unlock_submission = { + challenge : unlock_challenge; + operation_id : int; + stage : unlock_stage; +} +type unlock_attached = + Logseq_db_types.Sync_account.unlock_attached = { + submission : unlock_submission; + attached_scope : graph_scope; +} +type unlock_recovery = + Logseq_db_types.Sync_account.unlock_recovery = + Retry_password of unlock_challenge * string + | Retry_preparation of string +type unlock_phase = + Logseq_db_types.Sync_account.unlock_phase = + Unlock_inactive + | Unlock_challenge of unlock_challenge + | Unlock_running of unlock_submission + | Unlock_bootstrapping of unlock_submission + | Unlock_presentable of unlock_attached + | Unlock_failed of unlock_recovery + | Unlock_cancelled of unlock_submission +val unlock_challenge : unlock_phase -> unlock_challenge option +val equal_unlock_challenge : unlock_challenge -> unlock_challenge -> bool +val unlock_challenge_scope : unlock_challenge -> graph_scope +val unlock_submission_challenge : unlock_submission -> unlock_challenge +val equal_unlock_submission : unlock_submission -> unlock_submission -> bool +val unlock_attached_submission : unlock_attached -> unlock_submission +val unlock_attached_scope : unlock_attached -> graph_scope +type error_attention = + Logseq_db_types.Sync_account.error_attention = + Transient_connection + | Requires_attention +type snapshot = + Logseq_db_types.Sync_account.snapshot = { + sync_phase : sync_phase; + catalog : graph list; + selected_graph : graph_id option; + resident_graphs : graph_id list; + applied_server_t : Server_cursor.t option; + timeline_presentation_pending : bool; + unlock : unlock_phase; + startup : startup_facts; + last_error : string option; + last_error_attention : error_attention option; + local_deletion : local_deletion option; +} +type diagnostic_group = + Logseq_db_types.Sync_account.diagnostic_group = { + title : string; + entries : (string * string) list; +} +type diagnostics = + Logseq_db_types.Sync_account.diagnostics = { + groups : diagnostic_group list; +} +type state = + Logseq_db_types.Sync_account.state = { + snapshot : snapshot; + diagnostics : diagnostics; +} +type limits = { + maximum_response_bytes : int; + maximum_artifact_bytes : int; + submission_batch_size : int; +} +type config = { config_origin : Uri.t; limits : limits; } +type config_error = Invalid_config of string +val limits : + maximum_response_bytes:int -> + maximum_artifact_bytes:int -> + submission_batch_size:int -> (limits, config_error) result +val config : + managed_sync_origin:Uri.t -> limits:limits -> (config, config_error) result +type bootstrap_progress = { + graph_id : graph_id; + received_bytes : int64; + total_bytes : int64 option; +} +type invalidation = { + basis : int64; + changed_uuids : graph_id list; + changed_uuids_truncated : bool; +} +type effect_scope = { + account_generation : account_generation option; + graph_generation : graph_generation option; + connection_generation : connection_generation option; + presentation_generation : presentation_generation option; + lifecycle_generation : lifecycle_generation option; +} +type effect_id = int +type crypto_failure_kind = Invalid_key_material | Crypto_provider_unavailable +type effect_error = + Effect_failed of string + | Crypto_failed of crypto_failure_kind * string +type graph_key_handle = { handle_id : string; handle_scope : graph_scope; } +type snapshot_download = { + scope : graph_scope; + uri : Uri.t; + expected_bytes : int64 option; + maximum_bytes : int; + metadata_cursor : Server_cursor.t; + attempt_id : effect_id; + download_id : effect_id; +} +type staged_artifact = { + download : snapshot_download; + artifact_path : string; + artifact_expected_rows : int; +} +type catalog_cache = + Logseq_db_types.Sync_account.catalog_cache = { + cache_user_id : string; + cache_graphs : graph list; + cache_selected_graph : graph_id option; + cache_resident_graphs : graph_id list option; +} +val catalog_cache_to_yojson : catalog_cache -> Yojson.Safe.t +val catalog_cache_of_yojson : + Yojson.Safe.t -> catalog_cache Ppx_deriving_yojson_runtime.error_or +val graph_key_handle : id:string -> scope:graph_scope -> graph_key_handle +val graph_key_handle_id : graph_key_handle -> string +val graph_key_handle_scope : graph_key_handle -> graph_scope +val stage_snapshot_download : + snapshot_download -> + path:string -> expected_rows:int -> (staged_artifact, string) result +val snapshot_download_scope : snapshot_download -> graph_scope +val snapshot_download_uri : snapshot_download -> Uri.t +val snapshot_download_expected_bytes : snapshot_download -> int64 option +val snapshot_download_maximum_bytes : snapshot_download -> int +val staged_artifact_path : staged_artifact -> string +val staged_artifact_expected_rows : staged_artifact -> int +val effect_scope_of_account : account_scope -> effect_scope +val effect_scope_of_graph : graph_scope -> effect_scope +val effect_scope_of_connection : connection_scope -> effect_scope +val catalog_cache : + user_id:string -> + graphs:graph list -> selected_graph:graph_id option -> catalog_cache +val catalog_cache_user_id : catalog_cache -> string +val catalog_cache_graphs : catalog_cache -> graph list +val catalog_cache_selected_graph : catalog_cache -> graph_id option +val catalog_cache_resident_graphs : catalog_cache -> graph_id list +val encode_catalog_cache : catalog_cache -> string +val decode_catalog_cache : string -> (catalog_cache, string) result +type encryption_batch = { + scope : graph_scope; + key : graph_key_handle; + plaintexts : string list; +} +type encrypted_values = (string * string) list +type decryption_batch = { + scope : graph_scope; + key : graph_key_handle; + protected_values : (string * string) list; +} +type decrypted_values = string list +type snapshot_baseline = string +type snapshot_metadata = string +type graph_key_request = { + scope : graph_scope; + encrypted_graph_key : string; +} +type private_key_unlock = { + scope : account_scope; + password : string; + private_key_package : string; +} +type retirement_id = Logseq_db_types.Graph_types.Retirement_id.t +val retirement_id_of_string : + string -> (Logseq_db_types.Graph_types.Retirement_id.t, string) result +val retirement_id_to_string : + Logseq_db_types.Graph_types.Retirement_id.t -> string +type asset_deletion = { + deletion_scope : graph_scope; + retirement : retirement_id; +} +val asset_deletion_reference : + scope:graph_scope -> retirement:retirement_id -> asset_deletion +val asset_deletion_scope : asset_deletion -> graph_scope +val asset_deletion_retirement : asset_deletion -> retirement_id +type asset_context = { + scope : graph_scope; + encrypted : bool; + key : graph_key_handle option; +} +type asset_action = + Check_asset_cache of graph_id * Logseq_db_types.Asset_descriptor.version + | Fetch_asset of { asset : graph_id; + version : Logseq_db_types.Asset_descriptor.version; + maximum_plaintext_bytes : int; + } + | Stage_asset_file of { operation : graph_id; file_type : string; + source_file : string; + } + | Put_asset_file of { asset : graph_id; + version : Logseq_db_types.Asset_descriptor.version; file : string; + maximum_plaintext_bytes : int; + } + | Publish_staged_asset of { asset : Logseq_db_types.Graph_types.Uuid.t; + version : Logseq_db_types.Asset_descriptor.version; file : string; + } + | Retain_asset_file of string + | Retain_staged_file of string + | Release_asset_file of string + | Release_staged_file of string + | Prune_asset_staging of string list + | Await_asset_operations + | Close_asset_scope + | Prepare_graph_asset_deletion of retirement_id + | Commit_graph_asset_deletion of asset_deletion + | Cancel_graph_asset_deletion of asset_deletion + | Delete_account_assets + | Asset_retry_after of float + | Cancel_asset_operation of string +type asset_failure = + Asset_network + | Asset_not_found + | Asset_checksum_mismatch + | Asset_authentication + | Asset_locked + | Asset_storage_full + | Asset_missing_source + | Asset_size_rejected + | Asset_revoked_access + | Asset_cancelled + | Asset_invalid_content of string + | Asset_with_cause of asset_failure * Logseq_db_types.Failure_diagnostic.t +val asset_failure_kind : asset_failure -> asset_failure +val asset_failure_cause : + asset_failure -> Logseq_db_types.Failure_diagnostic.t option +val asset_failure_with_cause : + asset_failure -> Logseq_db_types.Failure_diagnostic.t -> asset_failure +type asset_value = + Asset_unit + | Asset_cached of string option + | Asset_downloaded of string + | Asset_staged of { file : string; checksum : string; size : int64; } + | Asset_retained of (string * string) option + | Asset_deletion_prepared of asset_deletion + | Asset_pruned of int + | Asset_retry_elapsed +type asset_result = (asset_value, asset_failure) result +type asset_request = { + scope : graph_scope; + operation : string; + action : asset_action; +} +type asset_io_request = { + context : asset_context; + operation : string; + action : asset_action; +} +type asset_ticket = { asset_serial : int; asset_request : asset_io_request; } +val asset_ticket_id : asset_ticket -> string +type asset_output = { request : asset_request; result : asset_result; } +type protected_action = + Encrypt_values of graph_key_handle * string list + | Decrypt_value of graph_key_handle * string +type protected_request = { + protected_operation : string; + protected_scope : graph_scope; + protected_action : protected_action; +} +type protected_value = + Encrypted_values of (string * string) list + | Decrypted_value of string +type protected_result = (protected_value, effect_error) result +type protected_output = { + protected_request : protected_request; + protected_result : protected_result; +} +type protected_ticket = { + protected_serial : int; + protected_input : protected_request; +} +val protected_ticket_id : protected_ticket -> string +type _ runner_request = + Load_catalog : account_scope -> catalog_cache option runner_request + | Save_catalog : { account : account_scope; cache : catalog_cache; + } -> unit runner_request + | Fetch_catalog : account_scope -> graph list runner_request + | Fetch_snapshot_baseline : graph_scope -> snapshot_baseline runner_request + | Fetch_snapshot_metadata : graph_scope -> snapshot_metadata runner_request + | Download_snapshot : snapshot_download -> staged_artifact runner_request + | Fetch_e2ee_graph_key : graph_scope -> string runner_request + | Fetch_e2ee_user_keys : account_scope -> string runner_request + | Load_and_unlock_graph_key : + graph_scope -> graph_key_handle runner_request + | Fetch_and_unlock_graph_key : + graph_key_request -> graph_key_handle runner_request + | Unlock_private_key : private_key_unlock -> unit runner_request + | Delete_account_secrets : account_scope -> unit runner_request + | Encrypt_protected_values : + encryption_batch -> encrypted_values runner_request + | Decrypt_protected_values : + decryption_batch -> decrypted_values runner_request +type _ request_kind = + Load_catalog_kind : catalog_cache option request_kind + | Save_catalog_kind : unit request_kind + | Fetch_catalog_kind : graph list request_kind + | Fetch_snapshot_baseline_kind : snapshot_baseline request_kind + | Fetch_snapshot_metadata_kind : snapshot_metadata request_kind + | Download_snapshot_kind : staged_artifact request_kind + | Fetch_e2ee_graph_key_kind : string request_kind + | Fetch_e2ee_user_keys_kind : string request_kind + | Load_and_unlock_graph_key_kind : graph_key_handle request_kind + | Fetch_and_unlock_graph_key_kind : graph_key_handle request_kind + | Unlock_private_key_kind : unit request_kind + | Delete_account_secrets_kind : unit request_kind + | Encrypt_protected_values_kind : encrypted_values request_kind + | Decrypt_protected_values_kind : decrypted_values request_kind +type 'a effect_ticket = { + id : effect_id; + scope : effect_scope; + kind : 'a request_kind; +} +val effect_ticket_id : 'a effect_ticket -> effect_id +val effect_ticket_scope : 'a effect_ticket -> effect_scope +val effect_id_to_string : int -> string +type websocket_request = { scope : connection_scope; uri : Uri.t; } +type websocket_send = { + scope : connection_scope; + message : Logseq_sync_protocol.Sync_protocol.Client.message; +} +type websocket_termination = + Recoverable_close of { message : string option; + attention : error_attention; + } + | Local_close + | Authentication_failure of string + | Protocol_failure of string +type timer_id = int +type timer_request = { + id : timer_id; + scope : effect_scope; + delay_seconds : float; +} +type runner_effect = + Await_io of int64 * effect_scope + | Asset_io of asset_ticket * asset_io_request + | Protected_io of protected_ticket * protected_request + | Request : 'a effect_ticket * 'a runner_request -> runner_effect + | Start_websocket of websocket_request + | Send_websocket of websocket_send + | Close_websocket of connection_scope + | Schedule_timer of timer_request + | Cancel_effects of effect_scope + | Cancel_network_effects of effect_scope +type runner_completion = + Asset_completion of asset_ticket * asset_result + | Protected_completion of protected_ticket * protected_result + | Completion : 'a effect_ticket * + ('a, effect_error) result -> runner_completion +val scope_of_request : 'a runner_request -> effect_scope +val runner_effect_scope : runner_effect -> effect_scope +val runner_effect_diagnostic : runner_effect -> string +val equal_runner_effect : 'a -> 'a -> bool +type mirror_request = { graph : graph; scope : graph_scope; } +type snapshot_activation_request = private { + artifact : staged_artifact; + scope : graph_scope; + applied_server_t : Server_cursor.t; + key : graph_key_handle option; +} +val activate_staged_snapshot : + staged_artifact -> key:graph_key_handle option -> (snapshot_activation_request, string) result +val snapshot_activation_scope : snapshot_activation_request -> graph_scope +val snapshot_activation_cursor : + snapshot_activation_request -> Server_cursor.t +val snapshot_activation_artifact : + snapshot_activation_request -> staged_artifact +val snapshot_activation_key : + snapshot_activation_request -> graph_key_handle option +type bootstrap = + No_bootstrap + | Preparing_snapshot of graph_scope + | Awaiting_snapshot_metadata of graph_scope * effect_id + | Downloading_snapshot of snapshot_download + | Activating_snapshot of snapshot_activation_request +val bootstrap_scope : bootstrap -> graph_scope option +val bootstrap_downloading : bootstrap -> bool +val bootstrap_activating : bootstrap -> bool +type mirror_deletion = { + account : account_scope; + graph_id : graph_id; + scope : effect_scope; +} +type mirror_inspection = + Mirror_available of mirror_request + | Mirror_absent of graph_scope +type outbox_transition_request = { + expected : Overlay.sync_token; + key : graph_key_handle option; + scope : graph_scope; + transition : Overlay.outbox_transition; +} +type outbox_transition_failure = { + request : outbox_transition_request; + message : string; +} +type authoritative_batch = { + input : Overlay.authoritative_batch; + key : graph_key_handle option; + scope : connection_scope; + presentation_generation : presentation_generation; + lifecycle_generation : lifecycle_generation; +} +type worker_effect = + Inspect_mirror of mirror_request + | Activate_snapshot of snapshot_activation_request + | Delete_mirror of mirror_deletion + | Attach_graph of mirror_request + | Detach_graph of graph_scope + | Reset_managed_account of account_scope + | Inspect_sync of graph_scope + | Apply_outbox_transition of outbox_transition_request + | Apply_authoritative_batch of authoritative_batch +type output = + Asset_finished of asset_output + | Protected_finished of protected_output + | State_changed of state + | Resident_scopes_changed of graph_scope list + | Session_state_changed of graph_scope * state + | Bootstrap_progressed of bootstrap_progress +type instruction = + Run of runner_effect + | Delegate of worker_effect + | Publish of output +val equal_instruction : 'a -> 'a -> bool +val equal_instructions : 'a -> 'a -> bool +type scoped_error = { scope : effect_scope; message : string; } +type graph_attachment = { scope : graph_scope; sync : Overlay.sync_view; } +type sync_inspection = { scope : graph_scope; sync : Overlay.sync_view; } +type authoritative_commit_result = { + scope : graph_scope; + commit : Overlay.authoritative_commit; + sync : Overlay.sync_view; +} +type authoritative_deferred_result = { + scope : graph_scope; + defer : Overlay.authoritative_defer; +} +type outbox_transition_result = { + scope : graph_scope; + commit : Overlay.outbox_commit; + sync : Overlay.sync_view; +} +type snapshot_activation = snapshot_activation_request +type snapshot_activation_error = { + activation : snapshot_activation_request; + message : string; +} +val snapshot_activation_completed : + snapshot_activation_request -> snapshot_activation +val snapshot_activation_failed : + snapshot_activation_request -> message:string -> snapshot_activation_error +type startup_scope = + Logseq_db_types.Sync_account.startup_scope = { + account_generation : account_generation; + presentation_generation : presentation_generation; +} +type timeline_scope = + Logseq_db_types.Sync_account.timeline_scope = { + startup : startup_scope; + graph_id : graph_id; + graph_generation : graph_generation; +} +type event = + Host_fence_started of host_fence + | Host_fence_released of Logseq_db_types.Graph_types.Uuid.t + | IO_fence_requested of int64 * effect_scope + | IO_fence_completed of int64 * effect_scope * (unit, string) result + | Asset_requested of asset_request + | Protected_requested of protected_request + | Restore_local_account of { user_id : string; } + | Account_authenticated of { user_id : string option; } + | Local_feed_acknowledged + | Local_outbox_changed + | Resident_graphs_changed of graph_id list + | Visible_graph_changed of graph_id + | Local_graph_outbox_changed of graph_scope + | Timeline_presented of timeline_scope + | Graph_selected of graph_id + | Graph_picker_requested + | Sign_in_requested of startup_scope + | Catalog_refresh_requested of startup_scope + | Online_recovery_requested of timeline_scope + | Online_cancelled of startup_scope + | Local_account_unavailable of startup_scope * local_unavailability + | E2ee_password_submitted of unlock_challenge * string + | Local_cache_deletion_requested of graph_id + | Foreground_changed of { foreground : bool; + lifecycle_generation : lifecycle_generation; + } + | Graph_detached of graph_scope * (unit, string) result + | Mirror_deleted of mirror_deletion * (unit, string) result + | Mirror_inspected of mirror_inspection + | Graph_attached of graph_attachment + | Graph_attachment_failed of scoped_error + | Sync_inspected of sync_inspection + | Outbox_transition_applied of outbox_transition_result + | Outbox_transition_superseded of { request : outbox_transition_request; + sync : Logseq_overlay_db.Types.sync_view; + } + | Outbox_transition_failed of outbox_transition_failure + | Authoritative_batch_applied of authoritative_commit_result + | Authoritative_batch_deferred of authoritative_deferred_result + | Authoritative_batch_failed of scoped_error + | Snapshot_activated of snapshot_activation + | Snapshot_activation_failed of snapshot_activation_error + | Runner_completed of runner_completion + | Snapshot_download_progress of bootstrap_progress + | Websocket_opened of connection_scope + | Websocket_message of connection_scope * + Logseq_sync_protocol.Sync_protocol.Server.message + | Websocket_protocol_error of connection_scope * + Logseq_sync_protocol.Sync_protocol.codec_error + | Websocket_closed of connection_scope * websocket_termination + | Websocket_retired of connection_scope + | Timer_elapsed of timer_id + | Shutdown +type coordination_state = + Logseq_db_types.Sync_account.coordination_state = { + account : account_scope option; + desired_residents : graph_id list; + residents : (graph_scope * state) list; + selected : timeline_scope option; + deletion : (graph_scope * local_deletion) option; +} diff --git a/logseq_sync/lib/domain/dune b/logseq_sync/lib/domain/dune new file mode 100644 index 00000000..cde2066f --- /dev/null +++ b/logseq_sync/lib/domain/dune @@ -0,0 +1,6 @@ +(library + (name logseq_sync_domain) + (public_name logseq_sync.domain) + (modules core) + (libraries logseq_sync.protocol logseq_db_types logseq_overlay_db ppx_deriving_yojson.runtime rrbvec uri yojson) + (preprocess (pps ppx_deriving_yojson))) diff --git a/logseq_sync/lib/effect_runner/dune b/logseq_sync/lib/effect_runner/dune deleted file mode 100644 index 3752658c..00000000 --- a/logseq_sync/lib/effect_runner/dune +++ /dev/null @@ -1,63 +0,0 @@ -(include_subdirs unqualified) - -(library - (name logseq_sync_effect_runner_impl) - (public_name logseq_sync.effect_runner.impl) - (implements logseq_sync_effect_runner) - (modules - asset_cache - asset_codec - artifact_decoder - bootstrap - catalog - e2ee - effect_runner - http - http_eio - platform_crypto - tls_client_eio - websocket_eio) - (private_modules - artifact_decoder - bootstrap - catalog - e2ee - http - http_eio - platform_crypto - tls_client_eio - websocket_eio) - (libraries - bigstringaf - ca-certs-nss - cstruct - digestif - domain-name - eio - eio.unix - faraday - httpun - httpun-eio - httpun-ws - logseq_overlay_db - logseq_sync.pure_reducer - logseq_db_types - melange-transit-core - melange-transit-native - mirage-crypto-rng - mirage-crypto-rng.unix - rrbvec - tls - tls-eio - unix - uri - x509 - ppx_deriving_yojson.runtime - yojson) - (preprocess - (pps ppx_deriving_yojson)) - (flags - (:standard -open Logseq_db_types)) - (foreign_stubs - (language c) - (names platform_crypto_stubs gzip_stubs))) diff --git a/logseq_sync/lib/effect_runner/effect_runner.ml b/logseq_sync/lib/effect_runner/effect_runner.ml deleted file mode 100644 index 7f710b5f..00000000 --- a/logseq_sync/lib/effect_runner/effect_runner.ml +++ /dev/null @@ -1,2395 +0,0 @@ -module Core = Logseq_sync_pure_reducer.Core -module Sync_protocol = Logseq_sync_pure_reducer.Sync_protocol -module Asset_cache = Bootstrap.Asset_cache -module Failure_diagnostic = Logseq_db_types.Failure_diagnostic - -type unlock_failure_category = Token | Transport | Http_status | Decode | Crypto - -type unlock_failure_trace = - { operation_id : string - ; mutable category : unlock_failure_category option - ; mutable http_status : int option - } - -let unlock_failure_trace = Eio.Fiber.create_key () - -let observe_unlock_failure category = - if Sys.getenv_opt "JOURNAL_UNLOCK_DIAGNOSTICS" = Some "1" then - Option.iter (fun trace -> trace.category <- Some category) (Eio.Fiber.get unlock_failure_trace) -;; - -let observe_unlock_http_status status = - if Sys.getenv_opt "JOURNAL_UNLOCK_DIAGNOSTICS" = Some "1" then - Option.iter (fun trace -> trace.http_status <- Some status) (Eio.Fiber.get unlock_failure_trace) -;; - -type dependency_error = Invalid_dependency of string -type create_error = Invalid_create of string - -type runtime = - { fork : sw:Eio.Switch.t -> (unit -> unit) -> unit - ; sleep : float -> unit - } - -let runtime ~fork ~sleep = Ok { fork; sleep } - -type transport = - { perform_http : sw:Eio.Switch.t -> Http.request -> (Http_eio.response, string) result - ; download : - sw:Eio.Switch.t - -> request:Http.request - -> destination:string - -> maximum_bytes:int - -> on_progress:(Bootstrap.progress -> unit) - -> (Http_eio.response, string) result - ; connect_websocket : - sw:Eio.Switch.t - -> uri:Uri.t - -> token:string - -> maximum_frame_bytes:int - -> on_message:(string -> unit) - -> on_close:(Core.websocket_termination -> unit) - -> (Websocket_eio.t, Websocket_eio.connect_error) result - ; send_websocket : Websocket_eio.t -> string -> (unit, string) result - ; await_websocket_retired : Websocket_eio.t -> unit - ; close_websocket : Websocket_eio.t -> unit - ; abort_websocket : Websocket_eio.t -> unit - ; activate_websocket : - Websocket_eio.t - -> on_open:(unit -> unit) - -> (unit, Websocket_eio.connect_error) result - } - -type tls_authenticator = X509.Authenticator.t - -type id_token_provider = - { acquire : Core.account_scope -> (string, string) result - ; invalidate : Core.account_scope -> token:string -> unit - } - -let id_token_provider ~acquire ~invalidate = { acquire; invalidate } - -type authenticated_failure = - | Unauthorized - | Forbidden - | Request_failed of string - -let authenticated_failure = function - | Unauthorized -> "Authentication failed." - | Forbidden -> "Authorization failed." - | Request_failed message -> message -;; - -let authenticated_operation provider ~account ~perform = - match provider.acquire account with - | Error message -> - observe_unlock_failure Token; - Error message - | Ok token -> - (match perform token with - | Ok _ as result -> result - | Error Unauthorized -> - provider.invalidate account ~token; - (match provider.acquire account with - | Error message -> - observe_unlock_failure Token; - Error message - | Ok refreshed -> Result.map_error authenticated_failure (perform refreshed)) - | Error failure -> Error (authenticated_failure failure)) -;; - -let tls_authenticator authenticator = authenticator - -let system_tls_authenticator () = - Ca_certs_nss.authenticator () - |> Result.map_error (fun (`Msg message) -> Invalid_dependency message) -;; - -type websocket_liveness = - | Disabled - | Ping_pong of - { interval_seconds : float - ; timeout_seconds : float - } - -let transport ~tls_authenticator ~network ~clock ~websocket_liveness = - let valid = function - | Disabled -> true - | Ping_pong { interval_seconds; timeout_seconds } -> - Float.is_finite interval_seconds - && Float.is_finite timeout_seconds - && interval_seconds > 0. - && timeout_seconds > 0. - in - let liveness = - match websocket_liveness with - | Disabled -> Websocket_eio.Disabled - | Ping_pong { interval_seconds; timeout_seconds } -> - Websocket_eio.Ping_pong { interval_seconds; timeout_seconds } - in - if not (valid websocket_liveness) - then - Error (Invalid_dependency "WebSocket liveness durations must be finite and positive") - else - Ok - { perform_http = - (fun ~sw request -> - Http_eio.perform ~sw ~authenticator:tls_authenticator ~network ~clock request) - ; download = - (fun ~sw ~request ~destination ~maximum_bytes ~on_progress -> - Http_eio.download - ~sw - ~authenticator:tls_authenticator - ~network - ~clock - ~request - ~destination - ~maximum_bytes - ~on_progress) - ; connect_websocket = - (fun ~sw ~uri ~token ~maximum_frame_bytes ~on_message ~on_close -> - Websocket_eio.connect - ~liveness - ~sw - ~authenticator:tls_authenticator - ~network - ~clock - ~uri - ~token - ~maximum_frame_bytes - ~on_message - ~on_close) - ; send_websocket = Websocket_eio.send - ; await_websocket_retired = Websocket_eio.await_retired - ; close_websocket = Websocket_eio.close - ; abort_websocket = Websocket_eio.abort - ; activate_websocket = Websocket_eio.activate - } -;; - -type local_store = - { application_support_directory : string - ; asset_cache_budget_bytes : int64 - ; asset_maximum_file_bytes : int - ; asset_pending_budget_bytes : int64 - } - -let existing_directory path = - String.length path > 0 && Sys.file_exists path && (Unix.stat path).st_kind = Unix.S_DIR -;; - -let local_store - ?(asset_cache_budget_bytes = 268435456L) - ?(asset_maximum_file_bytes = 8 * 1024 * 1024) - ?(asset_pending_budget_bytes = 268435456L) - ~application_support_directory - () - = - if - asset_cache_budget_bytes <= 0L - || asset_cache_budget_bytes > 268435456L - || asset_maximum_file_bytes <= 0 - || asset_maximum_file_bytes > 8 * 1024 * 1024 - || asset_pending_budget_bytes <= 0L - || asset_pending_budget_bytes > 268435456L - then Error (Invalid_dependency "Asset storage limits exceed the supported bounds") - else if existing_directory application_support_directory - then - Ok - { application_support_directory - ; asset_cache_budget_bytes - ; asset_maximum_file_bytes - ; asset_pending_budget_bytes - } - else Error (Invalid_dependency "application support directory does not exist") -;; - -type artifact_store = { staging_directory : string } - -let artifact_store ~staging_directory = - if String.length staging_directory = 0 - then Error (Invalid_dependency "staging directory must not be empty") - else Ok { staging_directory } -;; - -type wrapped_key_load_error = - | Wrapped_graph_key_unavailable of string - | Local_private_key_unavailable of string - -type secrets = - { unlock_private_key : - managed_sync_origin:Uri.t - -> user_id:string - -> password:string - -> private_key_package:string - -> (unit, string) result - ; unlock_graph_key : - managed_sync_origin:Uri.t - -> user_id:string - -> encrypted_graph_key:string - -> (string, string) result - ; load_wrapped_graph_key : - managed_sync_origin:Uri.t - -> user_id:string - -> graph_id:Core.graph_id - -> (string, wrapped_key_load_error) result - ; verify_and_save_wrapped_graph_key : - managed_sync_origin:Uri.t - -> user_id:string - -> graph_id:Core.graph_id - -> encrypted_graph_key:string - -> (unit, string) result - ; delete_account_secrets : - managed_sync_origin:Uri.t -> user_id:string -> (unit, string) result - } - -let secrets - ~unlock_private_key - ~unlock_graph_key - ~load_wrapped_graph_key - ~verify_and_save_wrapped_graph_key - ~delete_account_secrets - = - Ok - { unlock_private_key - ; unlock_graph_key - ; load_wrapped_graph_key - ; verify_and_save_wrapped_graph_key - ; delete_account_secrets - } -;; - -type crypto = - { encrypt_aes_gcm : key:string -> plaintext:string -> (string * string, string) result - ; decrypt_aes_gcm : - key:string -> iv:string -> ciphertext:string -> (string, string) result - } - -let crypto ~encrypt_aes_gcm ~decrypt_aes_gcm = Ok { encrypt_aes_gcm; decrypt_aes_gcm } - -let apple_secrets () = - secrets - ~unlock_private_key:Platform_crypto.unlock_private_key - ~unlock_graph_key:Platform_crypto.unlock_graph_key - ~load_wrapped_graph_key:(fun ~managed_sync_origin ~user_id ~graph_id -> - match - Platform_crypto.load_wrapped_graph_key ~managed_sync_origin ~user_id ~graph_id - with - | Ok key -> Ok key - | Error (Platform_crypto.Wrapped_graph_key_unavailable message) -> - Error (Wrapped_graph_key_unavailable message) - | Error (Platform_crypto.Local_private_key_unavailable message) -> - Error (Local_private_key_unavailable message)) - ~verify_and_save_wrapped_graph_key:Platform_crypto.verify_and_save_wrapped_graph_key - ~delete_account_secrets:Platform_crypto.delete_account_secrets -;; - -let apple_crypto () = - let adapter = Platform_crypto.crypto in - crypto ~encrypt_aes_gcm:adapter.encrypt_aes_gcm ~decrypt_aes_gcm:adapter.decrypt_aes_gcm -;; - -type dependencies = - { runtime : runtime - ; transport : transport - ; local_store : local_store - ; artifact_store : artifact_store - ; secrets : secrets - ; crypto : crypto - ; id_token_provider : id_token_provider - } - -let dependencies - ~runtime - ~transport - ~local_store - ~artifact_store - ~secrets - ~crypto - ~id_token_provider - = - Ok - { runtime - ; transport - ; local_store - ; artifact_store - ; secrets - ; crypto - ; id_token_provider - } -;; - -type operation = - { network : bool - ; scope : Core.effect_scope - ; asset_identity : (Core.graph_scope * string) option - ; mutable cancelled : bool - ; cancel : unit -> unit - } - -type key_entry = - { scope : Core.graph_scope - ; bytes : bytes - } - -(* Transfer lanes bound request counts; this shared semaphore bounds the - in-flight byte footprint of those lanes. Each admitted download or upload - reserves its worst-case wire plus plaintext footprint before buffers are - held, in 64 KiB units, so total asset memory stays bounded independently - of the lane permits. *) -let asset_byte_unit = 65536 -let asset_byte_budget_units = 1024 - -(* The private effect cannot be reconstructed by callers. Its identity must - stay replay-protected for as long as a caller can retain and resubmit it, - without retaining historical effects (or their scope strings) ourselves. *) -module Submitted_effects = Weak.Make (struct - type t = Core.runner_effect - - let equal = ( == ) - let hash instruction = Hashtbl.hash (Core.runner_effect_diagnostic instruction) - end) - -type websocket_owner = - { physical_scope : Core.connection_scope - ; operation : operation - ; pending : Websocket_eio.t option ref - ; finished : unit Eio.Promise.t - ; mutable retirement_requested : bool - } - -type t = - { sw : Eio.Switch.t - ; dependencies : dependencies - ; post : Core.event -> unit - ; on_asset_diagnostic : scope:Core.graph_scope -> operation:string - -> request_operation:string -> Failure_diagnostic.t -> unit - ; operations : (string, operation) Hashtbl.t - ; asset_jobs : (string,Core.graph_scope * unit Eio.Promise.t)Hashtbl.t - ; mutable next_general_job : int64 - ; general_jobs : (int64,Core.effect_scope * unit Eio.Promise.t)Hashtbl.t - ; submitted_operations : Submitted_effects.t - ; keys : (string, key_entry) Hashtbl.t - ; websockets : (Core.connection_scope, Websocket_eio.t) Hashtbl.t - ; websocket_owners : (Core.graph_scope, websocket_owner) Hashtbl.t - ; mutable retired_websockets : Core.connection_scope list - ; asset_download_slots : Eio.Semaphore.t - ; asset_upload_slots : Eio.Semaphore.t - ; asset_codec_slot : Eio.Semaphore.t - ; asset_byte_budget : Eio.Semaphore.t - ; asset_byte_reservation_lock : Eio.Semaphore.t - ; secret_lock : Eio.Mutex.t - ; asset_caches : (Core.graph_scope, Asset_cache.t) Hashtbl.t - ; asset_leases : ((Core.graph_scope * string), unit) Hashtbl.t - ; asset_retirements : ((string * string * string), Core.retirement_id) Hashtbl.t - ; preparing_retirements : ((string * string * string), Core.retirement_id) Hashtbl.t - ; mutable closed : bool - } - -let create ?(on_asset_diagnostic=(fun ~scope:_ ~operation:_ ~request_operation:_ _ -> ())) - ~sw dependencies ~post = - Ok - { sw - ; dependencies - ; post - ; on_asset_diagnostic - ; operations = Hashtbl.create 32 - ; asset_jobs = Hashtbl.create 32 - ; next_general_job = 0L - ; general_jobs = Hashtbl.create 32 - ; submitted_operations = Submitted_effects.create 32 - ; keys = Hashtbl.create 8 - ; websockets = Hashtbl.create 4 - ; websocket_owners = Hashtbl.create 3 - ; retired_websockets = [] - ; asset_download_slots = Eio.Semaphore.make 3 - ; asset_upload_slots = Eio.Semaphore.make 1 - ; asset_codec_slot = Eio.Semaphore.make 1 - ; asset_byte_budget = Eio.Semaphore.make asset_byte_budget_units - ; asset_byte_reservation_lock = Eio.Semaphore.make 1 - ; secret_lock = Eio.Mutex.create () - ; asset_caches = Hashtbl.create 4 - ; asset_leases = Hashtbl.create 16 - ; asset_retirements = Hashtbl.create 4 - ; preparing_retirements = Hashtbl.create 4 - ; closed = false - } -;; - -let fork_general t ~scope task = - if t.next_general_job=Int64.max_int then failwith "Sync I/O owner counter exhausted"; - t.next_general_job<-Int64.succ t.next_general_job; - let id=t.next_general_job in - let finished,resolve=Eio.Promise.create() in - Hashtbl.add t.general_jobs id(scope,finished); - let complete()=Hashtbl.remove t.general_jobs id;ignore(Eio.Promise.try_resolve resolve():bool) in - try t.dependencies.runtime.fork ~sw:t.sw(fun()->Fun.protect ~finally:complete task) - with error->complete();raise error -;; - -let asset_root t = - Filename.concat - t.dependencies.local_store.application_support_directory - "logseq-db-worker/assets" -;; - -let asset_target (scope : Core.graph_scope) = - Uri.to_string scope.account.managed_sync_origin, scope.account.user_id, - Graph_types.Uuid.to_string scope.graph_id -;; -let asset_blocked t scope = Hashtbl.mem t.asset_retirements (asset_target scope) -;; - -let delete_account_assets ?except_id t (account : Core.account_scope) = - Hashtbl.iter - (fun id (operation : operation) -> - match operation.asset_identity with - | Some (scope, _) - when Some id <> except_id - && Uri.equal scope.account.managed_sync_origin account.managed_sync_origin - && String.equal scope.account.user_id account.user_id -> - operation.cancelled <- true; - operation.cancel () - | Some _ | None -> ()) - t.operations; - let scopes=Hashtbl.to_seq_keys t.asset_caches |> Seq.filter(fun(scope:Core.graph_scope)-> - Uri.equal scope.account.managed_sync_origin account.managed_sync_origin - && scope.account.user_id=account.user_id) |> Rrbvec.of_seq in - let closed=Rrbvec.fold_left(fun result scope->Result.bind result(fun()-> - match Hashtbl.find_opt t.asset_caches scope with - |None->Ok() - |Some cache->Result.map(fun()->Hashtbl.remove t.asset_caches scope)(Asset_cache.close_checked cache))) - (Ok())scopes in - Result.bind closed(fun()->Asset_cache.delete_account ~root:(asset_root t) ~account) - |> Result.map (fun () -> - let origin = Uri.to_string account.managed_sync_origin in - Hashtbl.filter_map_inplace (fun (actual_origin, user, _) retirement -> - if actual_origin=origin && user=account.user_id then None else Some retirement) t.asset_retirements; - Hashtbl.filter_map_inplace (fun ((scope : Core.graph_scope), _) () -> - if Uri.equal scope.account.managed_sync_origin account.managed_sync_origin - && scope.account.user_id=account.user_id then None else Some ()) t.asset_leases) -;; - -let retire_account t ~account = - if t.closed then Error "Sync stopped before account retirement." else - let secrets=t.dependencies.secrets.delete_account_secrets - ~managed_sync_origin:account.Core.managed_sync_origin ~user_id:account.user_id in - Result.bind secrets(fun()->delete_account_assets t account - |> Result.map_error(fun _->"Account asset cache still has a physical cleanup obligation.")) -;; - -let catalog_root local_store = - Filename.concat - local_store.application_support_directory - "logseq-db-worker/sync-catalogs" -;; - -let cache_path local_store (account : Core.account_scope) = - let identity = account.user_id ^ "\000" ^ Uri.to_string account.managed_sync_origin in - let name = Digestif.SHA256.digest_string identity |> Digestif.SHA256.to_hex in - Filename.concat (catalog_root local_store) (name ^ ".json") -;; - -let load_catalog local_store account = - let path = cache_path local_store account in - if not (Sys.file_exists path) - then Ok None - else ( - try - let channel = open_in_bin path in - let source = - Fun.protect - ~finally:(fun () -> close_in_noerr channel) - (fun () -> really_input_string channel (in_channel_length channel)) - in - Core.decode_catalog_cache source - |> Result.map Option.some - |> Result.map_error (fun source -> Core.Effect_failed - Failure_diagnostic.(of_source ~operation:"loadCatalog" ~reason:"parseFailed" source |> encode)) - with - | Sys_error _ as error -> Error (Core.Effect_failed - Failure_diagnostic.(of_source ~operation:"loadCatalog" ~reason:"resourceFailed" - (Printexc.to_string error) |> encode))) -;; - -let ensure_directory path = - try - if Sys.file_exists path - then - if Sys.is_directory path - then Ok () - else Error (Core.Effect_failed Failure_diagnostic.( - create ~operation:"prepareCatalog" ~reason:"invalidPath" - ~source:"catalog directory is not a directory" () |> encode)) - else ( - Unix.mkdir path 0o700; - Ok ()) - with - | (Unix.Unix_error _ | Sys_error _) as error -> - Error (Core.Effect_failed Failure_diagnostic.( - of_source ~operation:"prepareCatalog" ~reason:"resourceFailed" - (Printexc.to_string error) |> encode)) -;; - -let save_catalog local_store account cache = - if not (String.equal account.Core.user_id (Core.catalog_cache_user_id cache)) - then Error (Core.Effect_failed "catalog cache owner does not match its account scope") - else ( - let worker_root = - Filename.concat local_store.application_support_directory "logseq-db-worker" - in - Result.bind (ensure_directory worker_root) (fun () -> - Result.bind - (ensure_directory (catalog_root local_store)) - (fun () -> - let path = cache_path local_store account in - try - let channel = open_out_bin path in - (try - output_string channel (Core.encode_catalog_cache cache); - close_out channel; - Ok () - with exn -> close_out_noerr channel; raise exn) - with - | Sys_error _ as error -> Error (Core.Effect_failed Failure_diagnostic.( - of_source ~operation:"saveCatalog" ~reason:"persistenceFailed" - (Printexc.to_string error) |> encode))))) -;; - -let effect_error message = Error (Core.Effect_failed message) - -let successful_response request (response : Http_eio.response) = - if response.status < 200 || response.status >= 300 - then ( - observe_unlock_failure Http_status; - effect_error - (Printf.sprintf "sync HTTP request failed with status %d" response.status)) - else ( - match Http.validate_response_content_type request response.headers with - | Error message -> - observe_unlock_failure Decode; - effect_error message - | Ok () -> Ok response) -;; - -let perform_authenticated_http t account make_request = - authenticated_operation t.dependencies.id_token_provider ~account ~perform:(fun token -> - let request = make_request token in - match t.dependencies.transport.perform_http ~sw:t.sw request with - | Error message -> - (* Http_eio validates Content-Type before returning a response. Its two - producer-owned codes retain the decode boundary without logging the - arbitrary media type suffix. Numeric status is no longer available. *) - observe_unlock_failure - (if message = "sync HTTP response is missing Content-Type" - || String.starts_with ~prefix:"sync HTTP response has unsupported Content-Type " message - then Decode else Transport); - Error (Request_failed message) - | Ok response when response.Http_eio.status = 401 -> - observe_unlock_http_status response.status; - observe_unlock_failure Http_status; - Error Unauthorized - | Ok response when response.Http_eio.status = 403 -> - observe_unlock_http_status response.status; - observe_unlock_failure Http_status; - Error Forbidden - | Ok response -> - observe_unlock_http_status response.status; - successful_response request response - |> Result.map_error (function - | Core.Effect_failed message | Crypto_failed (_, message) -> - Request_failed message)) - |> Result.map_error (fun message -> Core.Effect_failed message) -;; - -let key t handle = - let id = Core.graph_key_handle_id handle in - match Hashtbl.find_opt t.keys id with - | Some entry when entry.scope = Core.graph_key_handle_scope handle -> - Ok (Bytes.to_string entry.bytes) - | Some _ | None -> effect_error "graph key handle is unavailable or out of scope" -;; - -let decrypt_protected_value t handle source = - match key t handle with - | Error (Core.Effect_failed message | Crypto_failed (_, message)) -> Error message - | Ok graph_key -> - let crypto : E2ee.crypto = - { decrypt_aes_gcm = t.dependencies.crypto.decrypt_aes_gcm } - in - Result.bind (E2ee.decrypt_value ~crypto ~graph_key source) (function - | Transit_core.Json.String plaintext -> Ok plaintext - | _ -> Error "decrypted protected value must be a string") -;; - -let encrypt_protected_values t handle plaintexts = - match key t handle with - | Error (Core.Effect_failed message | Crypto_failed (_, message)) -> Error message - | Ok graph_key -> - let rec encrypt encrypted = function - | [] -> Ok (List.rev encrypted) - | plaintext :: rest -> - (match t.dependencies.crypto.encrypt_aes_gcm ~key:graph_key ~plaintext with - | Ok value -> encrypt (value :: encrypted) rest - | Error source -> Error Failure_diagnostic.( - of_source ~operation:"encryptValue" ~reason:"cryptoInvalid" source |> encode)) - in - encrypt [] plaintexts -;; - -let store_key t ticket scope plaintext = - let id = "graph-key-" ^ Core.effect_id_to_string (Core.effect_ticket_id ticket) in - Hashtbl.replace t.keys id { scope; bytes = Bytes.of_string plaintext }; - Core.graph_key_handle ~id ~scope -;; - -let map_error result = Result.map_error (fun message -> Core.Effect_failed message) result -let run_secret_action t action = Eio.Mutex.use_rw ~protect:true t.secret_lock action - -let unlock_log_count = Atomic.make 0 - -let unlock_log format = - Printf.ksprintf (fun fields -> - if Sys.getenv_opt "JOURNAL_UNLOCK_DIAGNOSTICS" = Some "1" then ( - let count = Atomic.fetch_and_add unlock_log_count 1 in - try - if count < 256 then Printf.eprintf "[journal-unlock] source=runner %s\n%!" fields - else if count = 256 then Printf.eprintf "[journal-unlock] source=runner limit=reached\n%!" - with _ -> ())) format -;; - -(* Opt-in structural metadata only. Counts are top-level field occurrences; - no arbitrary names or values can reach the logger. The independent64KiB - observation cap avoids copying/parsing large responses for diagnostics. *) -let observe_graph_key_response_shape body = - if Sys.getenv_opt "JOURNAL_UNLOCK_DIAGNOSTICS" = Some "1" - && Atomic.get unlock_log_count <= 256 - then Option.iter (fun trace -> - let kind, known, unknown, present, base64_kind = - if String.length body > 65_536 then "unavailable", "unavailable", "unavailable", "unavailable", "unavailable" - else try - let json = Yojson.Safe.from_string body in - let value_kind = function - | `String _ -> "string" | `Null -> "null" | `Bool _ -> "boolean" - | `Int _ | `Intlit _ | `Float _ -> "number" - | `Assoc _ -> "object" | `List _ -> "array" in - let base64_kind = match json with - | `Assoc fields -> - (match List.assoc_opt "encrypted-aes-key-base64" fields with - | None -> "absent" | Some value -> value_kind value) - | _ -> "absent" in - let kind, known, unknown = match json with - | `Assoc fields -> - let known, unknown = List.fold_left (fun (known, unknown) (name, _) -> - if String.equal name "encrypted-aes-key" - || String.equal name "encrypted-aes-key-base64" then known+1, unknown - else known, unknown+1) (0, 0) fields in - "object", known, unknown - | `List _ -> "array", 0, 0 - | `String _ -> "string", 0, 0 - | `Null -> "null", 0, 0 - | `Bool _ -> "boolean", 0, 0 - | `Int _ | `Intlit _ | `Float _ -> "number", 0, 0 in - let present = match json with - | `Assoc fields -> List.mem_assoc "encrypted-aes-key" fields - | _ -> false in - kind, string_of_int known, string_of_int unknown, string_of_bool present, base64_kind - with _ -> "invalid", "unavailable", "unavailable", "unavailable", "unavailable" - in - unlock_log "op=%s request=fetch-e2ee-graph-key phase=response-shape json_kind=%s known_fields=%s unknown_fields=%s encrypted_key_present=%s base64_field_kind=%s" - trace.operation_id kind known unknown present base64_kind) (Eio.Fiber.get unlock_failure_trace) -;; - -let execute_request - : type a. - t -> a Core.effect_ticket -> a Core.runner_request -> (a, Core.effect_error) result - = - fun t ticket request -> - match request with - | Core.Load_catalog account -> load_catalog t.dependencies.local_store account - | Save_catalog { account; cache } -> - save_catalog t.dependencies.local_store account cache - | Fetch_catalog account -> - Result.bind - (perform_authenticated_http t account (fun token -> - Http.catalog ~base_url:account.managed_sync_origin ~token)) - (fun response -> - Catalog.decode response.body - |> Result.map_error (fun message -> - observe_unlock_failure Decode; - Core.Effect_failed message)) - | Fetch_snapshot_baseline graph -> - Result.map - (fun response -> response.Http_eio.body) - (perform_authenticated_http t graph.account (fun token -> - Http.snapshot_baseline - ~base_url:graph.account.managed_sync_origin - ~graph_id:graph.graph_id - ~token)) - | Fetch_snapshot_metadata graph -> - Result.map - (fun response -> response.Http_eio.body) - (perform_authenticated_http t graph.account (fun token -> - Http.snapshot_metadata - ~base_url:graph.account.managed_sync_origin - ~graph_id:graph.graph_id - ~token)) - | Fetch_e2ee_graph_key graph -> - Result.bind - (perform_authenticated_http t graph.account (fun token -> - Http.e2ee_graph_key - ~base_url:graph.account.managed_sync_origin - ~graph_id:graph.graph_id - ~token)) - (fun response -> - observe_graph_key_response_shape response.Http_eio.body; - E2ee.graph_key_response response.Http_eio.body - |> Result.map_error (fun message -> - observe_unlock_failure Decode; - Core.Effect_failed message)) - | Fetch_e2ee_user_keys account -> - Result.bind - (perform_authenticated_http t account (fun token -> - Http.e2ee_user_keys ~base_url:account.managed_sync_origin ~token)) - (fun response -> - E2ee.user_keys_response response.Http_eio.body - |> Result.map_error (fun message -> - observe_unlock_failure Decode; - Core.Effect_failed message)) - | Download_snapshot download -> - let staging = t.dependencies.artifact_store.staging_directory in - let ensure_staging () = - if Sys.file_exists staging - then existing_directory staging - else ( - try - Unix.mkdir staging 0o700; - true - with - | Unix.Unix_error _ -> false) - in - if not (ensure_staging ()) - then effect_error "snapshot staging directory is unavailable" - else ( - let id = Core.effect_id_to_string (Core.effect_ticket_id ticket) in - let raw = Filename.concat staging ("snapshot-" ^ id ^ ".download") in - let destination = Filename.concat staging ("snapshot-" ^ id ^ ".artifact") in - let temporary_paths = - [ Filename.concat staging ("snapshot-" ^ id ^ ".gzip-1") - ; Filename.concat staging ("snapshot-" ^ id ^ ".gzip-2") - ] - in - let base_url = (Core.snapshot_download_scope download).account.managed_sync_origin in - let bearer_authorized = Http.same_origin base_url (Core.snapshot_download_uri download) in - let download_once token = - let request = Http.artifact ~base_url ~uri:(Core.snapshot_download_uri download) ~token in - let response = - t.dependencies.transport.download - ~sw:t.sw - ~request - ~destination:raw - ~maximum_bytes:(Core.snapshot_download_maximum_bytes download) - ~on_progress:(fun progress -> - if not t.closed - then - t.post - (Core.Snapshot_download_progress - { graph_id = (Core.snapshot_download_scope download).graph_id - ; received_bytes = Int64.of_int progress.received_bytes - ; total_bytes = Option.map Int64.of_int progress.total_bytes - })) - |> Result.map_error (fun message -> Core.Effect_failed message) - in - Result.map (fun response -> request, response) response - in - let response = - let perform token = - match download_once token with - | Error (Core.Effect_failed message | Crypto_failed (_, message)) -> - Error (Request_failed message) - | Ok (_, response) when bearer_authorized && response.Http_eio.status = 401 -> - Error Unauthorized - | Ok (_, response) when response.Http_eio.status = 403 -> Error Forbidden - | Ok (request, response) -> - successful_response request response - |> Result.map_error (function - | Core.Effect_failed message | Crypto_failed (_, message) -> - Request_failed message) - in - (if bearer_authorized - then - authenticated_operation - t.dependencies.id_token_provider - ~account:(Core.snapshot_download_scope download).account - ~perform:(fun token -> perform (Some token)) - else Result.map_error authenticated_failure (perform None)) - |> Result.map_error (fun message -> Core.Effect_failed message) - in - Result.bind response (fun response -> - Result.bind (Bootstrap.artifact_row_count response.headers) (fun expected_rows -> - Result.bind - (Bootstrap.peel_gzip_layers - ~decompress_gzip:Artifact_decoder.decompress_gzip - ~maximum_bytes:(Core.snapshot_download_maximum_bytes download) - ~source:raw - ~destination - ~temporary_paths) - (fun () -> - Bootstrap.cleanup (raw :: temporary_paths); - Core.stage_snapshot_download download ~path:destination ~expected_rows)) - |> Result.map_error (fun message -> Core.Effect_failed message))) - | Load_and_unlock_graph_key scope -> - run_secret_action t (fun () -> - let account = scope.account in - match - t.dependencies.secrets.load_wrapped_graph_key - ~managed_sync_origin:account.managed_sync_origin - ~user_id:account.user_id - ~graph_id:scope.graph_id - with - | Error - (Wrapped_graph_key_unavailable message | Local_private_key_unavailable message) - -> effect_error message - | Ok encrypted_graph_key -> - t.dependencies.secrets.unlock_graph_key - ~managed_sync_origin:account.managed_sync_origin - ~user_id:account.user_id - ~encrypted_graph_key - |> Result.map (store_key t ticket scope) - |> map_error) - | Fetch_and_unlock_graph_key request -> - run_secret_action t (fun () -> - let account = request.scope.account in - Result.bind - (t.dependencies.secrets.unlock_graph_key - ~managed_sync_origin:account.managed_sync_origin - ~user_id:account.user_id - ~encrypted_graph_key:request.encrypted_graph_key) - (fun key -> - Result.map - (fun () -> store_key t ticket request.scope key) - (t.dependencies.secrets.verify_and_save_wrapped_graph_key - ~managed_sync_origin:account.managed_sync_origin - ~user_id:account.user_id - ~graph_id:request.scope.graph_id - ~encrypted_graph_key:request.encrypted_graph_key)) - |> map_error) - | Unlock_private_key request -> - run_secret_action t (fun () -> - let account = request.scope in - t.dependencies.secrets.unlock_private_key - ~managed_sync_origin:account.managed_sync_origin - ~user_id:account.user_id - ~password:request.password - ~private_key_package:request.private_key_package - |> map_error) - | Delete_account_secrets account -> - run_secret_action t (fun () -> - retire_account t ~account |> Result.map_error(fun message->Core.Effect_failed message)) - | Encrypt_protected_values request -> - Result.bind - (Result.map_error - (function - | Core.Effect_failed message | Crypto_failed (_, message) -> - Core.Crypto_failed (Invalid_key_material, message)) - (key t request.key)) - (fun key -> - let rec encrypt acc = function - | [] -> Ok (List.rev acc) - | plaintext :: rest -> - (match t.dependencies.crypto.encrypt_aes_gcm ~key ~plaintext with - | Ok encrypted -> encrypt (encrypted :: acc) rest - | Error message -> - Error (Core.Crypto_failed (Crypto_provider_unavailable, message))) - in - encrypt [] request.plaintexts) - | Decrypt_protected_values request -> - Result.bind - (Result.map_error - (function - | Core.Effect_failed message | Crypto_failed (_, message) -> - Core.Crypto_failed (Invalid_key_material, message)) - (key t request.key)) - (fun key -> - let rec decrypt acc = function - | [] -> Ok (List.rev acc) - | (iv, ciphertext) :: rest -> - (match t.dependencies.crypto.decrypt_aes_gcm ~key ~iv ~ciphertext with - | Ok plaintext -> decrypt (plaintext :: acc) rest - | Error message -> - Error (Core.Crypto_failed (Crypto_provider_unavailable, message))) - in - decrypt [] request.protected_values) -;; - -let field_matches expected actual = - match expected with - | None -> true - | Some expected -> Some expected = actual -;; - -let scope_matches (expected : Core.effect_scope) (actual : Core.effect_scope) = - field_matches expected.Core.account_generation actual.Core.account_generation - && field_matches expected.graph_generation actual.graph_generation - && field_matches expected.connection_generation actual.connection_generation - && field_matches expected.presentation_generation actual.presentation_generation - && field_matches expected.lifecycle_generation actual.lifecycle_generation -;; - -let zeroize bytes = Bytes.fill bytes 0 (Bytes.length bytes) '\000' - -let retire_websockets ?(remove = true) t matches retire = - let selected = - Hashtbl.fold - (fun scope websocket selected -> - if matches scope then (scope, websocket) :: selected else selected) - t.websockets - [] - in - if remove then List.iter (fun (key, _) -> Hashtbl.remove t.websockets key) selected; - List.iter (fun (_, websocket) -> retire websocket) selected -;; - -let retire_websocket_owner ?(abort = false) t (scope:Core.connection_scope) = - match Hashtbl.find_opt t.websocket_owners scope.Core.graph with - | Some owner when owner.physical_scope = scope -> - if abort - then ( - owner.operation.cancelled <- true; - owner.operation.cancel ()); - if not owner.retirement_requested - then ( - owner.retirement_requested <- true; - (match !(owner.pending) with - | Some handle -> t.dependencies.transport.close_websocket handle - | None -> - owner.operation.cancelled <- true; - owner.operation.cancel ()); - (* submit runs on the event consumer. Joining here would deadlock a callback - waiting to publish into its full sink. The waiter is never a callback. *) - fork_general t ~scope:({(Core.effect_scope_of_graph scope.graph) with connection_generation=Some scope.connection_generation}) (fun () -> - (try Eio.Promise.await owner.finished with - | Eio.Cancel.Cancelled _ -> - Eio.Cancel.protect (fun () -> Eio.Promise.await owner.finished)); - (match Hashtbl.find_opt t.websocket_owners scope.graph with - | Some current when current == owner -> Hashtbl.remove t.websocket_owners scope.graph - | _ -> ()); - t.retired_websockets <- scope :: List.take 31 (List.filter ((<>) scope) t.retired_websockets); - if not t.closed then t.post (Core.Websocket_retired scope))) - | _ when List.mem scope t.retired_websockets -> () - | _ -> - fork_general t ~scope:({(Core.effect_scope_of_graph scope.graph) with connection_generation=Some scope.connection_generation}) (fun () -> - if not t.closed then t.post (Core.Websocket_retired scope)) -;; - -let cancel_scope ?(network_only = false) t scope = - Hashtbl.iter - (fun _ (operation : operation) -> - if scope_matches scope operation.scope && (not network_only || operation.network) - then ( - operation.cancelled <- true; - operation.cancel ())) - t.operations; - if not network_only then Hashtbl.filter_map_inplace - (fun _ (entry : key_entry) -> - let entry_scope = Core.effect_scope_of_graph entry.scope in - if scope_matches scope entry_scope - then ( - zeroize entry.bytes; - None) - else Some entry) - t.keys; - retire_websockets - t - (fun connection -> - scope_matches - scope - { (Core.effect_scope_of_graph connection.Core.graph) with - connection_generation = Some connection.connection_generation - }) - t.dependencies.transport.abort_websocket; - Hashtbl.to_seq_values t.websocket_owners |> List.of_seq |> List.iter (fun owner -> - if scope_matches scope - { (Core.effect_scope_of_graph owner.physical_scope.graph) with - connection_generation=Some owner.physical_scope.connection_generation } - then retire_websocket_owner ~abort:true t owner.physical_scope) -;; - -let unlock_request_kind : type a. a Core.runner_request -> string option = function - | Core.Fetch_catalog _ -> Some "fetch-catalog" - | Fetch_e2ee_graph_key _ -> Some "fetch-e2ee-graph-key" - | Fetch_e2ee_user_keys _ -> Some "fetch-e2ee-user-keys" - | Load_and_unlock_graph_key _ -> Some "load-and-unlock-graph-key" - | Fetch_and_unlock_graph_key _ -> Some "fetch-and-unlock-graph-key" - | Unlock_private_key _ -> Some "unlock-private-key" - | _ -> None -;; - -let unlock_result : type a. (a, Core.effect_error) result -> string = function - | Ok _ -> "ok" - | Error (Core.Crypto_failed (Invalid_key_material, _)) -> "invalid-key-material" - | Error (Crypto_failed (Crypto_provider_unavailable, _)) -> "crypto-provider-unavailable" - | Error (Effect_failed message) -> - (match message with - | "Authentication failed." -> "authentication-failed" - | "Authorization failed." -> "authorization-failed" - | "localPrivateKeyUnavailable" -> "local-private-key-unavailable" - | "wrappedGraphKeyUnavailable" -> "wrapped-graph-key-unavailable" - | "cryptoOperationFailed" -> "crypto-failed" - | _ -> - let prefix = "sync HTTP request failed with status " in - let length = String.length prefix in - if String.length message = length + 3 - && String.sub message 0 length = prefix - && String.for_all (fun c -> c >= '0' && c <= '9') (String.sub message length 3) - then "http-" ^ String.sub message length 3 - else "effect-failed") -;; - -exception Runner_cancelled - -let network_request : type a. a Core.runner_request -> bool = function - | Fetch_catalog _ | Fetch_snapshot_baseline _ | Fetch_snapshot_metadata _ - | Download_snapshot _ | Fetch_e2ee_graph_key _ | Fetch_e2ee_user_keys _ -> true - | Load_catalog _ | Save_catalog _ | Load_and_unlock_graph_key _ - | Fetch_and_unlock_graph_key _ | Unlock_private_key _ | Delete_account_secrets _ - | Encrypt_protected_values _ | Decrypt_protected_values _ -> false -;; - -let request_diagnostic_operation : type a. a Core.runner_request -> string = function - | Core.Fetch_catalog _ | Load_catalog _ -> "loadCatalog" - | Save_catalog _ -> "saveCatalog" - | Fetch_snapshot_baseline _ | Fetch_snapshot_metadata _ | Download_snapshot _ -> "decodeSnapshot" - | Delete_account_secrets _ -> "resetAccount" - | Encrypt_protected_values _ -> "encryptValue" - | Decrypt_protected_values _ -> "decryptValue" - | Fetch_e2ee_graph_key _ | Fetch_e2ee_user_keys _ | Load_and_unlock_graph_key _ - | Fetch_and_unlock_graph_key _ | Unlock_private_key _ -> "platformCrypto" -;; - -let submit_request : type a. t -> a Core.effect_ticket -> a Core.runner_request -> unit = - fun t ticket request -> - let id = Core.effect_ticket_id ticket |> Core.effect_id_to_string in - let diagnostic_kind = unlock_request_kind request in - let failure_trace = - if Sys.getenv_opt "JOURNAL_UNLOCK_DIAGNOSTICS" = Some "1" && Option.is_some diagnostic_kind - then Some { operation_id = id; category = None; http_status = None } - else None - in - let began = Unix.gettimeofday () in - Option.iter (fun kind -> unlock_log "op=%s request=%s phase=begin" id kind) diagnostic_kind; - let cancelled, resolve_cancelled = Eio.Promise.create () in - let operation = - { network = network_request request - ; scope = Core.effect_ticket_scope ticket - ; asset_identity = None - ; cancelled = false - ; cancel = (fun () -> ignore (Eio.Promise.try_resolve resolve_cancelled () : bool)) - } - in - Hashtbl.replace t.operations id operation; - fork_general t ~scope:(Core.effect_ticket_scope ticket) (fun () -> - let result = - try - Some - (Eio.Fiber.first - (fun () -> - if operation.cancelled || t.closed then raise Runner_cancelled; - let run () = Platform_crypto.with_diagnostic_operation id (fun () -> execute_request t ticket request) in - match failure_trace with - | None -> run () - | Some trace -> Eio.Fiber.with_binding unlock_failure_trace trace run) - (fun () -> - Eio.Promise.await cancelled; - raise Runner_cancelled)) - with - | Runner_cancelled -> None - | Eio.Cancel.Cancelled _ as error -> raise error - | error -> Some (Error (Core.Effect_failed Failure_diagnostic.( - of_source ~operation:(request_diagnostic_operation request) ~reason:"unavailable" - (Printexc.to_string error) |> encode))) - in - Hashtbl.remove t.operations id; - Option.iter (fun kind -> - let outcome = match result with - | None -> "cancelled" - | Some _ when t.closed || operation.cancelled -> "retired" - | Some result -> unlock_result result in - let category = match result with - | None -> "cancelled" - | Some _ when t.closed || operation.cancelled -> "retired" - | Some (Ok _) -> "none" - | Some (Error error) -> - let observed = Option.bind failure_trace (fun trace -> trace.category) in - let observed = match observed, error with - | Some _ as category, _ -> category - | None, Core.Crypto_failed _ -> Some Crypto - | None, Effect_failed _ when kind = "unlock-private-key" - || kind = "fetch-and-unlock-graph-key" || kind = "load-and-unlock-graph-key" -> Some Crypto - | None, _ -> None in - (match observed with - | Some Token -> "token" - | Some Transport -> "transport" - | Some Http_status -> "http" - | Some Decode -> "decode" - | Some Crypto -> "crypto" - | None -> "unknown") in - let http_status = Option.bind failure_trace (fun trace -> trace.http_status) - |> Option.fold ~none:"none" ~some:string_of_int in - unlock_log "op=%s request=%s phase=end result=%s failure_category=%s http_status=%s elapsed_ms=%.0f" - id kind outcome category http_status (Float.max 0. ((Unix.gettimeofday () -. began) *. 1000.))) diagnostic_kind; - match result with - | Some result when (not t.closed) && not operation.cancelled -> - t.post (Core.Runner_completed (Core.Completion (ticket, result))) - | Some _ | None -> ()) -;; - -let submit_nonasset t instruction = - if not t.closed - then ( - match instruction with - | Core.Asset_io _ | Core.Protected_io _ | Core.Await_io _ -> - invalid_arg "Effect requires its dedicated interpreter" - | Core.Request (ticket, request) -> submit_request t ticket request - | Cancel_effects scope -> cancel_scope t scope - | Cancel_network_effects scope -> cancel_scope ~network_only:true t scope - | Schedule_timer request -> - let id = Core.runner_effect_diagnostic instruction in - let cancelled, resolve_cancelled = Eio.Promise.create () in - let operation = - { network = true - ; scope = request.scope - ; asset_identity = None - ; cancelled = false - ; cancel = - (fun () -> ignore (Eio.Promise.try_resolve resolve_cancelled () : bool)) - } - in - Hashtbl.replace t.operations id operation; - fork_general t ~scope:(Core.runner_effect_scope instruction) (fun () -> - (try - Eio.Fiber.first - (fun () -> t.dependencies.runtime.sleep request.delay_seconds) - (fun () -> - Eio.Promise.await cancelled; - raise Runner_cancelled) - with - | Runner_cancelled -> ()); - Hashtbl.remove t.operations id; - if (not t.closed) && not operation.cancelled - then t.post (Core.Timer_elapsed request.id)) - | Start_websocket request -> - (match Hashtbl.find_opt t.websocket_owners request.scope.graph with - | Some owner when owner.physical_scope = request.scope -> () - | Some _ -> - fork_general t ~scope:(Core.runner_effect_scope instruction) (fun () -> - if not t.closed - then - t.post - (Core.Websocket_closed - ( request.scope - , Core.Protocol_failure "Previous WebSocket owner has not retired" ))) - | None -> - let scope=request.scope in - let key=Printf.sprintf "websocket:%d:%s:%d:%d" - scope.graph.account.account_generation - (Logseq_db_types.Graph_types.Uuid.to_string scope.graph.graph_id) - scope.graph.graph_generation scope.connection_generation in - let cancelled, resolve_cancelled = Eio.Promise.create () in - let finished, resolve_finished = Eio.Promise.create () in - let pending = ref None in - let operation = - { network = true - ; scope = Core.runner_effect_scope instruction - ; asset_identity = None - ; cancelled = false - ; cancel = - (fun () -> - ignore (Eio.Promise.try_resolve resolve_cancelled () : bool); - Option.iter t.dependencies.transport.abort_websocket !pending) - } - in - let owner = - { physical_scope = request.scope - ; operation - ; pending - ; finished - ; retirement_requested = false - } - in - Hashtbl.replace t.websocket_owners request.scope.graph owner; - Hashtbl.replace t.operations key operation; - fork_general t ~scope:(Core.runner_effect_scope instruction) (fun () -> - let registered = ref false in - Fun.protect - ~finally:(fun () -> - Hashtbl.remove t.operations key; - if not !registered - then Option.iter t.dependencies.transport.abort_websocket !pending; - Eio.Cancel.protect (fun () -> - Option.iter t.dependencies.transport.await_websocket_retired !pending); - ignore (Eio.Promise.try_resolve resolve_finished () : bool)) - (fun () -> - let connect token = - let result = - t.dependencies.transport.connect_websocket - ~sw:t.sw - ~uri:request.uri - ~token - ~maximum_frame_bytes:Logseq_db_types.Limits.maximum_response_bytes - ~on_message:(fun payload -> - if (not t.closed) && not operation.cancelled - then ( - match Sync_protocol.decode_server_message payload with - | Ok message -> - t.post (Core.Websocket_message (request.scope, message)) - | Error error -> - t.post (Core.Websocket_protocol_error (request.scope, error)))) - ~on_close:(fun termination -> - Hashtbl.remove t.websockets request.scope; - if (not t.closed) && not operation.cancelled - then - Eio.Fiber.first - (fun () -> - t.post (Core.Websocket_closed (request.scope, termination))) - (fun () -> Eio.Promise.await cancelled)) - in - (match result with - | Ok handle -> pending := Some handle - | Error _ -> ()); - result - in - let termination = function - | Websocket_eio.Unauthorized -> - Core.Authentication_failure "Authentication failed." - | Websocket_eio.Forbidden -> - Core.Authentication_failure "Authorization failed." - | Websocket_eio.Authentication_failed message -> - Core.Authentication_failure message - | Websocket_eio.Protocol_failed message -> Core.Protocol_failure message - | Websocket_eio.Connection_failed { message; attention } -> - Core.Recoverable_close { message = Some message; attention } - in - let connect_with_token () = - match - t.dependencies.id_token_provider.acquire request.scope.graph.account - with - | Error message -> Error (Core.Authentication_failure message) - | Ok token -> - (match connect token with - | Error Websocket_eio.Unauthorized -> - t.dependencies.id_token_provider.invalidate - request.scope.graph.account - ~token; - (match - t.dependencies.id_token_provider.acquire - request.scope.graph.account - with - | Error message -> Error (Core.Authentication_failure message) - | Ok refreshed -> Result.map_error termination (connect refreshed)) - | result -> Result.map_error termination result) - in - try - Eio.Fiber.first - (fun () -> - let result = connect_with_token () in - if (not t.closed) && not operation.cancelled - then ( - match result with - | Error termination -> - t.post (Core.Websocket_closed (request.scope, termination)) - | Ok handle -> - (match - t.dependencies.transport.activate_websocket - handle - ~on_open:(fun () -> - Hashtbl.replace t.websockets request.scope handle; - registered := true; - t.post (Core.Websocket_opened request.scope)) - with - | Ok () -> - t.dependencies.transport.await_websocket_retired handle - | Error error -> - t.post - (Core.Websocket_closed (request.scope, termination error))))) - (fun () -> - Eio.Promise.await cancelled; - raise Runner_cancelled) - with - | Runner_cancelled -> ()))) - | Send_websocket request -> - (match Sync_protocol.encode_client_message request.message with - | Error error -> t.post (Core.Websocket_protocol_error (request.scope, error)) - | Ok payload -> - let sent = ref false in - Hashtbl.iter - (fun scope websocket -> - if scope = request.scope - then ( - sent := true; - match t.dependencies.transport.send_websocket websocket payload with - | Ok () -> () - | Error message -> - t.post - (Core.Websocket_closed - (request.scope, Core.Recoverable_close { message = Some message; attention = Websocket_eio.failure_attention websocket })))) - t.websockets; - if not !sent - then - t.post - (Core.Websocket_closed - (request.scope, Core.Recoverable_close { message = Some "WebSocket is unavailable"; attention = Requires_attention }))) - | Close_websocket scope -> retire_websocket_owner t scope) -;; - -let shutdown t = - if not t.closed - then ( - t.closed <- true; - Hashtbl.iter (fun _ cache -> Asset_cache.close cache) t.asset_caches; - Hashtbl.clear t.asset_caches; - let all = - Core. - { account_generation = None - ; graph_generation = None - ; connection_generation = None - ; presentation_generation = None - ; lifecycle_generation = None - } - in - cancel_scope t all; - Hashtbl.clear t.operations) -;; - -let asset_diagnostic_capture = Eio.Fiber.create_key () - -let report_asset_failure ~reason source = - Option.iter (fun report -> report ~reason source) - (Eio.Fiber.get asset_diagnostic_capture) -;; - -let asset_error_source = function - | Asset_cache.Invalid source | Io source -> source - | Full -> "Asset staging storage is full" - | Stale -> "Asset staging scope is stale" - | Checksum_mismatch -> "Asset staging checksum mismatch" -;; - -let asset_diagnostic_operation = function - | Core.Put_asset_file _ -> "assetUpload" - | Core.Fetch_asset _ -> "assetDownload" - | Core.Release_asset_file _ | Release_staged_file _ | Close_asset_scope - | Prepare_graph_asset_deletion _ | Commit_graph_asset_deletion _ - | Cancel_graph_asset_deletion _ | Delete_account_assets | Cancel_asset_operation _ -> "assetRelease" - | _ -> "assetPreview" -;; - -let asset_exception (request : Core.asset_io_request) error = - Core.Asset_invalid_content Failure_diagnostic.( - of_source ~operation:(asset_diagnostic_operation request.action) ~reason:"invalidContent" - (Printexc.to_string error) |> encode) -;; - -let with_asset_diagnostics t (request : Core.asset_io_request) ~current work = - let operation = asset_diagnostic_operation request.action in - let captured = ref None in - let report ~reason:reason_name source = - captured := Some (Failure_diagnostic.of_source ~operation ~reason:reason_name source) in - let result = Eio.Fiber.with_binding asset_diagnostic_capture report (fun () -> - try work () with - | (Runner_cancelled | Eio.Cancel.Cancelled _) as error -> raise error - | error -> Error (asset_exception request error)) in - match result with - | Error Core.Asset_cancelled | Ok _ -> result - | Error failure -> - let fallback = match failure with - | Core.Asset_network -> "networkFailed", "" - | Asset_not_found -> "notFound", "" - | Asset_checksum_mismatch -> "checksumMismatch", "" - | Asset_authentication -> "authenticationFailed", "" - | Asset_locked -> "locked", "" - | Asset_storage_full -> "storageFull", "" - | Asset_missing_source -> "missingSource", "" - | Asset_size_rejected -> "sizeRejected", "" - | Asset_revoked_access -> "revokedAccess", "" - | Asset_invalid_content source -> "invalidContent", source - | Asset_cancelled -> "cancelled", "" - | Asset_with_cause (_, cause) -> Failure_diagnostic.reason cause, Failure_diagnostic.encode cause in - let reason_name, source = fallback in - let detail = Option.value !captured ~default:( - Failure_diagnostic.of_source ~operation ~reason:reason_name source) in - (* Diagnostics must never change an operation's completion or cleanup. *) - if not t.closed && current () then - (try t.on_asset_diagnostic ~scope:request.context.scope ~operation - ~request_operation:request.operation detail with _ -> ()); - Error (Core.asset_failure_with_cause failure detail) -;; - -type asset_encryption = - | Plaintext - | Encrypted of Core.graph_key_handle option - -let asset_cache_failure = function - | Asset_cache.Full -> Core.Asset_storage_full - | Checksum_mismatch -> Core.Asset_checksum_mismatch - | Stale -> Core.Asset_invalid_content "Asset scope is no longer current" - | Invalid message | Io message -> Core.Asset_invalid_content message -;; - -let asset_decode_failure error = - let reason_name, source = match error with - | Asset_codec.Invalid_limit -> "sizeRejected", "Invalid plaintext size limit" - | Wire_too_large -> "sizeRejected", "Asset exceeds wire size limit" - | Plaintext_too_large -> "sizeRejected", "Asset exceeds plaintext size limit" - | Malformed_envelope -> "invalidContent", "Invalid asset AES-GCM envelope" - | Malformed_transit _ -> "parseFailed", "Asset Transit payload is invalid" - | Decryption_failed source -> "cryptoInvalid", source - | Checksum_mismatch -> "checksumMismatch", "Asset checksum mismatch" in - report_asset_failure ~reason:reason_name source; - match error with - | Asset_codec.Checksum_mismatch -> Core.Asset_checksum_mismatch - | _ -> Core.Asset_invalid_content Failure_diagnostic.( - of_source ~operation:"assetDownload" ~reason:reason_name source |> encode) -;; - -let asset_graph_key t scope = function - | Plaintext -> Ok None - | Encrypted None -> - report_asset_failure ~reason:"locked" "Asset graph key handle is missing"; - Error Core.Asset_locked - | Encrypted (Some handle) -> - if Core.graph_key_handle_scope handle <> scope - then ( - report_asset_failure ~reason:"locked" "Asset graph key scope does not match"; - Error Core.Asset_locked) - else - key t handle - |> Result.map Option.some - |> Result.map_error (fun error -> - let source = match error with Core.Effect_failed source | Crypto_failed (_, source) -> source in - report_asset_failure ~reason:"locked" source; - Core.Asset_locked) -;; - -let with_asset_slot slots work = - Eio.Semaphore.acquire slots; - Fun.protect ~finally:(fun () -> Eio.Semaphore.release slots) work -;; - -let asset_wire_bytes ~maximum_plaintext_bytes encrypted = - if encrypted - then (4 * ((maximum_plaintext_bytes + 18) / 3)) + 128 - else maximum_plaintext_bytes -;; - -let with_byte_reservation t bytes work = - let units = (max bytes 0 + asset_byte_unit - 1) / asset_byte_unit in - if units > asset_byte_budget_units - then Error Core.Asset_size_rejected - else ( - let acquired = ref 0 in - Fun.protect - ~finally:(fun () -> - for _ = 1 to !acquired do - Eio.Semaphore.release t.asset_byte_budget - done) - (fun () -> - with_asset_slot t.asset_byte_reservation_lock (fun () -> - for _ = 1 to units do - Eio.Semaphore.acquire t.asset_byte_budget; - incr acquired - done); - work ())) -;; - -let fetch_asset_admitted - t - ~cache - ~encryption - ~maximum_plaintext_bytes - ~current - ~(scope : Core.graph_scope) - ~asset - ~(version : Logseq_db_types.Asset_descriptor.version) - = - let ( let* ) = Result.bind in - if maximum_plaintext_bytes < 0 || maximum_plaintext_bytes > 100 * 1024 * 1024 - then Error (Core.Asset_invalid_content "Invalid asset size limit") - else - let* graph_key = asset_graph_key t scope encryption in - let base_url = scope.account.managed_sync_origin in - let* () = - Http.validate_base_url base_url - |> Result.map_error (fun message -> Core.Asset_invalid_content message) - in - let path = - Printf.sprintf - "/assets/%s/%s.%s" - (Graph_types.Uuid.to_string scope.graph_id) - (Graph_types.Uuid.to_string asset) - version.file_type - in - let uri = Uri.with_path base_url path in - let maximum_response_bytes = - asset_wire_bytes ~maximum_plaintext_bytes (Option.is_some graph_key) - in - let last_failure = ref Core.Asset_authentication in - let* response = - authenticated_operation - t.dependencies.id_token_provider - ~account:scope.account - ~perform:(fun token -> - let request : Http.request = - { operation = Get - ; uri - ; headers = [ "authorization", "Bearer " ^ token ] - ; maximum_response_bytes - ; expected_content_type = Asset_binary - } - in - match t.dependencies.transport.perform_http ~sw:t.sw request with - | Error message -> - last_failure := Core.Asset_network; - Error (Request_failed message) - | Ok response when response.status = 401 -> - last_failure := Core.Asset_authentication; - Error Unauthorized - | Ok response when response.status = 403 -> - last_failure := Core.Asset_revoked_access; - Error Forbidden - | Ok response -> Ok response) - |> Result.map_error (fun cause -> - let reason = match !last_failure with - | Core.Asset_network -> "networkFailed" - | Asset_revoked_access -> "revokedAccess" - | _ -> "authenticationFailed" in - let source = match cause with - | "Authentication failed." -> "Asset HTTP status 401" - | "Authorization failed." -> "Asset HTTP status 403" - | source -> source in - report_asset_failure ~reason source; - !last_failure) - in - let* () = - match response.status with - | status when status >= 200 && status < 300 -> Ok () - | 404 -> - report_asset_failure ~reason:"notFound" "Asset HTTP status 404"; - Error Core.Asset_not_found - | (408 | 429 | 500 | 502 | 503 | 504) as status -> - report_asset_failure ~reason:"networkFailed" (Printf.sprintf "Asset HTTP status %d" status); - Error Core.Asset_network - | status -> - Error (Core.Asset_invalid_content (Printf.sprintf "Asset HTTP status %d" status)) - in - if t.closed || not (current ()) - then Error Core.Asset_cancelled - else - with_asset_slot t.asset_codec_slot (fun () -> - if t.closed || not (current ()) - then Error Core.Asset_cancelled - else ( - let crypto : Asset_codec.crypto = - { encrypt = t.dependencies.crypto.encrypt_aes_gcm - ; decrypt = t.dependencies.crypto.decrypt_aes_gcm - } - in - let* verified = - Asset_codec.decode - ~maximum_plaintext_bytes - ~crypto - ~key:graph_key - ~expected:version - response.body - |> Result.map_error asset_decode_failure - in - Asset_cache.publish - cache - ~asset - ~current:(fun () -> (not t.closed) && current ()) - ~verified - |> Result.map_error asset_cache_failure)) -;; - -let fetch_asset - t - ~cache - ~encryption - ~maximum_plaintext_bytes - ~current - ~scope - ~asset - ~version - = - with_asset_slot t.asset_download_slots (fun () -> - if t.closed || not (current ()) - then Error Core.Asset_cancelled - else ( - let encrypted = - match encryption with - | Plaintext -> false - | Encrypted _ -> true - in - let reservation = - asset_wire_bytes ~maximum_plaintext_bytes encrypted + maximum_plaintext_bytes - in - with_byte_reservation t reservation (fun () -> - fetch_asset_admitted - t - ~cache - ~encryption - ~maximum_plaintext_bytes - ~current - ~scope - ~asset - ~version))) -;; - -let close_asset_scope_checked ?except_id t scope = - Hashtbl.iter - (fun id (operation : operation) -> - if - String.starts_with ~prefix:"asset:" id - && Some id <> except_id - && Option.fold - ~none:false - ~some:(fun (actual, _) -> actual = scope) - operation.asset_identity - then ( - operation.cancelled <- true; - operation.cancel ())) - t.operations; - match Hashtbl.find_opt t.asset_caches scope with - |None->Ok() - |Some cache->Asset_cache.close_checked cache - |> Result.map(fun()->Hashtbl.remove t.asset_caches scope) -;; -let close_asset_scope ?except_id t scope=ignore(close_asset_scope_checked ?except_id t scope) -;; - -let scoped_asset_cache t scope = - match Hashtbl.find_opt t.asset_caches scope with - | Some cache -> Ok cache - | None -> - let root = asset_root t in - Asset_cache.create - ~root - ~scope - ~budget_bytes:t.dependencies.local_store.asset_cache_budget_bytes - ~maximum_file_bytes:t.dependencies.local_store.asset_maximum_file_bytes - |> Result.map (fun cache -> - Hashtbl.add t.asset_caches scope cache; - cache) -;; - -let retain_asset_file t ~scope ~handle = - if t.closed || asset_blocked t scope - then None - else - Option.bind (Hashtbl.find_opt t.asset_caches scope) (fun cache -> - Option.bind (Asset_cache.retain cache handle) (fun lease -> - match Asset_cache.path cache lease with - | Some path -> Hashtbl.replace t.asset_leases (scope, lease) (); Some (lease, path) - | None -> - Asset_cache.release cache lease; - None)) -;; - -let release_asset_file t ~scope ~handle = - let result=match Hashtbl.find_opt t.asset_caches scope with - | None -> Ok () - | Some cache -> Asset_cache.release_checked cache handle |> Result.map_error asset_cache_failure in - Result.map(fun()->Hashtbl.remove t.asset_leases(scope,handle))result -;; - -let cancel_graph_asset_operations ?except_id t (request : Core.mirror_deletion) = - Hashtbl.iter - (fun id (operation : operation) -> - match operation.asset_identity with - | Some (scope, _) - when Some id <> except_id - && scope.graph_id = request.graph_id - && scope.account.user_id = request.account.user_id - && Uri.equal - scope.account.managed_sync_origin - request.account.managed_sync_origin -> - operation.cancelled <- true; - operation.cancel () - | Some _ | None -> ()) - t.operations -;; - -let prepare_graph_asset_deletion ?except_id t (scope : Core.graph_scope) retirement ~current = - match Asset_cache.graph_deletion_terminal ~root:(asset_root t) ~scope ~retirement with - | Error error -> Error (asset_cache_failure error) - | Ok (Some `Committed) -> Ok (Core.Asset_deletion_prepared (Core.asset_deletion_reference ~scope ~retirement)) - | Ok (Some `Aborted) -> Error Core.Asset_cancelled - | Ok None -> - let request : Core.mirror_deletion = - { account = scope.account; graph_id = scope.graph_id; scope = Core.effect_scope_of_graph scope } in - cancel_graph_asset_operations ?except_id t request; - let same_target scope' = asset_target scope' = asset_target scope in - let busy () = - Hashtbl.fold (fun id (operation : operation) busy -> - busy || (Some id <> except_id && Option.fold ~none:false - ~some:(fun (scope, _) -> same_target scope) operation.asset_identity)) t.operations false - || Hashtbl.fold (fun (scope, _) () busy -> busy || same_target scope) t.asset_leases false in - let rec wait attempts = - if not (current ()) || Hashtbl.find_opt t.asset_retirements (asset_target scope) <> Some retirement - then Error Core.Asset_cancelled - else if not (busy ()) then Ok () - else if attempts = 0 then Error (Core.Asset_invalid_content "Attachment resources are still in use.") - else (t.dependencies.runtime.sleep 0.005; wait (attempts - 1)) in - let result = Result.bind (wait 1000) (fun () -> - let scopes=Hashtbl.fold(fun scope _ scopes->if same_target scope then Rrbvec.push_back scopes scope else scopes) - t.asset_caches Rrbvec.empty in - let closed=Rrbvec.fold_left(fun result scope->Result.bind result(fun()-> - close_asset_scope_checked ?except_id t scope |> Result.map_error asset_cache_failure))(Ok())scopes in - Result.bind closed(fun()->Asset_cache.prepare_graph_deletion ~root:(asset_root t) ~scope ~retirement - |> Result.map_error asset_cache_failure - |> Result.map(fun()->Core.Asset_deletion_prepared(Core.asset_deletion_reference ~scope ~retirement)))) in - (* A failed preparation may already own a durable rename. Its coordinating - journal will cancel it; keep the barrier until that exact terminal action. *) - result -;; - -let finish_graph_asset_deletion t deletion ~commit = - let scope = Core.asset_deletion_scope deletion in - let retirement = Core.asset_deletion_retirement deletion in - Asset_cache.finish_graph_deletion ~root:(asset_root t) ~deletion ~commit - |> Result.map_error asset_cache_failure - |> Result.map (fun () -> - if Hashtbl.find_opt t.asset_retirements (asset_target scope) = Some retirement - then Hashtbl.remove t.asset_retirements (asset_target scope); - Core.Asset_unit) -;; - -let read_upload_source ~maximum_plaintext_bytes source_file = - try - let input = open_in_bin source_file in - Fun.protect - ~finally:(fun () -> close_in_noerr input) - (fun () -> - let stat = Unix.fstat (Unix.descr_of_in_channel input) in - if stat.st_kind <> Unix.S_REG - then Error (Core.Asset_invalid_content "Asset upload source is not a regular file") - else if stat.st_size > maximum_plaintext_bytes - then Error Core.Asset_size_rejected - else ( - let bytes = really_input_string input stat.st_size in - match input_char input with - | _ -> Error (Core.Asset_invalid_content "Asset upload source grew during reading") - | exception End_of_file -> Ok bytes)) - with - | (Sys_error _ | Unix.Unix_error (Unix.ENOENT, _, _)) as error -> - report_asset_failure ~reason:"missingSource" (Printexc.to_string error); - Error Core.Asset_missing_source - | End_of_file -> - Error (Core.Asset_invalid_content "Asset upload source ended during reading") - | Unix.Unix_error _ as error -> - Error (Core.Asset_invalid_content Failure_diagnostic.( - of_source ~operation:"assetUpload" ~reason:"invalidContent" - (Printexc.to_string error) |> encode)) -;; - -let put_asset_admitted - t - ~(context : Core.asset_context) - ~asset - ~version - ~source_file - ~maximum_plaintext_bytes - ~current - = - let ( let* ) = Result.bind in - let valid () = (not t.closed) && current () in - if not (valid ()) - then Error Core.Asset_cancelled - else if maximum_plaintext_bytes < 0 || maximum_plaintext_bytes > 100 * 1024 * 1024 - then Error Core.Asset_size_rejected - else ( - let encryption = if context.encrypted then Encrypted context.key else Plaintext in - let* graph_key = - asset_graph_key t context.scope encryption - |> Result.map_error (fun _ -> Core.Asset_locked) - in - let* body = - with_asset_slot t.asset_codec_slot (fun () -> - if not (valid ()) - then Error Core.Asset_cancelled - else - let* plaintext = read_upload_source ~maximum_plaintext_bytes source_file in - if - not - (String.equal - (Asset_codec.checksum plaintext) - version.Logseq_db_types.Asset_descriptor.checksum) - then Error (Core.Asset_invalid_content "Asset upload checksum mismatch") - else ( - let crypto : Asset_codec.crypto = - { encrypt = t.dependencies.crypto.encrypt_aes_gcm - ; decrypt = t.dependencies.crypto.decrypt_aes_gcm - } - in - Asset_codec.encode ~maximum_plaintext_bytes ~crypto ~key:graph_key plaintext - |> Result.map_error (fun source -> - Core.Asset_invalid_content Failure_diagnostic.( - of_source ~operation:"assetUpload" ~reason:( - if source = "Asset exceeds plaintext size limit" then "sizeRejected" else "cryptoInvalid") - source |> encode)))) - in - let base_url = context.scope.account.managed_sync_origin in - let* () = - Http.validate_base_url base_url - |> Result.map_error (fun _ -> - Core.Asset_invalid_content "Asset upload base URL is invalid") - in - let path = - Printf.sprintf - "/assets/%s/%s.%s" - (Graph_types.Uuid.to_string context.scope.graph_id) - (Graph_types.Uuid.to_string asset) - version.file_type - in - let failure = ref Core.Asset_authentication in - let* response = - authenticated_operation - t.dependencies.id_token_provider - ~account:context.scope.account - ~perform:(fun token -> - if not (valid ()) - then ( - failure := Core.Asset_cancelled; - Error (Request_failed "Asset upload expired")) - else ( - let request : Http.request = - { operation = Put body - ; uri = Uri.with_path base_url path - ; headers = - [ "authorization", "Bearer " ^ token - ; "x-amz-meta-checksum", version.checksum - ; "x-amz-meta-type", version.file_type - ; "content-type", "application/octet-stream" - ] - ; maximum_response_bytes = 65536 - ; expected_content_type = Asset_binary - } - in - match t.dependencies.transport.perform_http ~sw:t.sw request with - | Error message -> - failure := Core.Asset_network; - Error (Request_failed message) - | Ok response when response.status = 401 -> - failure := Core.Asset_authentication; - Error Unauthorized - | Ok response when response.status = 403 -> - failure := Core.Asset_revoked_access; - Error Forbidden - | Ok response -> Ok response)) - |> Result.map_error (fun cause -> - if !failure <> Core.Asset_cancelled then ( - let reason = match !failure with - | Core.Asset_network -> "networkFailed" - | Asset_revoked_access -> "revokedAccess" - | _ -> "authenticationFailed" in - let source = match cause with - | "Authentication failed." -> "Asset HTTP status 401" - | "Authorization failed." -> "Asset HTTP status 403" - | source -> source in - report_asset_failure ~reason source); - !failure) - in - if not (valid ()) - then Error Core.Asset_cancelled - else ( - match response.status with - | status when status >= 200 && status < 300 -> Ok () - | 413 -> - report_asset_failure ~reason:"sizeRejected" "Asset HTTP status 413"; - Error Core.Asset_size_rejected - | (408 | 429 | 500 | 502 | 503 | 504) as status -> - report_asset_failure ~reason:"networkFailed" (Printf.sprintf "Asset HTTP status %d" status); - Error Core.Asset_network - | status -> Error (Core.Asset_invalid_content (Printf.sprintf "Asset HTTP status %d" status)))) -;; - -let put_asset - t - ~(context : Core.asset_context) - ~asset - ~version - ~source_file - ~maximum_plaintext_bytes - ~current - = - with_asset_slot t.asset_upload_slots (fun () -> - let reservation = - asset_wire_bytes - ~maximum_plaintext_bytes - (context.encrypted && Option.is_some context.key) - + maximum_plaintext_bytes - in - with_byte_reservation t reservation (fun () -> - put_asset_admitted - t - ~context - ~asset - ~version - ~source_file - ~maximum_plaintext_bytes - ~current)) -;; - -let staged_asset_path t ~scope ~file = - if t.closed - then None - else ( - match scoped_asset_cache t scope with - | Error error -> - report_asset_failure ~reason:"unavailable" (asset_error_source error); - None - | Ok cache -> Asset_cache.staged_path cache ~file) -;; - -let release_staged_asset t ~scope ~file = - match scoped_asset_cache t scope with - | Error error -> - let source = match error with - | Asset_cache.Invalid source | Io source -> source - | Full -> "Asset staging storage is full" - | Stale -> "Asset staging scope is stale" - | Checksum_mismatch -> "Asset staging checksum mismatch" in - Error Failure_diagnostic.(of_source ~operation:"assetRelease" - ~reason:"unavailable" source |> encode) - | Ok cache -> - Result.map_error - (fun error -> - let source = match error with - | Asset_cache.Invalid source | Io source -> source - | Full -> "Asset staging storage is full" - | Stale -> "Asset staging scope is stale" - | Checksum_mismatch -> "Asset staging checksum mismatch" in - Failure_diagnostic.(of_source ~operation:"assetRelease" - ~reason:"cleanupFailed" source |> encode)) - (Asset_cache.release_staged cache ~file) -;; - -let prune_staged_assets t ~scope ~keep = - if t.closed - then Error Core.Asset_cancelled - else ( - match scoped_asset_cache t scope with - | Error error -> Error (asset_cache_failure error) - | Ok cache -> - let retained = Hashtbl.create (List.length keep) in - List.iter (fun file -> Hashtbl.replace retained file ()) keep; - Asset_cache.prune_staged cache ~keep:(fun file -> Ok (Hashtbl.mem retained file)) - |> Result.map_error asset_cache_failure) -;; - -let capture_asset_jobs t (scope:Core.graph_scope) = - let same (actual:Core.graph_scope)= - Uri.equal actual.account.managed_sync_origin scope.account.managed_sync_origin - && actual.account.user_id=scope.account.user_id - && actual.account.account_generation=scope.account.account_generation - && actual.graph_id=scope.graph_id && actual.graph_generation=scope.graph_generation in - Hashtbl.fold(fun _ (owner,finished) jobs-> - if same owner then Rrbvec.push_back jobs finished else jobs)t.asset_jobs Rrbvec.empty -;; - -let execute_asset ?except_id ?(asset_snapshot=Rrbvec.empty) t (request : Core.asset_io_request) ~current = - let scope = request.context.scope in - let map_unit result = Result.map (fun () -> Core.Asset_unit) result in - let storage_result result = - Result.map_error (fun message -> Core.Asset_invalid_content message) result - in - let cache action = - Result.bind - (scoped_asset_cache t scope |> Result.map_error (fun error -> - report_asset_failure ~reason:"unavailable" (asset_error_source error); - asset_cache_failure error)) - action - in - let permitted = match request.action with - | Core.Release_asset_file _ | Await_asset_operations | Cancel_asset_operation _ | Prepare_graph_asset_deletion _ - | Commit_graph_asset_deletion _ | Cancel_graph_asset_deletion _ | Delete_account_assets -> true - | _ -> not (asset_blocked t scope) in - if not permitted then Error Core.Asset_cancelled else - match request.action with - | Core.Check_asset_cache (asset, version) -> - cache (fun cache -> - Asset_cache.lookup cache ~asset ~version - |> Result.map (fun handle -> Core.Asset_cached handle) - |> Result.map_error asset_cache_failure) - | Fetch_asset { asset; version; maximum_plaintext_bytes } -> - cache (fun cache -> - let encryption = - if request.context.encrypted then Encrypted request.context.key else Plaintext - in - fetch_asset - t - ~cache - ~encryption - ~maximum_plaintext_bytes - ~current - ~scope - ~asset - ~version - |> Result.map (fun handle -> Core.Asset_downloaded handle)) - | Stage_asset_file { operation; file_type; source_file } -> - cache (fun cache -> - Asset_cache.stage - cache - ~operation - ~file_type - ~source_file - ~pending_budget_bytes:t.dependencies.local_store.asset_pending_budget_bytes - |> Result.map (fun staged -> - Core.Asset_staged - { file = staged.Asset_cache.file - ; checksum = staged.checksum - ; size = staged.size - }) - |> Result.map_error asset_cache_failure) - | Put_asset_file { asset; version; file; maximum_plaintext_bytes } -> - (match staged_asset_path t ~scope ~file with - | None -> Error Core.Asset_missing_source - | Some source_file -> - put_asset - t - ~context:request.context - ~asset - ~version - ~source_file - ~maximum_plaintext_bytes - ~current - |> map_unit) - | Publish_staged_asset { asset; version; file } -> - let ( let* ) = Result.bind in - cache (fun cache -> - match Asset_cache.retain_staged cache ~file with - | None -> Error Core.Asset_missing_source - | Some lease -> - Fun.protect - ~finally:(fun () -> Asset_cache.release cache lease) - (fun () -> - match Asset_cache.staged_path cache ~file with - | None -> Error Core.Asset_missing_source - | Some source_file -> - let* plaintext = read_upload_source - ~maximum_plaintext_bytes:t.dependencies.local_store.asset_maximum_file_bytes - source_file in - let crypto : Asset_codec.crypto = - { encrypt = t.dependencies.crypto.encrypt_aes_gcm - ; decrypt = t.dependencies.crypto.decrypt_aes_gcm } - in - (* This source is the retained local plaintext staging file, not - HTTP wire data. It still crosses bounded checksum verification. *) - let* verified = Asset_codec.decode - ~maximum_plaintext_bytes:t.dependencies.local_store.asset_maximum_file_bytes - ~crypto ~key:None ~expected:version plaintext - |> Result.map_error asset_decode_failure in - let* handle = Asset_cache.publish cache ~asset ~current ~verified - |> Result.map_error asset_cache_failure in - Asset_cache.release cache handle; - Ok Core.Asset_unit)) - | Retain_asset_file handle -> - Ok (Core.Asset_retained (retain_asset_file t ~scope ~handle)) - | Retain_staged_file file -> - if t.closed then Ok (Core.Asset_retained None) else cache (fun cache -> - let retained = Option.bind (Asset_cache.retain_staged cache ~file) (fun lease -> - match Asset_cache.path cache lease with - | Some path -> Hashtbl.replace t.asset_leases (scope, lease) (); Some (lease, path) - | None -> Asset_cache.release cache lease; None) in - Ok (Core.Asset_retained retained)) - |> Result.map_error (function - | Core.Asset_invalid_content _ -> - Core.Asset_invalid_content "Attachment preview storage is unavailable" - | failure -> failure) - | Release_asset_file handle -> - release_asset_file t ~scope ~handle |> map_unit - | Release_staged_file file -> - release_staged_asset t ~scope ~file |> storage_result |> map_unit - | Prune_asset_staging keep -> - prune_staged_assets t ~scope ~keep - |> Result.map (fun count -> Core.Asset_pruned count) - | Await_asset_operations -> - Rrbvec.iter Eio.Promise.await asset_snapshot; - if t.closed then Error Core.Asset_cancelled else Ok Core.Asset_unit - | Close_asset_scope -> - close_asset_scope_checked ?except_id t scope |> Result.map_error asset_cache_failure |> map_unit - | Prepare_graph_asset_deletion retirement -> - prepare_graph_asset_deletion ?except_id t scope retirement ~current - | Commit_graph_asset_deletion deletion -> finish_graph_asset_deletion t deletion ~commit:true - | Cancel_graph_asset_deletion deletion -> finish_graph_asset_deletion t deletion ~commit:false - | Delete_account_assets -> - delete_account_assets ?except_id t scope.account - |> Result.map_error asset_cache_failure - |> map_unit - | Asset_retry_after seconds -> - t.dependencies.runtime.sleep seconds; - Ok Core.Asset_retry_elapsed - | Cancel_asset_operation target -> - Hashtbl.iter - (fun _ (pending : operation) -> - if pending.asset_identity = Some (scope, target) - then ( - pending.cancelled <- true; - pending.cancel ())) - t.operations; - Ok Core.Asset_unit -;; - -let scoped_operation_id kind (scope : Core.graph_scope) ticket = - Printf.sprintf - "%s:%S:%S:%d:%d:%Ld:%S:%d:%S" - kind - (Uri.to_string scope.account.managed_sync_origin) - scope.account.user_id - scope.account.account_generation - scope.account.presentation_generation - scope.account.lifecycle_generation - (Graph_types.Uuid.to_string scope.graph_id) - scope.graph_generation - ticket -;; - -let claim_operation t instruction = - if Submitted_effects.mem t.submitted_operations instruction - then false - else ( - Submitted_effects.add t.submitted_operations instruction; - true) -;; - -let fork_asset_task t id (request:Core.asset_io_request) task = - let finished,resolve=Eio.Promise.create() in - let tracked=match request.action with Core.Await_asset_operations->false|_->true in - if tracked then Hashtbl.add t.asset_jobs id(request.context.scope,finished); - let complete()= - if tracked then Hashtbl.remove t.asset_jobs id; - ignore(Eio.Promise.try_resolve resolve():bool) in - try t.dependencies.runtime.fork ~sw:t.sw(fun()->Fun.protect ~finally:complete task) - with error->complete();raise error -;; - -let submit_asset_io t instruction ticket (request : Core.asset_io_request) = - let id = - scoped_operation_id "asset" request.context.scope (Core.asset_ticket_id ticket) - in - if claim_operation t instruction - then ( - let busy = match request.action with - | Core.Prepare_graph_asset_deletion retirement -> - let key = asset_target request.context.scope in - Hashtbl.mem t.preparing_retirements key - || (match Hashtbl.find_opt t.asset_retirements key with Some other -> other <> retirement | None -> false) - | _ -> false in - if busy then t.post (Core.Runner_completed (Core.Asset_completion (ticket, - Error (Core.Asset_invalid_content "Attachment retirement is already in progress.")))) - else ( - (match request.action with - | Core.Prepare_graph_asset_deletion retirement -> - let scope = request.context.scope in - (match Asset_cache.graph_deletion_terminal ~root:(asset_root t) ~scope ~retirement with - | Ok None -> - Hashtbl.replace t.preparing_retirements (asset_target scope) retirement; - (match Hashtbl.find_opt t.asset_retirements (asset_target scope) with - | None -> Hashtbl.add t.asset_retirements (asset_target scope) retirement - | Some _ -> ()); - if Hashtbl.find_opt t.asset_retirements (asset_target scope) = Some retirement then - cancel_graph_asset_operations t - { Core.account = scope.account; graph_id = scope.graph_id; scope = Core.effect_scope_of_graph scope } - | _ -> ()) - | _ -> ()); - let cancelled, resolve_cancelled = Eio.Promise.create () in - let operation = - { network = (match request.action with - | Core.Fetch_asset _ | Put_asset_file _ | Asset_retry_after _ -> true - | _ -> false) - ; scope = Core.effect_scope_of_graph request.context.scope - ; asset_identity = Some (request.context.scope, request.operation) - ; cancelled = false - ; cancel = (fun () -> ignore (Eio.Promise.try_resolve resolve_cancelled () : bool)) - } - in - Hashtbl.add t.operations id operation; - let asset_snapshot=match request.action with - |Core.Await_asset_operations->capture_asset_jobs t request.context.scope - |_->Rrbvec.empty in - fork_asset_task t id request (fun () -> - let current () = (not t.closed) && not operation.cancelled in - let result = - Fun.protect - ~finally:(fun () -> - Hashtbl.remove t.operations id; - match request.action with - | Core.Prepare_graph_asset_deletion retirement -> - let key = asset_target request.context.scope in - if Hashtbl.find_opt t.preparing_retirements key = Some retirement - then Hashtbl.remove t.preparing_retirements key - | _ -> ()) - (fun () -> - try - Eio.Fiber.first - (fun () -> - if not (current ()) then raise Runner_cancelled; - with_asset_diagnostics t request ~current (fun () -> - execute_asset ~except_id:id ~asset_snapshot t request ~current)) - (fun () -> - Eio.Promise.await cancelled; - raise Runner_cancelled) - with - | Runner_cancelled -> Error Core.Asset_cancelled - | Eio.Cancel.Cancelled _ as error -> raise error - | error -> Error (asset_exception request error)) - in - (* A completed resource must reach the reducer even after cancellation. - The reducer owns acceptance and issues cleanup for stale resources. *) - t.post (Core.Runner_completed (Core.Asset_completion (ticket, result)))))) -;; - -let execute_protected t (request : Core.protected_request) = - let valid handle = Core.graph_key_handle_scope handle = request.protected_scope in - let run handle action = - if valid handle - then Result.map_error (fun message -> Core.Effect_failed message) (action ()) - else Error (Core.Effect_failed "Protected request graph key is out of scope") - in - match request.protected_action with - | Core.Encrypt_values (handle, values) -> - run handle (fun () -> encrypt_protected_values t handle values) - |> Result.map (fun values -> Core.Encrypted_values values) - | Decrypt_value (handle, source) -> - run handle (fun () -> decrypt_protected_value t handle source) - |> Result.map (fun value -> Core.Decrypted_value value) -;; - -let submit_protected t instruction ticket (request : Core.protected_request) = - let id = - scoped_operation_id - "protected" - request.protected_scope - (Core.protected_ticket_id ticket) - in - if claim_operation t instruction - then ( - let cancelled, resolve_cancelled = Eio.Promise.create () in - let operation = - { network = false - ; scope = Core.effect_scope_of_graph request.protected_scope - ; asset_identity = None - ; cancelled = false - ; cancel = (fun () -> ignore (Eio.Promise.try_resolve resolve_cancelled () : bool)) - } - in - Hashtbl.add t.operations id operation; - fork_general t ~scope:(Core.effect_scope_of_graph request.protected_scope) (fun () -> - let result = - Fun.protect - ~finally:(fun () -> Hashtbl.remove t.operations id) - (fun () -> - try - Eio.Fiber.first - (fun () -> - if t.closed || operation.cancelled then raise Runner_cancelled; - execute_protected t request) - (fun () -> - Eio.Promise.await cancelled; - raise Runner_cancelled) - with - | Runner_cancelled -> - Error (Core.Effect_failed "Protected request cancelled") - | Eio.Cancel.Cancelled _ as error -> raise error - | error -> Error (Core.Effect_failed Failure_diagnostic.( - of_source ~operation:(match request.protected_action with - | Core.Encrypt_values _ -> "encryptValue" | Decrypt_value _ -> "decryptValue") - ~reason:"cryptoInvalid" (Printexc.to_string error) |> encode))) - in - t.post (Core.Runner_completed (Core.Protected_completion (ticket, result))))) -;; - -let cleanup_asset_action = function - | Core.Release_asset_file _ - | Release_staged_file _ - | Close_asset_scope - | Await_asset_operations - | Prepare_graph_asset_deletion _ - | Commit_graph_asset_deletion _ - | Cancel_graph_asset_deletion _ - | Delete_account_assets - | Cancel_asset_operation _ -> true - | Check_asset_cache _ - | Fetch_asset _ - | Stage_asset_file _ - | Put_asset_file _ - | Publish_staged_asset _ - | Retain_asset_file _ - | Retain_staged_file _ - | Prune_asset_staging _ - | Asset_retry_after _ -> false -;; - -let submit_io_fence t id scope = - (* Capture before yielding/forking. Later unrelated work is not part of this - round; waiting must never occupy the serial callback consumer. *) - let jobs=Hashtbl.fold(fun _ (owner,finished) jobs-> - if scope_matches scope owner then Rrbvec.push_back jobs finished else jobs) - t.general_jobs Rrbvec.empty in - let jobs=Hashtbl.fold(fun _ (owner,finished) jobs-> - if scope_matches scope(Core.effect_scope_of_graph owner) - then Rrbvec.push_back jobs finished else jobs)t.asset_jobs jobs in - t.dependencies.runtime.fork ~sw:t.sw(fun()-> - let result=if t.closed then Error "Sync stopped before I/O settlement." else ( - Rrbvec.iter Eio.Promise.await jobs; - if t.closed then Error "Sync stopped before I/O settlement." else Ok()) in - t.post(Core.IO_fence_completed(id,scope,result))) -;; - -let submit t instruction = - match instruction with - |Core.Await_io(id,scope)->submit_io_fence t id scope - | Core.Asset_io (ticket, ({ action = (Core.Prepare_graph_asset_deletion _ | Await_asset_operations); _ } as request)) -> - (* Waiting must leave the serial event consumer free to deliver the I/O - callbacks or native lease releases being joined. *) - if t.closed - then t.post (Core.Runner_completed (Core.Asset_completion (ticket, Error Core.Asset_cancelled))) - else submit_asset_io t instruction ticket request - | Core.Asset_io (ticket, request) -> - if cleanup_asset_action request.action - then ( - if claim_operation t instruction - then ( - let result = - try - Fun.protect - ~finally:(fun () -> - if t.closed then close_asset_scope t request.context.scope) - (fun () -> with_asset_diagnostics t request - ~current:(fun () -> not t.closed) - (fun () -> execute_asset t request ~current:(fun () -> false))) - with - | Eio.Cancel.Cancelled _ as error -> raise error - | error -> Error (asset_exception request error) - in - t.post (Core.Runner_completed (Core.Asset_completion (ticket, result))))) - else if t.closed - then ( - if claim_operation t instruction - then - t.post - (Core.Runner_completed - (Core.Asset_completion (ticket, Error Core.Asset_cancelled)))) - else submit_asset_io t instruction ticket request - | Protected_io (ticket, request) -> - if t.closed - then ( - if claim_operation t instruction - then - t.post - (Core.Runner_completed - (Core.Protected_completion - (ticket, Error (Core.Effect_failed "Protected request cancelled"))))) - else submit_protected t instruction ticket request - | _ -> submit_nonasset t instruction -;; diff --git a/logseq_sync/lib/effect_runner/eio/http_eio.mli b/logseq_sync/lib/effect_runner/eio/http_eio.mli deleted file mode 100644 index 7925273c..00000000 --- a/logseq_sync/lib/effect_runner/eio/http_eio.mli +++ /dev/null @@ -1,24 +0,0 @@ -type response = - { status : int - ; headers : (string * string) list - ; body : string - } - -val perform - : sw:Eio.Switch.t - -> authenticator:X509.Authenticator.t - -> network:_ Eio.Net.t - -> clock:_ Eio.Time.clock - -> Http.request - -> (response, string) result - -val download - : sw:Eio.Switch.t - -> authenticator:X509.Authenticator.t - -> network:_ Eio.Net.t - -> clock:_ Eio.Time.clock - -> request:Http.request - -> destination:string - -> maximum_bytes:int - -> on_progress:(Bootstrap.progress -> unit) - -> (response, string) result diff --git a/logseq_sync/lib/effect_runner/protocol/catalog.ml b/logseq_sync/lib/effect_runner/protocol/catalog.ml deleted file mode 100644 index e3374ae9..00000000 --- a/logseq_sync/lib/effect_runner/protocol/catalog.ml +++ /dev/null @@ -1,107 +0,0 @@ -type schema = Managed_graph.schema = - { major : int - ; minor : int - ; exact : bool - } - -type graph = Managed_graph.t = - { graph_id : Graph_types.Uuid.t - ; name : string - ; schema : schema - ; encrypted : bool - } - -let bind result f = - match result with - | Ok value -> f value - | Error _ as error -> error -;; - -let bounded_text ~maximum name value = - if - String.length value > 0 - && String.length value <= maximum - && String.is_valid_utf_8 value - && not (String.contains value '\000') - then Ok value - else Error (name ^ " must be bounded non-empty UTF-8 text") -;; - -let schema = function - | None -> Ok { major = 0; minor = 0; exact = false } - | Some value -> - (match String.split_on_char '.' value with - | [ major ] -> - (match int_of_string_opt major with - | Some major when major >= 0 -> Ok { major; minor = 0; exact = false } - | Some _ | None -> Error "graph schema version is invalid") - | [ major; minor ] -> - (match int_of_string_opt major, int_of_string_opt minor with - | Some major, Some minor when major >= 0 && minor >= 0 -> - Ok { major; minor; exact = true } - | _ -> Error "graph schema version is invalid") - | _ -> Error "graph schema version is invalid") -;; - -type readiness = { ready : bool option [@key "graph-ready-for-use?"] [@default None] } -[@@deriving of_yojson { strict = false }] - -type graph_dto = - { graph_id : Graph_types.Uuid.t [@key "graph-id"] - ; graph_name : string [@key "graph-name"] - ; schema_version : string option [@key "schema-version"] [@default None] - ; encrypted : bool option [@key "graph-e2ee?"] [@default None] - } -[@@deriving of_yojson { strict = false }] - -type response = { graphs : Yojson.Safe.t list } [@@deriving of_yojson { strict = false }] - -let decode_fields decoder json = - let fields = List.map (fun (field, wire) -> field, - "field " ^ wire ^ " is missing or has an invalid type") - [ "Catalog.response.graphs", "graphs" - ; "Catalog.readiness.ready", "graph-ready-for-use?" - ; "Catalog.graph_dto.graph_id", "graph-id" - ; "Catalog.graph_dto.graph_name", "graph-name" - ; "Catalog.graph_dto.schema_version", "schema-version" - ; "Catalog.graph_dto.encrypted", "graph-e2ee?" ] in - Logseq_db_types.Json_codec.decode decoder json - |> Result.map_error (fun message -> - "graph catalog: " ^ Logseq_db_types.Json_codec.error_context ~fields message) -;; - -let decode_graph json = - bind (decode_fields readiness_of_yojson json) (fun ready -> - match ready.ready with - | None | Some false -> Ok None - | Some true -> - bind (decode_fields graph_dto_of_yojson json) (fun dto -> - bind (bounded_text ~maximum:512 "graph name" dto.graph_name) (fun name -> - bind (schema dto.schema_version) (fun schema -> - Ok - (Some - { graph_id = dto.graph_id - ; name - ; schema - ; encrypted = Option.value dto.encrypted ~default:true - }))))) -;; - -let decode source = - try - bind - (decode_fields - response_of_yojson - (Yojson.Safe.from_string source)) - (fun response -> - let rec loop graphs = function - | [] -> Ok (List.rev graphs) - | json :: rest -> - bind (decode_graph json) (function - | None -> loop graphs rest - | Some graph -> loop (graph :: graphs) rest) - in - loop [] response.graphs) - with - | Yojson.Json_error _ -> Error "graph catalog must be valid JSON" -;; diff --git a/logseq_sync/lib/effect_runner/protocol/catalog.mli b/logseq_sync/lib/effect_runner/protocol/catalog.mli deleted file mode 100644 index 2e8b7c7c..00000000 --- a/logseq_sync/lib/effect_runner/protocol/catalog.mli +++ /dev/null @@ -1,14 +0,0 @@ -type schema = Logseq_db_types.Managed_graph.schema = - { major : int - ; minor : int - ; exact : bool - } - -type graph = Logseq_db_types.Managed_graph.t = - { graph_id : Graph_types.Uuid.t - ; name : string - ; schema : schema - ; encrypted : bool - } - -val decode : string -> (graph list, string) result diff --git a/logseq_sync/lib/effect_runner/storage/asset_cache.ml b/logseq_sync/lib/effect_runner/storage/asset_cache.ml deleted file mode 100644 index 8485dca0..00000000 --- a/logseq_sync/lib/effect_runner/storage/asset_cache.ml +++ /dev/null @@ -1,6 +0,0 @@ -(* This virtual-module implementation intentionally exposes no cache IO. - The implementation belongs to private [Bootstrap.Asset_cache]. *) -type t = unit -type handle = unit -type error = unit -type staged = unit diff --git a/logseq_sync/lib/effect_runner/storage/bootstrap.ml b/logseq_sync/lib/effect_runner/storage/bootstrap.ml deleted file mode 100644 index c9a0c474..00000000 --- a/logseq_sync/lib/effect_runner/storage/bootstrap.ml +++ /dev/null @@ -1,1084 +0,0 @@ -module Failure_diagnostic = Logseq_db_types.Failure_diagnostic - -let failure_message ~reason:reason_name source = - Failure_diagnostic.(of_source ~operation:"decodeSnapshot" ~reason:reason_name source |> message) -;; - -type progress = - { received_bytes : int - ; total_bytes : int option - ; datom_count : int option - } - -let artifact_row_count headers = - let value = - List.find_map - (fun (name, value) -> - if String.equal (String.lowercase_ascii name) "x-snapshot-row-count" - then Some value - else None) - headers - in - match Option.bind value int_of_string_opt with - | Some value when value > 0 -> Ok value - | _ -> Error "snapshot row-count header is invalid" -;; - -let remove_if_exists path = - try Sys.remove path with - | Sys_error _ as error -> - (* Missing artifacts are normal idempotent cleanup. Preserve the original - unit completion if an artifact remains, but retain a safe support cause. *) - if Sys.file_exists path then ( - let detail = Failure_diagnostic.of_source ~operation:"decodeSnapshot" - ~reason:"cleanupFailed" (Printexc.to_string error) in - try Printf.eprintf - "journal: phase=cleanup operation=removeSnapshotArtifact code=cleanupPending source=%s\n%!" - (Failure_diagnostic.source_class detail) with _ -> ()) -;; - -let cleanup = List.iter remove_if_exists - -let gzip_signature path = - try - let channel = open_in_bin path in - Fun.protect - ~finally:(fun () -> close_in_noerr channel) - (fun () -> - try - let first = input_char channel in - let second = input_char channel in - Ok (first = '\x1f' && second = '\x8b') - with - | End_of_file -> Ok false) - with - | Sys_error _ as error -> - Error (failure_message ~reason:"resourceFailed" (Printexc.to_string error)) -;; - -let copy_file ~maximum_bytes source destination = - try - let input_channel = open_in_bin source in - Fun.protect - ~finally:(fun () -> close_in_noerr input_channel) - (fun () -> - let output_channel = - open_out_gen - [ Open_wronly; Open_creat; Open_excl; Open_binary ] - 0o600 - destination - in - Fun.protect - ~finally:(fun () -> close_out_noerr output_channel) - (fun () -> - let buffer = Bytes.create 16_384 in - let copied = ref 0 in - let rec loop () = - let count = input input_channel buffer 0 (Bytes.length buffer) in - if count > 0 - then ( - if !copied + count > maximum_bytes - then failwith "snapshot artifact exceeds its decompressed bound"; - output output_channel buffer 0 count; - copied := !copied + count; - loop ()) - in - loop ())); - Ok () - with - | Failure source when source = "snapshot artifact exceeds its decompressed bound" -> - remove_if_exists destination; - Error "snapshot artifact exceeds its decompressed bound" - | exception_ -> - remove_if_exists destination; - Error (failure_message ~reason:"resourceFailed" (Printexc.to_string exception_)) -;; - -let nonempty path = - try (Unix.stat path).st_size > 0 with - | Unix.Unix_error _ -> false -;; - -let decompress ~decompress_gzip ~maximum_bytes source destination = - remove_if_exists destination; - match decompress_gzip source destination maximum_bytes with - | 0 when nonempty destination -> Ok () - | 0 -> - remove_if_exists destination; - Error "snapshot gzip layer is empty" - | status -> - remove_if_exists destination; - Error (match status with - | 1 -> "snapshot gzip input could not be opened" - | 2 -> "snapshot gzip output could not be opened" - | 3 -> "snapshot gzip decode failed" - | 4 -> "snapshot gzip output write failed" - | 5 -> "snapshot gzip decompressed limit exceeded" - | _ -> "snapshot gzip layer is invalid") -;; - -let peel_gzip_layers ~decompress_gzip ~maximum_bytes ~source ~destination ~temporary_paths - = - if maximum_bytes <= 0 - then Error "snapshot decompressed bound must be positive" - else ( - match temporary_paths with - | first :: second :: _ -> - cleanup [ destination; first; second ]; - let fail message = - cleanup [ destination; first; second ]; - Error message - in - (match gzip_signature source with - | Error message -> fail message - | Ok false -> copy_file ~maximum_bytes source destination - | Ok true -> - (match decompress ~decompress_gzip ~maximum_bytes source first with - | Error message -> fail message - | Ok () -> - (match gzip_signature first with - | Error message -> fail message - | Ok false -> - (try - Sys.rename first destination; - cleanup [ second ]; - Ok () - with - | Sys_error _ as error -> - fail (failure_message ~reason:"persistenceFailed" (Printexc.to_string error))) - | Ok true -> - (match decompress ~decompress_gzip ~maximum_bytes first second with - | Error message -> fail message - | Ok () -> - (match gzip_signature second with - | Error message -> fail message - | Ok true -> fail "snapshot exceeds the supported gzip layer count" - | Ok false -> - (try - Sys.rename second destination; - cleanup [ first ]; - Ok () - with - | Sys_error _ as error -> - fail (failure_message ~reason:"persistenceFailed" (Printexc.to_string error)))))))) - | [] | [ _ ] -> Error "two private gzip staging paths are required") -;; - -(* Cache IO is private to the runner interpreter. *) -module Asset_cache = struct - module Core = Logseq_sync_pure_reducer.Core - module Asset = Logseq_db_types.Asset_descriptor - module Uuid = Logseq_db_types.Graph_types.Uuid - - type handle = string - - type error = - | Invalid of string - | Io of string - | Full - | Stale - | Checksum_mismatch - - type manifest = - { asset : Uuid.t - ; file_type : string - ; checksum : string - ; size : int64 - } - [@@deriving yojson] - - type record = - { name : string - ; file_type : string - ; checksum : string - ; size : int64 - ; mutable touched : float - ; mutable pins : int - } - - type staged_record = - { staged_name : string - ; staged_location : string - ; mutable staged_pins : int - ; mutable retired : bool - } - - type t = - { directory : string - ; budget : int64 - ; maximum_file_bytes : int - ; records : (string, record) Hashtbl.t - ; handles : (handle, record) Hashtbl.t - ; staged_records : (string, staged_record) Hashtbl.t - ; staged_handles : (handle, staged_record) Hashtbl.t - ; mutable closed : bool - } - - let next_handle = Atomic.make 0 - let ( let* ) = Result.bind - - let protect f = - try f () with - | Failure source as error -> - let diagnostic = Failure_diagnostic.of_source ~operation:"assetCache" - ~reason:"persistenceFailed" source in - let diagnostic = - if Failure_diagnostic.source_class diagnostic = "unknown" - && Option.is_none (Failure_diagnostic.decode source) - then Failure_diagnostic.of_source ~operation:"assetCache" ~reason:"persistenceFailed" - (Printexc.to_string error) - else diagnostic in - Error (Io (Failure_diagnostic.encode diagnostic)) - | (Sys_error _ | Unix.Unix_error _) as error -> - Error (Io Failure_diagnostic.(of_source ~operation:"assetCache" - ~reason:"persistenceFailed" (Printexc.to_string error) |> encode)) - | Yojson.Json_error _ | Yojson.Safe.Util.Type_error _ -> - Error (Invalid "Asset cache metadata shape is invalid") - ;; - - let rec mkdir path = - if not (Sys.file_exists path) - then ( - mkdir (Filename.dirname path); - Unix.mkdir path 0o700; - let parent = Unix.openfile (Filename.dirname path) [ Unix.O_RDONLY ] 0 in - Fun.protect ~finally:(fun () -> Unix.close parent) (fun () -> Unix.fsync parent)) - ;; - - let unlink path = - try Unix.unlink path with - | Unix.Unix_error (Unix.ENOENT, _, _) -> () - ;; - - (* Data files carry the validated attachment type so native presentation - (for example Quick Look) classifies them correctly. *) - let data_path t record = - Filename.concat t.directory (record.name ^ "." ^ record.file_type) - ;; - - let manifest_path t name = Filename.concat t.directory (name ^ ".json") - - let name asset (version : Asset.version) = - Asset_codec.checksum - (Uuid.to_string asset ^ ":" ^ version.file_type ^ ":" ^ version.checksum) - ;; - - let sync_directory t = - let fd = Unix.openfile t.directory [ Unix.O_RDONLY ] 0 in - Fun.protect ~finally:(fun () -> Unix.close fd) (fun () -> Unix.fsync fd) - ;; - - let write path bytes = - let fd = Unix.openfile path [ Unix.O_WRONLY; Unix.O_CREAT; Unix.O_TRUNC ] 0o600 in - Fun.protect - ~finally:(fun () -> Unix.close fd) - (fun () -> - let rec loop offset = - if offset < String.length bytes - then ( - let count = - Unix.write_substring fd bytes offset (String.length bytes - offset) - in - if count = 0 - then raise (Sys_error "Asset file write made no progress") - else loop (offset + count)) - in - loop 0; - Unix.fsync fd) - ;; - - let checksum_file path = - let channel = open_in_bin path in - Fun.protect - ~finally:(fun () -> close_in channel) - (fun () -> - let buffer = Bytes.create 65536 in - let rec read context = - match input channel buffer 0 (Bytes.length buffer) with - | 0 -> Digestif.SHA256.(to_hex (get context)) - | length -> read (Digestif.SHA256.feed_bytes context ~off:0 ~len:length buffer) - in - read Digestif.SHA256.empty) - ;; - - let remove_record t record = - unlink (manifest_path t record.name); - unlink (data_path t record); - Hashtbl.remove t.records record.name - ;; - - let pin t record = - record.pins <- record.pins + 1; - record.touched <- Unix.gettimeofday (); - Unix.utimes (manifest_path t record.name) record.touched record.touched; - let handle = Printf.sprintf "asset:%d" (Atomic.fetch_and_add next_handle 1) in - Hashtbl.add t.handles handle record; - handle - ;; - - let read_manifest directory file = - let path = Filename.concat directory file in - let invalid message = Error (Invalid message) in - try - if (Unix.stat path).st_size > 1024 - then invalid "Asset cache manifest exceeds size limit" - else ( - match - Logseq_db_types.Json_codec.decode - manifest_of_yojson - (Yojson.Safe.from_file path) - with - | Error _ -> invalid "Asset cache manifest shape is invalid" - | Ok manifest -> - (match Asset.version ~checksum:manifest.checksum ~file_type:manifest.file_type with - | Error _ -> invalid "Asset cache manifest asset version is invalid" - | Ok _ when manifest.size < 0L -> invalid "Asset cache manifest size is invalid" - | Ok version when name manifest.asset version ^ ".json" <> file -> - invalid "Asset cache manifest filename does not match its identity" - | Ok version -> - Ok { name = name manifest.asset version; file_type = version.file_type - ; checksum = version.checksum; size = manifest.size - ; touched = (Unix.stat path).st_mtime; pins = 0 })) - with - | Yojson.Json_error _ -> invalid "Asset cache manifest JSON is invalid" - | Failure _ | Invalid_argument _ -> invalid "Asset cache manifest shape is invalid" - ;; - - type account_identity = string * string [@@deriving to_yojson] - - let account_directory ~root (account : Core.account_scope) = - let name = - Asset_codec.checksum - (account_identity_to_yojson - (Uri.to_string account.managed_sync_origin, account.user_id) - |> Yojson.Safe.to_string) - in - Filename.concat root name - ;; - - - let lookup t ~asset ~version = - protect (fun () -> - if t.closed - then Error Stale - else ( - match Hashtbl.find_opt t.records (name asset version) with - | None -> Ok None - | Some record -> - let path = data_path t record in - let valid = - try - let stat = Unix.stat path in - stat.st_kind = Unix.S_REG - && Int64.of_int stat.st_size = record.size - && stat.st_size <= t.maximum_file_bytes - && checksum_file path = record.checksum - with - | Sys_error _ | Unix.Unix_error _ -> false - in - if valid - then Ok (Some (pin t record)) - else ( - if record.pins = 0 then remove_record t record; - Ok None))) - ;; - - let reserve t needed = - let usage = - Hashtbl.fold (fun _ record total -> Int64.add total record.size) t.records 0L - in - let candidates = - Hashtbl.fold - (fun _ record acc -> if record.pins = 0 then record :: acc else acc) - t.records - [] - |> List.sort (fun a b -> Float.compare a.touched b.touched) - in - let rec evict usage = function - | _ when needed <= t.budget && usage <= Int64.sub t.budget needed -> Ok () - | [] -> Error Full - | record :: rest -> - remove_record t record; - evict (Int64.sub usage record.size) rest - in - if needed > t.budget then Error Full else evict usage candidates - ;; - - let publish t ~asset ~current ~verified = - let version = Asset_codec.verified_version verified in - let plaintext = Asset_codec.verified_bytes verified in - protect (fun () -> - if t.closed || not (current ()) - then Error Stale - else if String.length plaintext > t.maximum_file_bytes - then Error (Invalid "Asset exceeds cache file limit") - else if Asset_codec.checksum plaintext <> version.checksum - then Error Checksum_mismatch - else - let* existing = lookup t ~asset ~version in - match existing with - | Some handle -> Ok handle - | None -> - let size = Int64.of_int (String.length plaintext) in - let* () = reserve t size in - let name = name asset version in - (* A corrupt pinned record cannot be replaced beneath a renderer. *) - if Hashtbl.mem t.records name - then Error Full - else ( - let record = - { name - ; file_type = version.file_type - ; checksum = version.checksum - ; size - ; touched = Unix.gettimeofday () - ; pins = 0 - } - in - let data = data_path t record - and manifest = manifest_path t name in - let temporary = data ^ ".part" - and temporary_manifest = manifest ^ ".part" in - Fun.protect - ~finally:(fun () -> - unlink temporary; - unlink temporary_manifest) - (fun () -> - write temporary plaintext; - write - temporary_manifest - (manifest_to_yojson - { asset - ; file_type = version.file_type - ; checksum = version.checksum - ; size - } - |> Yojson.Safe.to_string); - if t.closed || not (current ()) - then Error Stale - else ( - Unix.rename temporary data; - Unix.rename temporary_manifest manifest; - sync_directory t; - Hashtbl.replace t.records name record; - Ok (pin t record))))) - ;; - - let pin_staged t record = - record.staged_pins <- record.staged_pins + 1; - let handle = Printf.sprintf "staged:%d" (Atomic.fetch_and_add next_handle 1) in - Hashtbl.add t.staged_handles handle record; - handle - ;; - - let report_cleanup ?(code="collectionPending") operation error = - let source = match error with - | Invalid source | Io source -> source - | Full -> "Asset staging storage is full" - | Stale -> "Asset staging scope is stale" - | Checksum_mismatch -> "Asset staging checksum mismatch" in - let diagnostic = Failure_diagnostic.of_source ~operation:"assetRelease" - ~reason:"cleanupFailed" source in - Printf.eprintf "journal: phase=cleanup operation=%s code=%s source=%s\n%!" - operation code (Failure_diagnostic.source_class diagnostic) - ;; - - let report_cleanup_exception ?code operation error = - report_cleanup ?code operation (Io (Printexc.to_string error)) - ;; - - let clean_retired_staging t record = - let result = if record.retired && record.staged_pins = 0 - then - protect (fun () -> - unlink record.staged_location; - let directory = Filename.dirname record.staged_location in - if Sys.file_exists directory - then ( - let fd = Unix.openfile directory [ Unix.O_RDONLY ] 0 in - Fun.protect ~finally:(fun () -> Unix.close fd) (fun () -> Unix.fsync fd)); - Hashtbl.remove t.staged_records record.staged_name; - Ok ()) - else Ok () in - (match result with - | Ok () -> () - | Error error -> report_cleanup ~code:"cleanupPending" "releaseRetiredStaging" error); - result - ;; - - let path t handle = - if t.closed - then None - else ( - match Hashtbl.find_opt t.handles handle with - | Some record -> Some (data_path t record) - | None -> - Option.map - (fun record -> record.staged_location) - (Hashtbl.find_opt t.staged_handles handle)) - ;; - - let retain t handle = - if t.closed - then None - else ( - match Hashtbl.find_opt t.handles handle with - | Some record -> Some (pin t record) - | None -> Option.map (pin_staged t) (Hashtbl.find_opt t.staged_handles handle)) - ;; - - let release_checked t handle = - match Hashtbl.find_opt t.handles handle with - | Some record -> - record.pins <- record.pins - 1; - Hashtbl.remove t.handles handle; - Ok () - | None -> - (match Hashtbl.find_opt t.staged_handles handle with - | None -> Ok () - | Some record -> - record.staged_pins <- record.staged_pins - 1; - let result=if record.retired then clean_retired_staging t record else Ok () in - match result with - | Error _ -> - (* The exact handle still owns this failed release. Restoring its pin - permits a same-handle retry without a second decrement. *) - record.staged_pins <- record.staged_pins + 1; - result - | Ok () -> - Hashtbl.remove t.staged_handles handle; - if not record.retired && record.staged_pins=0 - then Hashtbl.remove t.staged_records record.staged_name; - Ok ()) - ;; - - let release t handle = ignore(release_checked t handle) - - let close t = - t.closed <- true; - Hashtbl.clear t.handles; - Hashtbl.iter (fun _ record -> record.pins <- 0) t.records; - Hashtbl.clear t.staged_handles; - Hashtbl.fold (fun _ record records -> record :: records) t.staged_records [] - |> List.iter (fun record -> - record.staged_pins <- 0; - ignore (clean_retired_staging t record)); - Hashtbl.clear t.staged_records - ;; - - let close_checked t = - if Hashtbl.length t.handles>0 || Hashtbl.length t.staged_handles>0 - then Error(Invalid "Attachment resources are still retained.") else - let retired=Hashtbl.fold(fun _ record records->Rrbvec.push_back records record) - t.staged_records Rrbvec.empty in - let cleaned=Rrbvec.fold_left(fun result record-> - Result.bind result(fun()->clean_retired_staging t record))(Ok())retired in - Result.map(fun()->close t)cleaned - ;; - - let remove_tree path = - let rec remove path = - match Unix.lstat path with - | { Unix.st_kind = Unix.S_DIR; _ } -> - Array.iter (fun name -> remove (Filename.concat path name)) (Sys.readdir path); - Unix.rmdir path - | _ -> Unix.unlink path - | exception Unix.Unix_error (Unix.ENOENT, _, _) -> () - in - protect (fun () -> - remove path; - Ok ()) - ;; - - external rename_exclusive : string -> string -> int = "logseq_journal_rename_exclusive" - - let graph_deletion_paths ~root (scope : Core.graph_scope) retirement = - let account = account_directory ~root scope.account in - let graph = Uuid.to_string scope.graph_id in - let directory = Filename.concat - (Filename.concat (Filename.concat root ".asset-retirements") (Filename.basename account)) graph in - Filename.concat account graph, directory, - Filename.concat (Filename.concat directory "states") (Core.retirement_id_to_string retirement ^ ".json"), - Filename.concat (Filename.concat directory "pending") (Core.retirement_id_to_string retirement ^ ".data") - ;; - - let sync_path directory = - let fd = Unix.openfile directory [Unix.O_RDONLY] 0 in - Fun.protect ~finally:(fun () -> Unix.close fd) (fun () -> Unix.fsync fd) - ;; - - let write_retirement path json = - let temporary = path ^ ".part" in - write temporary (Yojson.Safe.to_string json); - Unix.rename temporary path; - sync_path (Filename.dirname path) - ;; - - let identity path = - match Unix.lstat path with - | { Unix.st_kind = Unix.S_DIR; st_dev; st_ino; _ } -> Printf.sprintf "%d:%d" st_dev st_ino - | _ -> failwith "asset namespace is not a directory" - ;; - - let retirement_state manifest_path = - let open Yojson.Safe.Util in - let json = Yojson.Safe.from_file manifest_path in - if json |> member "version" |> to_int <> 1 then failwith "unsupported asset retirement"; - json |> member "identity" |> to_string, json |> member "state" |> to_string - ;; - - let retirement_json captured state = - `Assoc ["version", `Int 1; "identity", `String captured; "state", `String state] - ;; - - let graph_deletion_terminal ~root ~scope ~retirement = - protect (fun () -> - let _, _, manifest, _ = graph_deletion_paths ~root scope retirement in - if not (Sys.file_exists manifest) then Ok None else - let _, state = retirement_state manifest in - Ok (match state with - | "complete" | "committed" -> - let fd = Unix.openfile manifest [Unix.O_RDONLY] 0 in - Fun.protect ~finally:(fun () -> Unix.close fd) (fun () -> Unix.fsync fd); - sync_path (Filename.dirname manifest); - Some `Committed - | "aborted" | "account-deleted" -> Some `Aborted - | _ -> None)) - ;; - - let prepare_graph_deletion ~root ~(scope : Core.graph_scope) ~retirement = - protect (fun () -> - let canonical, directory, manifest, data = graph_deletion_paths ~root scope retirement in - mkdir directory; - mkdir (Filename.dirname manifest); mkdir (Filename.dirname data); - let captured, state = - if Sys.file_exists manifest then retirement_state manifest - else ( - let captured = if Sys.file_exists canonical then identity canonical else "absent" in - write_retirement manifest (retirement_json captured "preparing"); captured, "preparing") in - if state <> "preparing" && state <> "prepared" then Error Stale - else ( - if Sys.file_exists data then ( - if identity data <> captured then failwith "retired asset identity changed") - else if captured <> "absent" then ( - if not (Sys.file_exists canonical) || identity canonical <> captured - then failwith "canonical asset identity changed"; - let error = rename_exclusive canonical data in - if error <> 0 then ( - Printf.eprintf "journal: phase=cleanup operation=prepareAssetRetirement code=renameFailed errno=%d\n%!" error; - failwith "asset retirement rename failed"); - sync_path (Filename.dirname canonical); sync_path (Filename.dirname data)); - write_retirement manifest (retirement_json captured "prepared"); - Ok ())) - ;; - - let finish_graph_deletion ~root ~deletion ~commit = - protect (fun () -> - let scope = Core.asset_deletion_scope deletion in - let canonical, _directory, manifest, data = graph_deletion_paths ~root scope - (Core.asset_deletion_retirement deletion) in - if not (Sys.file_exists manifest) then ( - if Sys.file_exists data then Error Stale - else if commit then Error Stale - else ( - mkdir (Filename.dirname manifest); - write_retirement manifest (retirement_json "absent" "aborted"); Ok ())) - else - let captured, state = retirement_state manifest in - if state = "account-deleted" then ( - (match remove_tree data with - | Ok () -> () - | Error error -> report_cleanup "collectDeletedAccountAssets" error); Ok ()) - else if state = "aborted" || state = "complete" then Ok () - else if commit then ( - if Sys.file_exists data && identity data <> captured then failwith "retired asset identity changed"; - write_retirement manifest (retirement_json captured "committed"); - (* The producer owns deferred collection. A purge failure must not - block a replacement cache or undo the durable commit decision. *) - (match remove_tree data with - | Error error -> report_cleanup "collectRetiredAssets" error - | Ok () -> - (try sync_path (Filename.dirname data); - write_retirement manifest (retirement_json captured "complete") with error -> - report_cleanup_exception ~code:"cleanupPending" "completeRetirement" error)); - Ok ()) - else if state = "committed" then Ok () - else ( - if Sys.file_exists data then ( - if identity data <> captured then failwith "retired asset identity changed"; - mkdir (Filename.dirname canonical); - let error = rename_exclusive data canonical in - if error <> 0 then ( - Printf.eprintf "journal: phase=cleanup operation=restoreRetiredAssets code=renameFailed errno=%d\n%!" error; - failwith "asset restore conflict; retirement retained"); - sync_path (Filename.dirname canonical); sync_path (Filename.dirname data)) - else if captured <> "absent" && - (not (Sys.file_exists canonical) || identity canonical <> captured) - then failwith "asset restore is incomplete; retirement retained"; - if Sys.file_exists (Filename.dirname canonical) then sync_path (Filename.dirname canonical); - if Sys.file_exists (Filename.dirname data) then sync_path (Filename.dirname data); - write_retirement manifest (retirement_json captured "aborted"); - Ok ())) - ;; - - let collect_retired_assets ~root ~(scope : Core.graph_scope) = - let _, directory, _, _ = graph_deletion_paths ~root scope - (Core.retirement_id_of_string "00000000-0000-4000-8000-000000000000" |> Result.get_ok) in - let pending = Filename.concat directory "pending" in - if Sys.file_exists pending then - Array.iter (fun name -> - if Filename.check_suffix name ".data" then - try - let retirement = Filename.chop_suffix name ".data" |> Core.retirement_id_of_string |> Result.get_ok in - let _, _, manifest, _ = graph_deletion_paths ~root scope retirement in - let _, state = retirement_state manifest in - if state = "committed" || state = "complete" || state = "account-deleted" then - (match finish_graph_deletion ~root - ~deletion:(Core.asset_deletion_reference ~scope ~retirement) ~commit:true with - | Ok () -> () - | Error error -> report_cleanup "finishRetiredAssets" error) - with error -> report_cleanup_exception "collectRetiredAssets" error) (Sys.readdir pending) - ;; - - let delete_account ~root ~account = - protect (fun () -> - let directory = account_directory ~root account in - let retired = Filename.concat (Filename.concat root ".asset-retirements") (Filename.basename directory) in - if Sys.file_exists retired then - Array.iter (fun graph -> - match Uuid.of_string graph with - | Error _ -> () - | Ok _ -> - let folder = Filename.concat retired graph in - let states = Filename.concat folder "states" in - if Sys.file_exists states then Array.iter (fun file -> - if Filename.check_suffix file ".json" then ( - let manifest = Filename.concat states file in - let captured, _ = retirement_state manifest in - write_retirement manifest (retirement_json captured "account-deleted"))) (Sys.readdir states); - match remove_tree (Filename.concat folder "pending") with - | Ok () -> () | Error error -> - report_cleanup "collectDeletedAccountAssets" error; - let source = match error with Invalid source | Io source -> source - | Full -> "Asset staging storage is full" - | Stale -> "Asset staging scope is stale" - | Checksum_mismatch -> "Asset staging checksum mismatch" in - failwith Failure_diagnostic.(of_source ~operation:"assetRelease" - ~reason:"cleanupFailed" source |> encode)) (Sys.readdir retired); - remove_tree directory) - ;; - - let create ~root ~(scope : Core.graph_scope) ~budget_bytes ~maximum_file_bytes = - (try collect_retired_assets ~root ~scope with error -> - report_cleanup_exception "collectRetiredAssets" error); - protect (fun () -> - if - budget_bytes < 0L - || maximum_file_bytes < 0 - || maximum_file_bytes > 100 * 1024 * 1024 - then Error (Invalid "Invalid cache size limits") - else ( - let directory = - Filename.concat - (account_directory ~root scope.account) - (Uuid.to_string scope.graph_id) - in - mkdir directory; - (* Validate all persisted JSON before cleaning anything. Unsupported - internal formats require an explicit caller-visible failure. *) - let rec load_manifests records = function - | [] -> Ok (List.rev records) - | file :: rest when Filename.check_suffix file ".json" -> - let* record = read_manifest directory file in - load_manifests ((file, record) :: records) rest - | _ :: rest -> load_manifests records rest - in - let* manifests = load_manifests [] (Array.to_list (Sys.readdir directory)) in - let pending = Filename.concat directory "pending" in - if Sys.file_exists pending - then - Array.iter - (fun file -> - if Filename.check_suffix file ".part" - then unlink (Filename.concat pending file)) - (Sys.readdir pending); - let t = - { directory - ; budget = budget_bytes - ; maximum_file_bytes - ; records = Hashtbl.create 64 - ; handles = Hashtbl.create 64 - ; staged_records = Hashtbl.create 8 - ; staged_handles = Hashtbl.create 8 - ; closed = false - } - in - List.iter - (fun (file, (record : record)) -> - if - record.size <= Int64.of_int maximum_file_bytes - && Sys.file_exists (data_path t record) - then Hashtbl.replace t.records record.name record - else unlink (Filename.concat directory file)) - manifests; - Array.iter - (fun file -> - if Filename.check_suffix file ".part" - then unlink (Filename.concat directory file)) - (Sys.readdir directory); - let kept = Hashtbl.create (2 * Hashtbl.length t.records) in - Hashtbl.iter - (fun _ record -> - Hashtbl.replace kept (record.name ^ ".json") (); - Hashtbl.replace kept (record.name ^ "." ^ record.file_type) ()) - t.records; - Array.iter - (fun file -> - let path = Filename.concat directory file in - if (not (Hashtbl.mem kept file)) && not (Sys.is_directory path) - then unlink path) - (Sys.readdir directory); - Ok t)) - ;; - - type staged = - { file : string - ; checksum : string - ; size : int64 - } - - let pending_directory t = Filename.concat t.directory "pending" - - let valid_staged_type file_type = - Result.is_ok (Asset.version ~checksum:(String.make 64 '0') ~file_type) - ;; - - let valid_staged_nonce nonce = - String.length nonce = 64 - && String.for_all - (function - | '0' .. '9' | 'a' .. 'f' -> true - | _ -> false) - nonce - ;; - - let valid_staged_file file = - Filename.basename file = file - && - match String.split_on_char '.' file with - | [ operation; file_type ] -> - Result.is_ok (Uuid.of_string operation) && valid_staged_type file_type - | [ operation; nonce; file_type ] -> - Result.is_ok (Uuid.of_string operation) - && valid_staged_nonce nonce - && valid_staged_type file_type - | _ -> false - ;; - - let staged_path t ~file = - if t.closed || not (valid_staged_file file) - then None - else ( - let path = Filename.concat (pending_directory t) file in - match Unix.lstat path with - | { Unix.st_kind = Unix.S_REG; _ } -> Some path - | _ -> None - | exception Unix.Unix_error _ -> None) - ;; - - let retain_staged t ~file = - Option.bind (staged_path t ~file) (fun location -> - match Hashtbl.find_opt t.staged_records file with - | Some record when record.retired -> None - | Some record -> Some (pin_staged t record) - | None -> - let record = - { staged_name = file - ; staged_location = location - ; staged_pins = 0 - ; retired = false - } - in - Hashtbl.add t.staged_records file record; - Some (pin_staged t record)) - ;; - - let release_staged t ~file = - if t.closed - then Error Stale - else if not (valid_staged_file file) - then Error (Invalid "Invalid staged file") - else ( - let record = - match Hashtbl.find_opt t.staged_records file with - | Some record -> record - | None -> - { staged_name = file - ; staged_location = Filename.concat (pending_directory t) file - ; staged_pins = 0 - ; retired = true - } - in - record.retired <- true; - clean_retired_staging t record) - ;; - - let stage t ~operation ~file_type ~source_file ~pending_budget_bytes = - if t.closed - then Error Stale - else if not (valid_staged_type file_type) - then Error (Invalid "Invalid staged file type") - else if pending_budget_bytes < 0L - then Error (Invalid "Invalid staging budget") - else - protect (fun () -> - let directory = pending_directory t in - mkdir directory; - (* A delayed completion must name this physical instance, even if its - business operation is staged again after pruning or recovery. *) - let nonce = Asset_codec.checksum (Mirage_crypto_rng_unix.getrandom 16) in - let file = Uuid.to_string operation ^ "." ^ nonce ^ "." ^ file_type in - let target = Filename.concat directory file in - let partial = target ^ ".part" in - if Sys.file_exists target - then Error (Invalid "Staged import already exists") - else ( - let source = - Unix.openfile source_file [ Unix.O_RDONLY; Unix.O_NONBLOCK; Unix.O_CLOEXEC ] 0 - in - Fun.protect - ~finally:(fun () -> Unix.close source) - (fun () -> - let metadata = Unix.fstat source in - if metadata.st_kind <> Unix.S_REG - then Error (Invalid "Import source must be a regular file") - else if metadata.st_size > t.maximum_file_bytes - then Error (Invalid "Import exceeds file size limit") - else ( - let files = Sys.readdir directory in - let used = - Array.fold_left - (fun total name -> - let size = - (Unix.lstat (Filename.concat directory name)).st_size - in - Int64.add total (Int64.of_int (max 1 size))) - 0L - files - in - let allocation = Int64.of_int (max 1 metadata.st_size) in - if - Array.length files >= 4096 - || used > Int64.sub pending_budget_bytes allocation - then Error Full - else ( - let output = - Unix.openfile - partial - [ Unix.O_WRONLY; Unix.O_CREAT; Unix.O_EXCL; Unix.O_CLOEXEC ] - 0o600 - in - Fun.protect - ~finally:(fun () -> - Unix.close output; - unlink partial) - (fun () -> - let buffer = Bytes.create 65536 in - let rec copy count digest = - match Unix.read source buffer 0 (Bytes.length buffer) with - | 0 -> Ok (count, digest) - | length -> - if count + length > metadata.st_size - then Error (Invalid "Import source changed during staging") - else ( - let rec write offset = - if offset < length - then ( - let written = - Unix.write output buffer offset (length - offset) - in - if written = 0 - then raise (Sys_error "Staging write made no progress") - else write (offset + written)) - in - write 0; - copy - (count + length) - (Digestif.SHA256.feed_bytes - digest - ~off:0 - ~len:length - buffer)) - in - let* size, digest = copy 0 Digestif.SHA256.empty in - let after = Unix.fstat source in - if - size <> metadata.st_size - || after.st_mtime <> metadata.st_mtime - || after.st_size <> metadata.st_size - then Error (Invalid "Import source changed during staging") - else ( - Unix.fsync output; - (* Publish without replacing another physical instance. *) - Unix.link partial target; - unlink partial; - let fd = Unix.openfile directory [ Unix.O_RDONLY ] 0 in - Fun.protect - ~finally:(fun () -> Unix.close fd) - (fun () -> Unix.fsync fd); - Ok - { file - ; checksum = - Digestif.SHA256.to_hex (Digestif.SHA256.get digest) - ; size = Int64.of_int size - }))))))) - ;; - - let prune_staged t ~keep = - if t.closed - then Error Stale - else - protect (fun () -> - let directory = pending_directory t in - if not (Sys.file_exists directory) - then Ok 0 - else ( - let stream = Unix.opendir directory in - let rec inspect count orphaned = - match Unix.readdir stream with - | "." | ".." -> inspect count orphaned - | _ when count >= 4096 -> Error Full - | file -> - let path = Filename.concat directory file in - if - (not (valid_staged_file file)) - || - match Hashtbl.find_opt t.staged_records file with - | Some record -> record.staged_pins > 0 - | None -> false - then inspect (count + 1) orphaned - else ( - match Unix.lstat path with - | { Unix.st_kind = Unix.S_REG; _ } -> - (match keep file with - | Error message -> Error (Io message) - | Ok true -> inspect (count + 1) orphaned - | Ok false -> inspect (count + 1) (file :: orphaned)) - | _ -> inspect (count + 1) orphaned - | exception Unix.Unix_error (Unix.ENOENT, _, _) -> - inspect (count + 1) orphaned) - | exception End_of_file -> Ok orphaned - in - match - Fun.protect ~finally:(fun () -> Unix.closedir stream) (fun () -> inspect 0 []) - with - | Error _ as error -> error - | Ok orphaned -> - List.iter (fun file -> unlink (Filename.concat directory file)) orphaned; - if orphaned <> [] - then ( - let fd = Unix.openfile directory [ Unix.O_RDONLY ] 0 in - Fun.protect ~finally:(fun () -> Unix.close fd) (fun () -> Unix.fsync fd)); - Ok (List.length orphaned))) - ;; -end diff --git a/logseq_sync/lib/effect_runner/storage/bootstrap.mli b/logseq_sync/lib/effect_runner/storage/bootstrap.mli deleted file mode 100644 index 8efe1e25..00000000 --- a/logseq_sync/lib/effect_runner/storage/bootstrap.mli +++ /dev/null @@ -1,113 +0,0 @@ -type progress = - { received_bytes : int - ; total_bytes : int option - ; datom_count : int option - } - -val artifact_row_count : (string * string) list -> (int, string) result - -val peel_gzip_layers - : decompress_gzip:(string -> string -> int -> int) - -> maximum_bytes:int - -> source:string - -> destination:string - -> temporary_paths:string list - -> (unit, string) result - -val cleanup : string list -> unit - -(** Private storage implementation, interpreted only by runner submission. *) -module Asset_cache : sig - (** Verified files are local state, scoped by origin, account and graph. Calls are - serialized by the effect runner; handles pin files until explicitly released. *) - type t - - type handle = string - - type error = - | Invalid of string - | Io of string - | Full - | Stale - | Checksum_mismatch - - val create - : root:string - -> scope:Logseq_sync_pure_reducer.Core.graph_scope - -> budget_bytes:int64 - -> maximum_file_bytes:int - -> (t, error) result - - val lookup - : t - -> asset:Logseq_db_types.Graph_types.Uuid.t - -> version:Logseq_db_types.Asset_descriptor.version - -> (handle option, error) result - - val publish - : t - -> asset:Logseq_db_types.Graph_types.Uuid.t - -> current:(unit -> bool) - -> verified:Asset_codec.verified - -> (handle, error) result - - val path : t -> handle -> string option - val retain : t -> handle -> handle option - val release : t -> handle -> unit - val release_checked : t -> handle -> (unit, error) result - (** Preserve all retained handles and failed cleanup ownership on error. *) - val close_checked : t -> (unit,error) result - val close : t -> unit - - (** Remove durable files for one origin/account after its live caches are closed. *) - val delete_account - : root:string - -> account:Logseq_sync_pure_reducer.Core.account_scope - -> (unit, error) result - - val graph_deletion_terminal - : root:string -> scope:Logseq_sync_pure_reducer.Core.graph_scope - -> retirement:Logseq_sync_pure_reducer.Core.retirement_id - -> ([ `Committed | `Aborted ] option, error) result - - val prepare_graph_deletion - : root:string - -> scope:Logseq_sync_pure_reducer.Core.graph_scope - -> retirement:Logseq_sync_pure_reducer.Core.retirement_id - -> (unit, error) result - - val finish_graph_deletion - : root:string - -> deletion:Logseq_sync_pure_reducer.Core.asset_deletion - -> commit:bool - -> (unit, error) result - - type staged = private - { file : string - ; checksum : string - ; size : int64 - } - - (** Copy an explicit import into the durable, non-evictable pending namespace. - Each copy has a distinct physical file identity, including copies of the - same operation. Legacy operation-only filenames remain readable. - The caller persists the returned identity before graph mutation or upload. *) - val stage - : t - -> operation:Logseq_db_types.Graph_types.Uuid.t - -> file_type:string - -> source_file:string - -> pending_budget_bytes:int64 - -> (staged, error) result - - (** Retain a staged file for a local preview. Completion cleanup waits for all leases. *) - val retain_staged : t -> file:string -> handle option - - val staged_path : t -> file:string -> string option - val release_staged : t -> file:string -> (unit, error) result - - (** Remove recognized orphan staging files only after all ownership checks succeed. - Ownership checks receive the exact physical file identity, including legacy - filenames. The caller must serialize this with staging and intent persistence. *) - val prune_staged : t -> keep:(string -> (bool, string) result) -> (int, error) result -end diff --git a/logseq_sync/lib/execute/core.ml b/logseq_sync/lib/execute/core.ml new file mode 100644 index 00000000..287298eb --- /dev/null +++ b/logseq_sync/lib/execute/core.ml @@ -0,0 +1,2693 @@ +module Dsl = Sync_dsl.Core +module Owner = Logseq_sync_execute_owners.Database_owner + +let ( let* ) = Result.bind + +type snapshot_source = + { path : string + ; expected_rows : int + } + +type database_owner = + { binding : Owner.t + ; snapshot_source : (Dsl.snapshot -> (snapshot_source, Dsl.error) result) option + ; mutable bound : bool + } + +type callback_control = + { stop : unit -> (unit, Dsl.error) result + ; await_stopped : deadline:float -> (unit, Dsl.error) result + } + +type effect_scope = + { check : unit -> (unit, Dsl.error) result + ; admit : unit -> (unit, Dsl.error) result + ; transfer : Dsl.resource_reference Rrbvec.t -> (unit, Dsl.error) result + ; reserve_producer : callback_control -> (unit, Dsl.error) result + ; forget_producer : callback_control -> unit + ; retain_failure : Dsl.error -> unit + ; now : unit -> float + ; cleanup_seconds : float + ; mutable live : bool + } + +let check_current scope = if scope.live then scope.check () else Error Dsl.Stale_context + +let admit_network scope = + let* () = check_current scope in + scope.admit () +;; + +let handoff scope resources = scope.transfer resources + +type dispatch = + { database : + 'a. + owner:database_owner + -> scope:effect_scope + -> identity:Dsl.identity + -> 'a Dsl.database + -> ('a, Dsl.error) result + ; transport : + 'a. + scope:effect_scope + -> identity:Dsl.identity + -> 'a Dsl.transport + -> ('a, Dsl.error) result + ; crypto : + 'a. + scope:effect_scope + -> identity:Dsl.identity + -> 'a Dsl.crypto + -> ('a, Dsl.error) result + ; assets : + 'a. + scope:effect_scope + -> identity:Dsl.identity + -> 'a Dsl.assets + -> ('a, Dsl.error) result + } + +type websocket_liveness = + | Peer_close_only + | Ping_pong of + { interval_seconds : float + ; timeout_seconds : float + } + +type network_deadlines = + { setup_seconds : float + ; tls_handshake_seconds : float + ; http_response_seconds : float + ; download_seconds : float + } + +type worker_network = + { with_admission : 'a. (unit -> ('a, Dsl.error) result) -> ('a, Dsl.error) result } + +module type WORKER_OWNER = sig + type scope + type imported_file + + val bind_runtime + : identity:Dsl.identity + -> database_owner:database_owner + -> limits:Dsl.limits + -> cleanup_seconds:float + -> network:worker_network + -> (unit, Dsl.error) result + + val with_scope + : identity:Dsl.identity + -> operation:Logseq_db_types.Graph_types.Uuid.t + -> (scope -> ('a, Dsl.error) result) + -> ('a, Dsl.error) result + + val crypto : scope -> 'a Dsl.crypto -> ('a, Dsl.error) result + + val prepare_import + : scope + -> Logseq_db_types.Asset_import.t + -> current:(unit -> bool) + -> (Logseq_db_types.Asset_upload_intent.t, Dsl.error) result + + val publish_import_preview + : scope + -> Logseq_db_types.Asset_upload_intent.t + -> (unit, Dsl.error) result + + val persist_import + : scope + -> Logseq_db_types.Asset_upload_intent.t + -> expected:int option + -> (unit, Dsl.error) result + + val inspect_import + : scope + -> Logseq_db_types.Asset_upload_intent.t + -> ( [ `Import_absent + | `Local_present + | `Metadata_present + | `Publication_acknowledged + | `Entity_cancelled + ] + , Dsl.error ) + result + + val apply_local_import + : scope + -> Logseq_db_types.Asset_upload_intent.t + -> (unit, Dsl.error) result + + val put_import + : scope + -> Logseq_db_types.Asset_upload_intent.t + -> (unit, Dsl.error) result + + val apply_import_metadata + : scope + -> Logseq_db_types.Asset_upload_intent.t + -> (unit, Dsl.error) result + + val await_import_publication + : scope + -> Logseq_db_types.Asset_upload_intent.t + -> (unit, Dsl.error) result + + val cancel_import + : scope + -> Logseq_db_types.Asset_upload_intent.t + -> (unit, Dsl.error) result + + val release_import_staging + : scope + -> Logseq_db_types.Asset_upload_intent.t + -> (unit, Dsl.error) result + + val recover_import_inputs + : scope + -> after:int64 option + -> maximum_items:int + -> maximum_bytes:int + -> (Logseq_db_types.Asset_import.Recovery.page, Dsl.error) result + + val confirm_import_input + : scope + -> string + -> (Logseq_db_types.Asset_import.Recovery.item option, Dsl.error) result + + val acknowledge_import_input + : scope + -> input:string + -> prepared:string + -> (unit, Dsl.error) result + + val cancel : scope -> (unit, Dsl.error) result + + val retain_imported_file + : scope + -> Logseq_db_types.Asset_upload_intent.t + -> (imported_file option, Dsl.error) result + + val imported_file_path : imported_file -> string + val release_imported_file : imported_file -> (unit, Dsl.error) result + val fence : identity:Dsl.identity -> (unit, Dsl.error) result + val await_quiescent : identity:Dsl.identity -> (unit, Dsl.error) result +end + +type capabilities = + { validate_identity : Dsl.identity -> (unit, Dsl.error) result + ; validate_limits : Dsl.limits -> (unit, Dsl.error) result + ; websocket_liveness : websocket_liveness + ; network_deadlines : network_deadlines + ; cleanup_seconds : float + ; acquire : Dsl.identity -> (unit, Dsl.error) result + ; fence_network : Dsl.identity -> (unit, Dsl.error) result + ; close_network : Dsl.identity -> (unit, Dsl.error) result + ; authorize_network : Dsl.identity -> epoch:int64 -> (unit, Dsl.error) result + ; validate_resource : Dsl.resource_reference -> (unit, Dsl.error) result + ; restore : scope:effect_scope -> Dsl.identity -> (Dsl.restore_report, Dsl.error) result + ; worker_owner : (module WORKER_OWNER) + ; dispatch : dispatch + ; observe : Dsl.identity -> (Dsl.observation, Dsl.error) result + ; fence : Dsl.identity -> (unit, Dsl.error) result + ; retire : Dsl.identity -> (unit, Dsl.error) result + ; release_file : Dsl.identity -> Dsl.asset_file -> (unit, Dsl.error) result + ; release_lease : Dsl.identity -> Dsl.asset_lease -> (unit, Dsl.error) result + ; release_snapshot : Dsl.identity -> Dsl.snapshot -> (unit, Dsl.error) result + } + +type version = unit ref + +type t = + { sw : Eio.Switch.t + ; now : unit -> float + ; sleep : float -> unit + ; id : Dsl.identity + ; limits : Dsl.limits + ; caps : capabilities + ; owner : database_owner + ; producer : Dsl.Producer.owner + ; changed : Eio.Condition.t + ; mutable actual : context option + ; mutable revision : int64 + ; mutable current_version : version + ; mutable busy : unit ref option + ; mutable fenced : bool + ; mutable gate : Dsl.network_gate + ; mutable generation : int64 + ; mutable retirement : network_retirement option + ; mutable initial_eligible : bool + ; mutable network_calls : network_call list + ; mutable producers : callback_control list + ; mutable resources : resource_entry list + ; mutable reserved_resources : int + ; mutable cleanup : Dsl.error list + ; mutable active : task_scope option + ; mutable scopes : task_scope list + ; mutable tasks : int + ; mutable asset_work : asset_work list + ; mutable capacity : int64 + ; mutable stop : bool + ; mutable network_input : Dsl.network_gate option + ; mutable unlock : unlock option + ; mutable demands : demand_entry list + ; mutable deliveries : delivery_entry list + ; mutable receiver : bool + ; mutable input_reader : bool + ; mutable outbox_epoch : int64 + ; mutable outbox_acked : int64 + ; mutable outbox_check : (Dsl.outbox_check * int64) option + ; mutable keys : Dsl.key_status + ; mutable status : Dsl.service_status + ; mutable last_failure : Dsl.error option + ; mutable observation : Dsl.observation option + ; mutable observation_reader : bool + } + +and 's graph_context = + { runtime : t + ; phase : 's Dsl.phase + ; version : version + } + +and context = Context : 's Dsl.phase * 's graph_context -> context + +and network_retirement = + { runtime : t + ; generation : int64 + ; fence_result : (unit, Dsl.error) result + ; mutable result : (unit, Dsl.error) result option + ; mutable joining : bool + } + +and network_call = + { generation : int64 + ; mutable cancel_network : (unit -> unit) option + ; done_ : unit Eio.Promise.t + } + +and resource_entry = + { reference : Dsl.resource_reference + ; mutable released : bool + ; mutable delivery_owned : bool + ; mutable releasing : bool + } + +and task_scope = + { handle : Dsl.scope + ; mutable live : bool + ; mutable children : child_control list + ; mutable notifications : notification_channel list + ; mutable prefer_child : bool + } + +and asset_work = + { asset : Dsl.asset + ; mutable interests : (unit ref * Dsl.demand_change option) list + ; mutable download : Dsl.asset_result asset_attempt option + ; mutable download_files : Dsl.asset_file list + ; mutable upload : unit asset_attempt option + } + +and 'a asset_attempt = + { outcome : ('a, Dsl.error) result Eio.Promise.t + ; mutable waiters : int + ; mutable cancellation : Eio.Cancel.t option + ; mutable physically_joined : bool + } + +and child_control = + { key : Obj.t + ; cancel_child : unit -> unit + ; join_child : unit -> Dsl.error option + ; mutable consumed : bool + ; release_capacity : unit -> unit + } + +and notification_channel = + { handle : Dsl.notifications + ; queue : (Dsl.notification * int) Queue.t + ; mutable reading : bool + } + +and unlock = + { mutable receipt : Dsl.unlock_receipt + ; input : Dsl.unlock_input + ; expires : float + ; mutable queued : bool + } + +and demand_entry = + { mutable receipt : Dsl.demand_receipt + ; mutable scope : task_scope + ; change : Dsl.demand_change + ; bytes : int + ; mutable queued : bool + ; mutable finished : bool + ; mutable download_waiting_capacity : bool + } + +and delivery_entry = + { output : Dsl.demand_output + ; source : demand_entry + ; references : resource_entry list + ; mutable received : bool + ; mutable released : bool + } + +type ('after, 'value) outcome = + | Completed of 'after graph_context * 'value + | Failed of context * Dsl.error + +type ('after, 'value) running = + { runtime : t + ; outcome_promise : ('after, 'value) outcome Eio.Promise.t + ; mutable cancellation : Eio.Cancel.t option + ; mutable cancelled : bool + ; mutable terminal : bool + } + +type ingress_ack = + | Queued + | Replaced + +type delivery_status = + | Current + | Revoked + +type observation_revision = + { owner : t + ; sequence : int64 + } + +type runtime_observation = + { revision : observation_revision + ; state : Dsl.observation + ; status : Dsl.service_status + } + +exception Forced of Dsl.error +exception Cancel_execution +exception Close_attempt + +let protect_result f = + try f () with + | Eio.Cancel.Cancelled _ as exn -> raise exn + | Forced _ as exn -> raise exn + | _ -> Error (Dsl.Unexpected_failure "Capability raised") +;; + +let pulse (t : t) = + t.revision <- Int64.succ t.revision; + Eio.Condition.broadcast t.changed +;; + +let capacity_changed (t : t) = + t.capacity <- Int64.succ t.capacity; + pulse t +;; + +let mark_dirty (t : t) = + t.outbox_epoch <- Int64.succ t.outbox_epoch; + pulse t +;; + +let cleanup_error ?primary failures = + Dsl.Cleanup_failure { primary; failures = Rrbvec.of_list failures } +;; + +let record_cleanup (t : t) error = + t.cleanup <- error :: t.cleanup; + t.fenced <- true; + t.gate <- Dsl.Closed; + let next = { runtime = t; phase = Dsl.Draining; version = t.current_version } in + t.actual <- Some (Context (Dsl.Draining, next)); + t.last_failure <- Some error; + pulse t +;; + +let set_phase (t : t) phase = + let next = { runtime = t; phase; version = t.current_version } in + t.actual <- Some (Context (phase, next)); + pulse t; + next +;; + +let actual (t : t) = + match t.actual with + | Some c -> c + | None -> assert false +;; + +let phase c = c.phase +let identity (c : _ graph_context) = c.runtime.id +let version c = c.version +let equal_version a b = a == b +let finite_positive n = Float.is_finite n && n > 0. + +let database_owner database = + { binding = Owner.database_owner database; snapshot_source = None; bound = false } +;; + +let database_if_open owner = Owner.database_if_open owner.binding +let database owner = Owner.database owner.binding + +let activation_owner ~sw ~snapshot_source ~dependencies ~inspection ~graph_name = + if String.trim graph_name = "" + then Error (Dsl.Invalid_input "Empty graph name") + else + Ok + { binding = Owner.activation_owner ~sw ~dependencies ~inspection ~graph_name + ; snapshot_source = Some snapshot_source + ; bound = false + } +;; + +let wait (t : t) poll = Eio.Condition.loop_no_mutex t.changed poll + +let still_current (t : t) token = + if + t.fenced + || not + (match t.busy with + | Some current -> current == token + | None -> false) + then Error Dsl.Stale_context + else Ok () +;; + +let reference_resource = function + | Dsl.Snapshot_reference s -> Some s.resource + | Dsl.File_reference f -> Some f.file + | Dsl.Lease_reference l -> Some l.lease + | Dsl.Connection_reference _ -> None +;; + +let same_reference a b = Dsl.Producer.same_reference_registration a b && a = b + +let find_resource (t : t) reference = + List.find_opt (fun e -> same_reference reference e.reference) t.resources +;; + +let validate_reference (t : t) reference = + let* () = + match reference_resource reference with + | Some r when r.owner <> t.id -> Error (Dsl.Identity_mismatch "Foreign resource") + | _ -> Ok () + in + protect_result (fun () -> t.caps.validate_resource reference) +;; + +let release_entry (t : t) entry = + if entry.released + then Ok () + else if entry.releasing + then Error (Dsl.Capacity_exceeded "Resource release in progress") + else ( + entry.releasing <- true; + Fun.protect + ~finally:(fun () -> entry.releasing <- false) + (fun () -> + let result = + Eio.Cancel.protect (fun () -> + protect_result (fun () -> + match entry.reference with + | Dsl.File_reference file -> t.caps.release_file t.id file + | Dsl.Lease_reference lease -> t.caps.release_lease t.id lease + | Dsl.Snapshot_reference snapshot -> t.caps.release_snapshot t.id snapshot + | Dsl.Connection_reference connection -> + let scope = + { check = (fun () -> Ok ()) + ; admit = (fun () -> Error Dsl.Network_closed) + ; transfer = (fun _ -> Error Dsl.Stale_context) + ; reserve_producer = (fun _ -> Error Dsl.Stale_context) + ; forget_producer = (fun _ -> ()) + ; retain_failure = record_cleanup t + ; now = t.now + ; cleanup_seconds = t.caps.cleanup_seconds + ; live = true + } + in + Fun.protect + ~finally:(fun () -> scope.live <- false) + (fun () -> + t.caps.dispatch.transport + ~scope + ~identity:t.id + (Dsl.Retire_connection connection)))) + in + match result with + | Ok () -> + let* () = Dsl.Producer.acknowledge_reference t.producer entry.reference in + entry.released <- true; + t.resources <- List.filter (fun current -> current != entry) t.resources; + capacity_changed t; + Ok () + | Error error -> + record_cleanup t error; + Error error)) +;; + +let release_reference (t : t) reference = + match find_resource t reference with + | None -> + let* state = Dsl.Producer.reference_state t.producer reference in + (match state with + | `Released -> Ok () + | `Retained | `Unregistered -> Error Dsl.Stale_context) + | Some e when e.delivery_owned -> + Error (Dsl.Invalid_input "Resource belongs to delivery") + | Some e -> release_entry t e +;; + +let transfer (t : t) reserved ~ceiling references = + let references = Rrbvec.to_list references in + let rec validate seen new_count = function + | [] -> Ok new_count + | reference :: rest -> + let* () = validate_reference t reference in + let* state = Dsl.Producer.reference_state t.producer reference in + let* () = + match state with + | `Released -> Error Dsl.Stale_context + | `Retained | `Unregistered -> Ok () + in + if List.exists (Dsl.Producer.same_reference_registration reference) seen + then Error (Dsl.Invalid_input "Duplicate handoff") + else ( + let count = + match find_resource t reference with + | None -> new_count + 1 + | Some _ -> new_count + in + validate (reference :: seen) count rest) + in + let* count = validate [] 0 references in + let live = List.fold_left (fun n e -> n + if e.released then 0 else 1) 0 t.resources in + let consumed = min count !reserved in + if live + count + List.length t.producers + t.reserved_resources - consumed > ceiling () + then Error (Dsl.Capacity_exceeded "Resource ledger") + else ( + let rec register = function + | [] -> Ok () + | reference :: rest -> + let* () = Dsl.Producer.register_reference t.producer reference in + (match find_resource t reference with + | Some _ -> () + | None -> + t.resources + <- { reference; released = false; delivery_owned = false; releasing = false } + :: t.resources); + register rest + in + let* () = register references in + reserved := !reserved - consumed; + t.reserved_resources <- t.reserved_resources - consumed; + pulse t; + Ok ()) +;; + +let with_network (t : t) check callback = + let* () = check () in + match t.gate with + | Dsl.Closed -> Error Dsl.Network_closed + | Dsl.Open { authorization_epoch } -> + let generation = t.generation in + let* () = + protect_result (fun () -> t.caps.authorize_network t.id ~epoch:authorization_epoch) + in + let* () = check () in + if generation <> t.generation || t.gate <> Dsl.Open { authorization_epoch } + then Error Dsl.Network_closed + else ( + let done_, resolve = Eio.Promise.create () in + let call = { generation; cancel_network = None; done_ } in + t.network_calls <- call :: t.network_calls; + Fun.protect + ~finally:(fun () -> + Eio.Promise.resolve resolve (); + t.network_calls <- List.filter (fun c -> c != call) t.network_calls; + pulse t) + (fun () -> + try + Eio.Cancel.sub (fun cc -> + call.cancel_network <- Some (fun () -> Eio.Cancel.cancel cc Close_attempt); + let* () = check () in + if generation <> t.generation + then Error Dsl.Network_closed + else callback ()) + with + | Eio.Cancel.Cancelled Close_attempt -> Error Dsl.Network_closed)) +;; + +let close_network (t : t) = + match t.retirement with + | Some receipt -> Ok receipt + | None -> + t.initial_eligible <- false; + let generation = t.generation in + t.gate <- Dsl.Closed; + t.generation <- Int64.succ t.generation; + t.network_input <- Some Dsl.Closed; + pulse t; + let fence_result = protect_result (fun () -> t.caps.fence_network t.id) in + let receipt = + { runtime = t; generation; fence_result; result = None; joining = false } + in + t.retirement <- Some receipt; + List.iter + (fun (c : network_call) -> + if c.generation = generation then Option.iter (fun f -> f ()) c.cancel_network) + t.network_calls; + Ok receipt +;; + +let await_network_retirement (receipt : network_retirement) = + let t = receipt.runtime in + match receipt.result with + | Some result -> result + | None when receipt.joining -> wait t (fun () -> receipt.result) + | None -> + receipt.joining <- true; + Eio.Cancel.protect (fun () -> + let failures = ref [] in + let add = function + | Ok () -> () + | Error e -> failures := e :: !failures + in + add receipt.fence_result; + List.iter + (fun (c : network_call) -> + if c.generation = receipt.generation then Eio.Promise.await c.done_) + t.network_calls; + add (protect_result (fun () -> t.caps.close_network t.id)); + let result = + match List.rev !failures with + | [] -> Ok () + | xs -> Error (cleanup_error xs) + in + receipt.result <- Some result; + (match result with + | Ok () -> () + | Error e -> record_cleanup t e); + pulse t; + result) +;; + +let open_network (t : t) ~authorization_epoch = + t.initial_eligible <- false; + if t.fenced + then Error Dsl.Stale_context + else ( + match t.retirement with + | Some { result = None; _ } -> Error Dsl.Network_closed + | Some { result = Some (Error e); _ } -> Error e + | None | Some { result = Some (Ok ()); _ } -> + let* () = + protect_result (fun () -> + t.caps.authorize_network t.id ~epoch:authorization_epoch) + in + if t.fenced + then Error Dsl.Stale_context + else ( + t.retirement <- None; + t.generation <- Int64.succ t.generation; + t.gate <- Dsl.Open { authorization_epoch }; + t.network_input <- Some t.gate; + mark_dirty t; + Ok ())) +;; + +let make_effect_scope (t : t) token acquired ~ceiling = + let check () = still_current t token in + let scope = + { check + ; admit = (fun () -> with_network t check (fun () -> Ok ())) + ; transfer = transfer t acquired ~ceiling + ; reserve_producer = + (fun producer -> + let* () = check () in + if + List.length t.producers + + List.fold_left + (fun count entry -> count + if entry.released then 0 else 1) + 0 + t.resources + + t.reserved_resources + - min 1 !acquired + >= ceiling () + then Error (Dsl.Capacity_exceeded "Producer ledger") + else ( + let consumed = min 1 !acquired in + acquired := !acquired - consumed; + t.reserved_resources <- t.reserved_resources - consumed; + t.producers <- producer :: t.producers; + Ok ())) + ; forget_producer = + (fun producer -> + t.producers <- List.filter (fun entry -> entry != producer) t.producers; + capacity_changed t) + ; retain_failure = record_cleanup t + ; now = t.now + ; cleanup_seconds = t.caps.cleanup_seconds + ; live = true + } + in + scope, acquired +;; + +let with_effect ?(reserve = 0) ?(background = fun () -> false) (t : t) token callback = + let ceiling () = + t.limits.maximum_resources - if background () then t.limits.foreground_reserve else 0 + in + if + reserve > 0 + && List.length t.resources + List.length t.producers + t.reserved_resources + reserve + > ceiling () + then Error (Dsl.Capacity_exceeded "Effect resources") + else ( + let acquired = ref reserve in + t.reserved_resources <- t.reserved_resources + reserve; + let scope, _ = make_effect_scope t token acquired ~ceiling in + Fun.protect + ~finally:(fun () -> + scope.live <- false; + if t.cleanup = [] + then ( + t.reserved_resources <- t.reserved_resources - !acquired; + if !acquired > 0 then capacity_changed t)) + (fun () -> + let* () = check_current scope in + let result = protect_result (fun () -> callback scope) in + match result with + | Error (Dsl.Cleanup_failure _ as e) -> + record_cleanup t e; + raise (Forced e) + | result -> result)) +;; + +module Callback_bridge = struct + type 'a producer = + { start : + deliver:(('a, Dsl.error) result -> (unit, Dsl.error) result) + -> (unit, Dsl.error) result + ; stop : unit -> (unit, Dsl.error) result + ; await_stopped : deadline:float -> (unit, Dsl.error) result + } + + let run ~scope ~resources producer = + let control = { stop = producer.stop; await_stopped = producer.await_stopped } in + let* () = check_current scope in + let* () = scope.reserve_producer control in + let terminal, resolve = Eio.Promise.create () in + let lock = Mutex.create () in + let selected = ref false in + let deliver result = + Mutex.lock lock; + Fun.protect + ~finally:(fun () -> Mutex.unlock lock) + (fun () -> + let transferred = + match result with + | Ok value -> handoff scope (resources value) + | Error _ -> Ok () + in + let terminal_result = + match transferred with + | Ok () -> result + | Error error -> Error error + in + if not !selected + then ( + selected := true; + Eio.Promise.resolve resolve terminal_result); + transferred) + in + let cleanup primary = + Eio.Cancel.protect (fun () -> + let deadline = scope.now () +. scope.cleanup_seconds in + let failures = ref [] in + let add = function + | Ok () -> () + | Error e -> failures := e :: !failures + in + add (protect_result control.stop); + add (protect_result (fun () -> control.await_stopped ~deadline)); + match List.rev !failures with + | [] -> + scope.forget_producer control; + Ok () + | xs -> + let e = cleanup_error ?primary xs in + scope.retain_failure e; + Error e) + in + let result = + try + let result = + match protect_result (fun () -> producer.start ~deliver) with + | Error e -> Error e + | Ok () -> Eio.Promise.await terminal + in + let joined = + cleanup + (match result with + | Error e -> Some e + | Ok _ -> None) + in + match joined with + | Ok () -> result + | Error e -> Error e + with + | exn -> + (match cleanup (Some Dsl.Cancelled) with + | Ok () -> raise exn + | Error e -> raise (Forced e)) + in + result + ;; +end + +let notify_outbox_changed (t : t) ~owner ~identity = + if owner != t.owner || identity <> t.id + then Error (Dsl.Identity_mismatch "Foreign database binding") + else if t.fenced + then Error Dsl.Stale_context + else ( + mark_dirty t; + Ok ()) +;; + +let active_scope (t : t) = + match t.active with + | Some s when s.live && not t.fenced -> Ok s + | _ -> Error Dsl.Stale_context +;; + +let submit_unlock (t : t) ~identity ~valid_for input = + let* scope = active_scope t in + if identity <> t.id + then Error (Dsl.Identity_mismatch "Foreign unlock identity") + else if (not (finite_positive valid_for)) || valid_for > 300. + then Error (Dsl.Invalid_input "Unlock lifetime") + else if String.length (Dsl.wrapped_graph_key input) > t.limits.maximum_message_bytes + then Error (Dsl.Capacity_exceeded "Unlock bytes") + else + let* receipt = Dsl.Producer.unlock_receipt t.producer scope.handle in + let ack = if Option.is_some t.unlock then Replaced else Queued in + t.unlock <- Some { receipt; input; expires = t.now () +. valid_for; queued = true }; + pulse t; + Ok ack +;; + +let demand_id = function + | Dsl.Demand d -> d.id + | Dsl.Withdraw id -> id +;; + +let same_demand a b = Dsl.compare_demand_id (demand_id a) (demand_id b) = 0 + +let notification_usage (t : t) = + List.fold_left + (fun (count, bytes) scope -> + List.fold_left + (fun (count, bytes) channel -> + Queue.fold + (fun (count, bytes) (_, size) -> count + 1, bytes + size) + (count, bytes) + channel.queue) + (count, bytes) + scope.notifications) + (0, 0) + t.scopes +;; + +let demand_usage (t : t) = + List.fold_left + (fun (count, bytes) demand -> count + 1, bytes + demand.bytes) + (0, 0) + t.demands +;; + +let descriptor_bytes = function + | Dsl.Withdraw _ -> 256 + | Dsl.Demand d -> + let module A = Logseq_db_types.Asset_descriptor in + 256 + + (match d.asset.source with + | A.External s -> String.length s + | A.Managed None -> 0 + | A.Managed (Some v) -> String.length v.checksum + String.length v.file_type) + + + (match d.asset.current_checksum with + | None -> 0 + | Some s -> String.length s) +;; + +let background = function + | Dsl.Demand { priority = Dsl.Background; _ } -> true + | _ -> false +;; + +let asset_binding : (t * asset_work) Eio.Fiber.key = Eio.Fiber.create_key () + +let work_background ?replacement demands work = + List.for_all + (fun (_, original) -> + match original with + | None -> false + | Some change -> + let current = + List.find_opt + (fun d -> (not d.finished) && same_demand d.change change) + demands + in + let selected = + match replacement with + | Some replacement when same_demand replacement change -> replacement + | _ -> + (match current with + | Some entry -> entry.change + | None -> change) + in + background + (match selected with + | Dsl.Demand demand when demand.asset = work.asset -> Dsl.Demand demand + | _ -> change)) + work.interests +;; + +let attempt_active = function + | Some attempt -> not attempt.physically_joined + | None -> false +;; + +let work_slots work = + max + 1 + ((if attempt_active work.download then 1 else 0) + + if attempt_active work.upload then 1 else 0) +;; + +let asset_slots work = List.fold_left (fun count work -> count + work_slots work) 0 work + +let asset_background_count demands work = + List.fold_left + (fun count entry -> + count + if work_background demands entry then work_slots entry else 0) + 0 + work +;; + +let reserve_asset (t : t) asset change = + let shared = + match change with + | None -> None + | Some _ -> + List.find_opt + (fun work -> + work.asset = asset + && List.for_all (fun (_, change) -> Option.is_some change) work.interests) + t.asset_work + in + let interest = ref () in + let work = + match shared with + | Some work -> work + | None -> + { asset; interests = []; download = None; download_files = []; upload = None } + in + let old = work.interests in + work.interests <- (interest, change) :: old; + let proposed = + match shared with + | Some _ -> t.asset_work + | None -> work :: t.asset_work + in + if + asset_slots proposed > t.limits.maximum_active_assets + || asset_background_count t.demands proposed + > t.limits.maximum_active_assets - t.limits.foreground_reserve + then ( + work.interests <- old; + Error (Dsl.Capacity_exceeded "Active assets")) + else ( + t.asset_work <- proposed; + let release () = + if t.cleanup = [] + then ( + work.interests <- List.filter (fun (key, _) -> key != interest) work.interests; + if work.interests = [] + then t.asset_work <- List.filter (fun entry -> entry != work) t.asset_work; + capacity_changed t) + in + Ok (work, release)) +;; + +let update_demand (t : t) change = + let* scope = active_scope t in + let previous = List.find_opt (fun d -> same_demand d.change change) t.demands in + if + previous = None + && + match change with + | Dsl.Withdraw _ -> true + | _ -> false + then Ok Queued + else ( + let bytes = descriptor_bytes change in + let other = List.filter (fun d -> not (same_demand d.change change)) t.demands in + let count = List.length other + 1 in + let background_count = + List.fold_left (fun n d -> n + if background d.change then 1 else 0) 0 other + in + let notification_count, notification_bytes = notification_usage t in + let used_bytes = List.fold_left (fun n d -> n + d.bytes) notification_bytes other in + let demoting = + background change + && Option.fold + ~none:false + ~some:(fun previous -> not (background previous.change)) + previous + in + let resources = + List.fold_left + (fun count entry -> count + if entry.released then 0 else 1) + 0 + t.resources + + List.length t.producers + + t.reserved_resources + in + if + (demoting + && (t.tasks + > min t.limits.maximum_tasks (t.limits.maximum_pending - 3) + - t.limits.foreground_reserve + || resources > t.limits.maximum_resources - t.limits.foreground_reserve)) + || List.fold_left + (fun count work -> + count + + + if work_background ~replacement:change t.demands work + then work_slots work + else 0) + 0 + t.asset_work + > t.limits.maximum_active_assets - t.limits.foreground_reserve + || count > t.limits.maximum_demands + || count + notification_count > t.limits.maximum_messages + || (background change + && background_count + 1 + > t.limits.maximum_demands - t.limits.foreground_demand_reserve) + || used_bytes + bytes > t.limits.maximum_message_bytes + || (background change + && List.fold_left + (fun total d -> total + if background d.change then d.bytes else 0) + (bytes + notification_bytes) + other + > t.limits.maximum_message_bytes + - t.limits.foreground_descriptor_reserve_bytes) + then Error (Dsl.Capacity_exceeded "Demand descriptors") + else + let* receipt = Dsl.Producer.demand_receipt t.producer scope.handle in + let entry = + { receipt + ; scope + ; change + ; bytes + ; queued = true + ; finished = false + ; download_waiting_capacity = false + } + in + t.demands + <- entry :: List.filter (fun d -> not (same_demand d.change change)) t.demands; + pulse t; + Ok + (match previous with + | None -> Queued + | Some _ -> Replaced)) +;; + +let current_demand (t : t) receipt = + match + List.find_opt + (fun (d : demand_entry) -> d.receipt == receipt && d.scope.live) + t.demands + with + | Some d -> Ok d + | None -> Error Dsl.Stale_context +;; + +let delivery_current (t : t) entry = + (not t.fenced) + && entry.source.scope.live + && List.exists (fun d -> d == entry.source) t.demands +;; + +let find_delivery (t : t) delivery = + let* () = Dsl.Producer.check_delivery t.producer delivery in + match List.find_opt (fun d -> d.output.delivery == delivery) t.deliveries with + | Some d -> Ok d + | None -> Error Dsl.Stale_context +;; + +let demand_status (t : t) delivery = + let* released = Dsl.Producer.delivery_released t.producer delivery in + if released + then Ok Revoked + else + let* d = find_delivery t delivery in + Ok (if delivery_current t d then Current else Revoked) +;; + +let release_delivery (t : t) (d : delivery_entry) = + if d.released + then Ok () + else ( + let rec release = function + | [] -> Ok () + | entry :: rest -> + let* () = release_entry t entry in + release rest + in + let* () = release d.references in + let* () = Dsl.Producer.acknowledge_delivery t.producer d.output.delivery in + d.released <- true; + t.deliveries <- List.filter (fun current -> current != d) t.deliveries; + capacity_changed t; + Ok ()) +;; + +let release_demand (t : t) delivery = + let* released = Dsl.Producer.delivery_released t.producer delivery in + if released + then Ok () + else + let* d = find_delivery t delivery in + release_delivery t d +;; + +let receive_demand (t : t) = + if t.receiver + then Error (Dsl.Capacity_exceeded "Demand receiver already waiting") + else ( + t.receiver <- true; + Fun.protect + ~finally:(fun () -> t.receiver <- false) + (fun () -> + let rec next () = + let candidate = + wait t (fun () -> + match + List.find_opt + (fun d -> (not d.received) && not d.released) + (List.rev t.deliveries) + with + | Some d -> Some (Ok d) + | None when t.active = None || t.fenced -> Some (Error Dsl.Cancelled) + | None -> None) + in + let* d = candidate in + Eio.Fiber.check (); + if not (delivery_current t d) + then + let* () = release_delivery t d in + next () + else ( + d.received <- true; + Ok d.output) + in + next ())) +;; + +let request_stop (t : t) = + let* _ = active_scope t in + t.stop <- true; + pulse t; + Ok () +;; + +let observation_state (t : t) = + let* state = protect_result (fun () -> t.caps.observe t.id) in + if state.identity <> t.id + then Error (Dsl.Identity_mismatch "Foreign observation") + else ( + let state = + { state with + Dsl.network = t.gate + ; keys = t.keys + ; cleanup_pending = max state.cleanup_pending (List.length t.cleanup) + ; last_failure = + (match t.last_failure with + | None -> state.last_failure + | Some _ as e -> e) + } + in + if t.observation <> Some state + then ( + t.observation <- Some state; + pulse t); + Ok state) +;; + +let observe_runtime (t : t) = + let* state = observation_state t in + Ok { revision = { owner = t; sequence = t.revision }; state; status = t.status } +;; + +let await_observation (t : t) ~after = + if after.owner != t + then Error Dsl.Stale_context + else if t.observation_reader + then Error (Dsl.Capacity_exceeded "Observation waiter") + else ( + t.observation_reader <- true; + Fun.protect + ~finally:(fun () -> t.observation_reader <- false) + (fun () -> + let* () = + wait t (fun () -> + if t.revision <> after.sequence + then Some (Ok ()) + else ( + match actual t with + | Context (Dsl.Retired, _) -> Some (Error Dsl.Stale_context) + | _ -> None)) + in + observe_runtime t)) +;; + +let check_scope (t : t) (scope : task_scope) = + if not scope.live + then Error Dsl.Stale_context + else Dsl.Producer.check_scope t.producer scope.handle +;; + +let make_task_scope (t : t) _parent = + let scope = + { handle = Dsl.Producer.scope t.producer + ; live = true + ; children = [] + ; notifications = [] + ; prefer_child = false + } + in + t.scopes <- scope :: t.scopes; + scope +;; + +let child_control (scope : task_scope) child = + (* Physical identity only; results stay typed in Producer.child_result. *) + List.find_opt (fun c -> c.key == Obj.repr child && not c.consumed) scope.children +;; + +let cleanup_scope (t : t) (scope : task_scope) = + Eio.Cancel.protect (fun () -> + List.iter (fun c -> if not c.consumed then c.cancel_child ()) scope.children; + let failures = ref [] in + List.iter + (fun c -> + if not c.consumed + then ( + Option.iter (fun e -> failures := e :: !failures) (c.join_child ()); + c.consumed <- true; + c.release_capacity (); + t.tasks <- t.tasks - 1)) + scope.children; + scope.children <- []; + scope.live <- false; + ignore (Dsl.Producer.close_scope t.producer scope.handle : (unit, Dsl.error) result); + List.iter + (fun d -> + if d.source.scope == scope && (not d.received) && not d.released + then ( + match release_delivery t d with + | Ok () -> () + | Error e -> failures := e :: !failures)) + t.deliveries; + scope.notifications <- []; + t.scopes <- List.filter (fun s -> s != scope) t.scopes; + capacity_changed t; + match List.rev !failures with + | [] -> Ok () + | xs -> Error (cleanup_error xs)) +;; + +let require_resource (t : t) reference = + let* () = validate_reference t reference in + match find_resource t reference with + | Some e when (not e.released) && not e.delivery_owned -> Ok e + | _ -> Error Dsl.Stale_context +;; + +let perform_crypto : type a. t -> unit ref -> a Dsl.crypto -> (a, Dsl.error) result = + fun t token operation -> + with_effect t token (fun scope -> + let* result = t.caps.dispatch.crypto ~scope ~identity:t.id operation in + let* () = check_current scope in + match operation with + | Dsl.Unlock _ -> + t.keys <- result; + mark_dirty t; + Ok result + | Dsl.Encrypt_values values -> + if Rrbvec.length result <> Rrbvec.length values + then Error (Dsl.Crypto_failure "Encryption cardinality") + else Ok result + | Dsl.Decrypt_value _ | Dsl.Protect _ | Dsl.Unprotect _ -> Ok result) +;; + +let perform_database : type a. t -> unit ref -> a Dsl.database -> (a, Dsl.error) result = + fun t token operation -> + let check_current () = still_current t token in + let with_database f = Owner.with_database t.owner.binding ~check_current f in + match operation with + | Dsl.Release_snapshot snapshot -> release_reference t (Dsl.Snapshot_reference snapshot) + | Dsl.Inspect -> + with_database (fun database -> + match Dsl.Overlay.Database.inspect_sync database with + | Error e -> Error (Dsl.Sync_read_failure e) + | Ok view -> + if + List.length (Dsl.Overlay.Types.sync_view_submissions view) + > t.limits.maximum_items + then Error (Dsl.Invalid_input "Sync view items") + else Ok view) + | Dsl.Verify_checksum (through, expected) -> + with_database (fun database -> + match + Dsl.Overlay.Database.check_checkpoint_checksum database ~through ~expected + with + | Error error -> Error (Dsl.Sync_read_failure error) + | Ok (Some false) -> + Error (Dsl.Protocol_failure "Remote checkpoint checksum mismatch") + | Ok (Some true | None) -> Ok ()) + | Dsl.Prepare_submission (expected, transition) -> + let* prepared, request = + with_database (fun database -> + Result.map_error + (fun e -> Dsl.Outbox_failure e) + (Dsl.Overlay.Database.begin_outbox_transition database ~expected transition)) + in + let* encrypted = + match request with + | None -> Ok None + | Some request -> + let* values = perform_crypto t token (Dsl.Protect request) in + Ok (Some (request, values)) + in + let* result = + with_database (fun database -> + let* () = check_current () in + Eio.Cancel.protect (fun () -> + Result.map_error + (fun e -> Dsl.Outbox_failure e) + (Dsl.Overlay.Database.apply_outbox_transition database prepared ~encrypted))) + in + mark_dirty t; + Ok result + | Dsl.Apply_authoritative (expected, batch) -> + let* prepared, request = + with_database (fun database -> + Result.map_error + (fun e -> Dsl.Authoritative_failure e) + (Dsl.Overlay.Database.begin_authoritative database ~expected batch)) + in + let* decrypted = + match request with + | None -> Ok None + | Some request -> + let* values = perform_crypto t token (Dsl.Unprotect request) in + Ok (Some (request, values)) + in + let* result = + with_database (fun database -> + let* () = check_current () in + Eio.Cancel.protect (fun () -> + Result.map_error + (fun e -> Dsl.Authoritative_failure e) + (Dsl.Overlay.Database.apply_authoritative database prepared ~decrypted))) + in + mark_dirty t; + Ok + (match result with + | Dsl.Overlay.Database.Authoritative_applied c -> Dsl.Applied c + | Dsl.Overlay.Database.Authoritative_deferred d -> Dsl.Deferred d) + | Dsl.Activate_snapshot snapshot -> + let* _ = require_resource t (Dsl.Snapshot_reference snapshot) in + let* source = + match t.owner.snapshot_source with + | None -> Error (Dsl.Invalid_input "Active mirror") + | Some resolve -> protect_result (fun () -> resolve snapshot) + in + let* () = check_current () in + let* result = + Owner.activate_snapshot + t.owner.binding + ~check_current + { snapshot + ; path = source.path + ; expected_rows = source.expected_rows + ; maximum_crypto_items = t.limits.maximum_items + ; maximum_crypto_bytes = t.limits.maximum_bytes + ; unprotect = (fun request -> perform_crypto t token (Dsl.Unprotect request)) + } + in + mark_dirty t; + Ok result +;; + +let perform_transport : type a. t -> unit ref -> a Dsl.transport -> (a, Dsl.error) result = + fun t token operation -> + match operation with + | Dsl.Retire_connection connection -> + release_reference t (Dsl.Connection_reference connection) + | _ -> + let validate : type b. b Dsl.transport -> (unit, Dsl.error) result = function + | Dsl.On_connection (connection, request) -> + let* _ = require_resource t (Dsl.Connection_reference connection) in + (match request with + | Dsl.Head | Dsl.Fetch_snapshot _ | Dsl.Pull _ | Dsl.Push _ -> Ok () + | _ -> Error (Dsl.Invalid_input "Invalid scoped transport")) + | Dsl.Receive connection -> + let* _ = require_resource t (Dsl.Connection_reference connection) in + Ok () + | Dsl.Connect | Dsl.Head | Dsl.Fetch_snapshot _ | Dsl.Pull _ | Dsl.Push _ -> Ok () + | Dsl.Retire_connection _ -> Ok () + in + let* () = validate operation in + with_network + t + (fun () -> still_current t token) + (fun () -> + with_effect t token (fun scope -> + let* result = t.caps.dispatch.transport ~scope ~identity:t.id operation in + let rec check_result : type b. b Dsl.transport -> b -> (unit, Dsl.error) result + = + fun op result -> + match op with + | Dsl.Connect -> + let* _ = require_resource t (Dsl.Connection_reference result) in + Ok () + | Dsl.Fetch_snapshot _ -> + let* _ = require_resource t (Dsl.Snapshot_reference result) in + Ok () + | Dsl.On_connection (_, op) -> check_result op result + | Dsl.Receive connection -> + if result.connection = connection + then Ok () + else Error (Dsl.Protocol_failure "Connection mismatch") + | Dsl.Head | Dsl.Pull _ | Dsl.Push _ | Dsl.Retire_connection _ -> Ok () + in + let* () = check_result operation result in + let* () = check_current scope in + Ok result)) +;; + +let admit_asset_direction (t : t) work opposite = + if + attempt_active opposite + && (asset_slots t.asset_work >= t.limits.maximum_active_assets + || (work_background t.demands work + && asset_background_count t.demands t.asset_work + >= t.limits.maximum_active_assets - t.limits.foreground_reserve)) + then Error (Dsl.Capacity_exceeded "Combined asset transfers") + else Ok () +;; + +let start_asset_attempt (t : t) ~register callback = + let outcome, resolve = Eio.Promise.create () in + let attempt = + { outcome; waiters = 0; cancellation = None; physically_joined = false } + in + register attempt; + Eio.Fiber.fork_daemon ~sw:t.sw (fun () -> + let result = + try + Eio.Cancel.sub (fun cc -> + attempt.cancellation <- Some cc; + callback ()) + with + | Eio.Cancel.Cancelled _ -> Error Dsl.Cancelled + | Forced error -> Error error + | _ -> Error (Dsl.Unexpected_failure "Asset transfer raised") + in + attempt.cancellation <- None; + attempt.physically_joined <- true; + Eio.Promise.resolve resolve result; + capacity_changed t; + `Stop_daemon); + attempt +;; + +let await_asset_attempt attempt = + attempt.waiters <- attempt.waiters + 1; + Fun.protect + ~finally:(fun () -> + attempt.waiters <- attempt.waiters - 1; + if attempt.waiters = 0 && not attempt.physically_joined + then + Eio.Cancel.protect (fun () -> + Option.iter + (fun cc -> Eio.Cancel.cancel cc Cancel_execution) + attempt.cancellation; + ignore (Eio.Promise.await attempt.outcome))) + (fun () -> Eio.Promise.await attempt.outcome) +;; + +let rec perform_assets : type a. t -> unit ref -> a Dsl.assets -> (a, Dsl.error) result = + fun t token operation -> + match operation with + | Dsl.Release_file file -> release_reference t (Dsl.File_reference file) + | Dsl.Release_lease lease -> release_reference t (Dsl.Lease_reference lease) + | _ -> + let* () = + match operation with + | Dsl.Retain file -> + let* _ = require_resource t (Dsl.File_reference file) in + Ok () + | _ -> Ok () + in + let work () = + let reserve = + match operation with + | Dsl.Download _ -> 2 + | Dsl.Lookup _ | Dsl.Retain _ | Dsl.Upload _ -> 1 + | Dsl.Prune_cache | Dsl.Reconcile_staging | Dsl.Release_file _ | Dsl.Release_lease _ -> 0 + in + let binding = Eio.Fiber.get asset_binding in + let background () = + match binding with + | Some (runtime, work) when runtime == t -> work_background t.demands work + | _ -> false + in + with_effect ~reserve ~background t token (fun scope -> + let* value = t.caps.dispatch.assets ~scope ~identity:t.id operation in + let* () = + match operation with + | Dsl.Lookup asset -> + (match value with + | None -> Ok () + | Some file -> + if file.asset <> asset + then Error (Dsl.Protocol_failure "Asset selection mismatch") + else + let* _ = require_resource t (Dsl.File_reference file) in + Ok ()) + | Dsl.Download asset -> + (match value with + | Dsl.Available file -> + if file.asset <> asset + then Error (Dsl.Protocol_failure "Asset selection mismatch") + else + let* _ = require_resource t (Dsl.File_reference file) in + Ok () + | _ -> Ok ()) + | Dsl.Retain file -> + if value.source <> file + then Error (Dsl.Protocol_failure "Lease source mismatch") + else + let* _ = require_resource t (Dsl.Lease_reference value) in + Ok () + | Dsl.Prune_cache | Dsl.Reconcile_staging | Dsl.Upload _ | Dsl.Release_file _ | Dsl.Release_lease _ -> + Ok () + in + let* () = check_current scope in + Ok value) + in + (match operation with + | Dsl.Download asset | Dsl.Upload asset -> + let remote () = with_network t (fun () -> still_current t token) work in + let reserved () : (a, Dsl.error) result = + match Eio.Fiber.get asset_binding with + | Some (runtime, bound) when runtime == t && bound.asset = asset -> + (match operation with + | Dsl.Download selected -> + let* attempt = + match bound.download with + | Some attempt -> Ok attempt + | None -> + let* () = admit_asset_direction t bound bound.upload in + Ok + (start_asset_attempt + t + ~register:(fun attempt -> bound.download <- Some attempt) + remote) + in + let outcome = await_asset_attempt attempt in + (match outcome with + | Error (Dsl.Capacity_exceeded _) when attempt.physically_joined -> + (match bound.download with + | Some current when current == attempt -> bound.download <- None + | _ -> ()) + | _ -> ()); + let* result = outcome in + (match result with + | Dsl.Available original -> + let* file = + if bound.download_files = [] + then Ok original + else + let* found = perform_assets t token (Dsl.Lookup selected) in + match found with + | Some file + when not + (List.exists + (fun claimed -> + Dsl.Producer.same_reference_registration + (Dsl.File_reference claimed) + (Dsl.File_reference file)) + bound.download_files) -> Ok file + | _ -> + Error + (Dsl.Protocol_failure + "Shared download requires an independent file reference") + in + let* _ = require_resource t (Dsl.File_reference file) in + bound.download_files <- file :: bound.download_files; + Ok (Dsl.Available file) + | result -> Ok result) + | Dsl.Upload _ -> + let* attempt = + match bound.upload with + | Some attempt -> Ok attempt + | None -> + let* () = admit_asset_direction t bound bound.download in + Ok + (start_asset_attempt + t + ~register:(fun attempt -> bound.upload <- Some attempt) + remote) + in + await_asset_attempt attempt + | _ -> work ()) + | _ -> + let* _, release = reserve_asset t asset None in + Fun.protect ~finally:release remote + in + reserved () + | _ -> work ()) +;; + +let take_input (t : t) (scope : task_scope) = + if t.stop + then ( + t.stop <- false; + Some (Ok Dsl.Stop_requested)) + else ( + match t.network_input with + | Some gate -> + t.network_input <- None; + Some (Ok (Dsl.Network_changed gate)) + | None -> + (match t.unlock with + | Some u when u.expires <= t.now () -> + t.unlock <- None; + None + | Some u when u.queued -> + Some + (let* receipt = Dsl.Producer.unlock_receipt t.producer scope.handle in + u.receipt <- receipt; + u.queued <- false; + Ok (Dsl.Unlock_requested receipt)) + | _ -> + (match + List.find_opt + (fun (d : demand_entry) -> d.queued && not d.finished) + (List.rev t.demands) + with + | None -> None + | Some d -> + Some + (let* receipt = Dsl.Producer.demand_receipt t.producer scope.handle in + d.scope <- scope; + d.receipt <- receipt; + d.queued <- false; + Ok (Dsl.Demand_changed (receipt, d.change)))))) +;; + +let input_wait (t : t) callback = + if t.input_reader + then Error (Dsl.Capacity_exceeded "Input already waiting") + else ( + t.input_reader <- true; + Fun.protect ~finally:(fun () -> t.input_reader <- false) callback) +;; + +let publish_demand (t : t) (scope : task_scope) receipt availability = + let* () = Dsl.Producer.check_demand_receipt t.producer scope.handle receipt in + let* source = current_demand t receipt in + let* references = + match availability, source.change with + | Dsl.Demand_available (file, lease), Dsl.Demand demand -> + if file.asset <> demand.asset || lease.source <> file + then Error (Dsl.Invalid_input "Demand resource selection") + else + let* lease_entry = require_resource t (Dsl.Lease_reference lease) in + let* file_entry = require_resource t (Dsl.File_reference file) in + Ok [ lease_entry; file_entry ] + | Dsl.Demand_available _, _ | Dsl.Demand_revoked, Dsl.Demand _ -> + Error Dsl.Stale_context + | _ -> Ok [] + in + if + List.length (List.filter (fun (d : delivery_entry) -> not d.released) t.deliveries) + >= t.limits.maximum_messages + then Error (Dsl.Capacity_exceeded "Demand output messages") + else ( + let old = + List.filter + (fun d -> d.source == source && (not d.received) && not d.released) + t.deliveries + in + let rec release = function + | [] -> Ok () + | d :: rest -> + let* () = release_delivery t d in + release rest + in + let* () = release old in + let* _ = current_demand t receipt in + let* output = + Dsl.Producer.demand_output + t.producer + scope.handle + receipt + ~id:(demand_id source.change) + availability + in + List.iter (fun e -> e.delivery_owned <- true) references; + t.deliveries + <- { output; source; references; received = false; released = false } :: t.deliveries; + pulse t; + Ok ()) +;; + +let asset_reason = function + | Dsl.Network_closed -> Dsl.Asset_network_closed + | Dsl.Network_failure message -> Dsl.Asset_network message + | Dsl.Locked -> Dsl.Asset_locked + | Dsl.Cancelled -> Dsl.Asset_cancelled + | error -> Dsl.Asset_other error +;; + +let task_operation + : type a. t -> unit ref -> task_scope -> a Dsl.task_operation -> (a, Dsl.error) result + = + fun t token scope operation -> + match operation with + | Dsl.Notifications handle -> + let* () = Dsl.Producer.check_scope t.producer handle in + if handle != scope.handle + then Error Dsl.Stale_context + else if + List.fold_left + (fun count scope -> count + List.length scope.notifications) + 0 + t.scopes + >= t.limits.maximum_messages + then Error (Dsl.Capacity_exceeded "Notification channels") + else + let* handle = Dsl.Producer.notifications t.producer scope.handle in + scope.notifications + <- { handle; queue = Queue.create (); reading = false } :: scope.notifications; + Ok handle + | Dsl.Notify (handle, notification) -> + let* () = Dsl.Producer.check_notifications t.producer scope.handle handle in + (match + List.find_opt + (fun (c : notification_channel) -> c.handle == handle) + scope.notifications + with + | None -> Error Dsl.Stale_context + | Some channel -> + let count, bytes = notification_usage t in + let demands, descriptors = demand_usage t in + let size = String.length (Marshal.to_string notification []) in + let* () = + if not scope.live + then Error Dsl.Cancelled + else if + count + demands >= t.limits.maximum_messages + || size > t.limits.maximum_message_bytes - bytes - descriptors + then Error (Dsl.Capacity_exceeded "Notification budget") + else Ok () + in + Queue.add (notification, size) channel.queue; + (match notification with + | Dsl.Connected _ -> mark_dirty t + | _ -> ()); + pulse t; + Ok ()) + | Dsl.Next_notification handle -> + let* () = Dsl.Producer.check_notifications t.producer scope.handle handle in + (match + List.find_opt + (fun (c : notification_channel) -> c.handle == handle) + scope.notifications + with + | None -> Error Dsl.Stale_context + | Some channel when channel.reading -> + Error (Dsl.Capacity_exceeded "Notification receiver") + | Some channel -> + channel.reading <- true; + Fun.protect + ~finally:(fun () -> channel.reading <- false) + (fun () -> + wait t (fun () -> + if not scope.live + then Some (Error Dsl.Cancelled) + else if Queue.is_empty channel.queue + then None + else ( + let n, _ = Queue.take channel.queue in + capacity_changed t; + Some (Ok n))))) + | Dsl.Await_outbox_changed -> + if Option.is_some t.outbox_check + then Error (Dsl.Capacity_exceeded "Outbox inspector") + else + let* () = + wait t (fun () -> + if not scope.live + then Some (Error Dsl.Cancelled) + else if t.outbox_epoch <> t.outbox_acked + then Some (Ok ()) + else None) + in + let* check = Dsl.Producer.outbox_check t.producer scope.handle in + t.outbox_check <- Some (check, t.outbox_epoch); + Ok check + | Dsl.Acknowledge_outbox_check check -> + let* () = Dsl.Producer.check_outbox_check t.producer scope.handle check in + (match t.outbox_check with + | Some (original, epoch) when original == check -> + t.outbox_acked <- epoch; + t.outbox_check <- None; + pulse t; + Ok () + | _ -> Error Dsl.Stale_context) + | Dsl.Abandon_outbox_check check -> + let* () = Dsl.Producer.check_outbox_check t.producer scope.handle check in + (match t.outbox_check with + | Some (original, _) when original == check -> + t.outbox_check <- None; + pulse t; + Ok () + | _ -> Error Dsl.Stale_context) + | Dsl.Resample_outbox -> + mark_dirty t; + Ok () + | Dsl.Capacity_revision -> + Dsl.Producer.capacity_revision t.producer scope.handle ~sequence:t.capacity + | Dsl.Await_capacity_changed revision -> + let* sequence = Dsl.Producer.capacity_sequence t.producer scope.handle revision in + wait t (fun () -> + if t.capacity <> sequence + then Some (Ok ()) + else if not scope.live + then Some (Error Dsl.Cancelled) + else None) + | Dsl.Task_observe -> observation_state t + | Dsl.Publish_status status -> + t.status <- status; + (match status with + | Dsl.Service_blocked error -> t.last_failure <- Some error + | Service_connecting | Service_online -> t.last_failure <- None + | _ -> ()); + pulse t; + Ok () + | Dsl.Task_database operation -> perform_database t token operation + | Dsl.Task_transport operation -> perform_transport t token operation + | Dsl.Task_crypto operation -> perform_crypto t token operation + | Dsl.Task_assets operation -> perform_assets t token operation + | Dsl.Consume_unlock receipt -> + let* () = Dsl.Producer.check_unlock_receipt t.producer scope.handle receipt in + (match t.unlock with + | Some u when u.receipt == receipt && u.expires > t.now () -> + t.unlock <- None; + perform_crypto t token (Dsl.Unlock u.input) + | _ -> Error Dsl.Stale_context) + | Dsl.Publish_demand (receipt, availability) -> + publish_demand t scope receipt availability + | Dsl.Finish_demand receipt -> + let* () = Dsl.Producer.check_demand_receipt t.producer scope.handle receipt in + let* d = current_demand t receipt in + d.finished <- true; + (match d.change with + | Dsl.Withdraw _ -> t.demands <- List.filter (fun current -> current != d) t.demands + | Dsl.Demand _ -> ()); + capacity_changed t; + Ok () + | Dsl.Download_demand receipt -> + let* () = Dsl.Producer.check_demand_receipt t.producer scope.handle receipt in + let* d = current_demand t receipt in + (match d.change with + | Dsl.Withdraw _ -> Error Dsl.Stale_context + | Dsl.Demand demand -> + let* () = + if d.download_waiting_capacity + then ( + let background = + match Eio.Fiber.get asset_binding with + | Some (runtime, work) when runtime == t -> work_background t.demands work + | _ -> false + in + if + List.length t.resources + List.length t.producers + t.reserved_resources + 2 + > t.limits.maximum_resources + - if background then t.limits.foreground_reserve else 0 + then Error (Dsl.Capacity_exceeded "Effect resources") + else Ok ()) + else Ok () + in + d.download_waiting_capacity <- false; + let* found = perform_assets t token (Dsl.Lookup demand.asset) in + (match found with + | Some file -> Ok { Dsl.result = Ok (Dsl.Available file); attempts = 0 } + | None when t.keys = Dsl.Key_locked -> + Ok { Dsl.result = Error Dsl.Asset_locked; attempts = 0 } + | None -> + (match demand.asset.source, t.gate with + | Logseq_db_types.Asset_descriptor.External _, _ -> + Ok { Dsl.result = Ok Dsl.Missing_local; attempts = 0 } + | Managed None, _ -> Ok { Dsl.result = Ok Dsl.Waiting_remote; attempts = 0 } + | _, Dsl.Closed -> + Ok { Dsl.result = Error Dsl.Asset_network_closed; attempts = 0 } + | Managed (Some _), Dsl.Open _ -> + let result = perform_assets t token (Dsl.Download demand.asset) in + let* _ = current_demand t receipt in + (match result with + | Error (Dsl.Capacity_exceeded _ as e) -> + d.download_waiting_capacity <- true; + Error e + | Error (Dsl.Cleanup_failure _ as e) -> Error e + | result -> + Ok { Dsl.result = Result.map_error asset_reason result; attempts = 1 })))) + | Dsl.Wait_network_open -> + wait t (fun () -> + if not scope.live + then Some (Error Dsl.Cancelled) + else ( + match t.gate, t.retirement with + | Dsl.Open _, None -> Some (Ok ()) + | _ -> None)) + | Dsl.Next_input -> + input_wait t (fun () -> + wait t (fun () -> + if not scope.live then Some (Error Dsl.Cancelled) else take_input t scope)) + | Dsl.Await_input_or_child children -> + let children = Rrbvec.to_list children in + let rec validate seen = function + | [] -> Ok () + | child :: rest -> + let* () = Dsl.Producer.check_child t.producer scope.handle child in + if List.exists (Dsl.equal_child child) seen + then Error (Dsl.Invalid_input "Duplicate child") + else validate (child :: seen) rest + in + let* () = validate [] children in + input_wait t (fun () -> + wait t (fun () -> + if not scope.live + then Some (Error Dsl.Cancelled) + else ( + let rec poll = function + | [] -> None + | child :: rest -> + (match Dsl.Producer.child_result t.producer scope.handle child with + | Error e -> Some (Error e) + | Ok (Some _) -> + scope.prefer_child <- false; + Some (Ok (Dsl.Child_ready child)) + | Ok None -> poll rest) + in + let input () = + match take_input t scope with + | Some result -> + scope.prefer_child <- true; + Some (Result.map (fun input -> Dsl.Input input) result) + | None -> None + in + if t.stop + then input () + else if scope.prefer_child + then ( + match poll children with + | Some _ as ready -> ready + | None -> input ()) + else ( + match input () with + | Some _ as ready -> ready + | None -> poll children)))) + | Dsl.Sleep seconds -> + if (not (Float.is_finite seconds)) || seconds < 0. + then Error (Dsl.Invalid_input "Sleep duration") + else ( + t.sleep seconds; + Ok ()) + | Dsl.Yield -> + Eio.Fiber.yield (); + Ok () +;; + +type (_, _) task_continuation = + | Task_done : ('a, 'a) task_continuation + | Task_then : + ('a -> 'b Dsl.task) * task_scope * ('b, 'c) task_continuation + -> ('a, 'c) task_continuation + | Task_exit : task_scope * ('a, 'b) task_continuation -> ('a, 'b) task_continuation + +type 'a task_step = + | Task_step of (unit -> 'a task_step) + | Task_result of ('a, Dsl.error) result + +let rec task_step + : type a b. + t + -> unit ref + -> task_scope + -> a Dsl.task + -> (a, b) task_continuation + -> int + -> b task_step + = + fun t token scope task continuation steps -> + Task_step + (fun () -> + try + let steps = + if steps >= t.limits.steps_per_turn + then ( + Eio.Fiber.yield (); + 0) + else steps + in + Eio.Fiber.check (); + match Dsl.task_view task with + | Dsl.Task_return value -> + task_continue t token scope value continuation (steps + 1) + | Dsl.Task_fail error -> task_unwind t continuation error + | Dsl.Task_bind (task, next) -> + task_step t token scope task (Task_then (next, scope, continuation)) (steps + 1) + | Dsl.Task_scope f -> + let inner = make_task_scope t (Some scope) in + task_step + t + token + inner + (f inner.handle) + (Task_exit (inner, continuation)) + (steps + 1) + | Dsl.Task_perform (operation, success, failure) -> + (match + let* () = check_scope t scope in + task_operation t token scope operation + with + | Ok value -> task_step t token scope (success value) continuation (steps + 1) + | Error (Dsl.Cleanup_failure _ as e) -> + record_cleanup t e; + raise (Forced e) + | Error e -> task_step t token scope (failure e) continuation (steps + 1)) + | Dsl.Task_spawn (handle, task) -> + let result = spawn_task t token scope handle None task in + task_continue t token scope result continuation (steps + 1) + | Dsl.Task_spawn_asset (handle, receipt, task) -> + let result = spawn_task t token scope handle (Some receipt) task in + task_continue t token scope result continuation (steps + 1) + | Dsl.Task_join child -> + let result = + let* () = Dsl.Producer.check_child t.producer scope.handle child in + match child_control scope child with + | None -> Error Dsl.Stale_context + | Some control -> + ignore (control.join_child () : Dsl.error option); + let* result = Dsl.Producer.child_result t.producer scope.handle child in + (match result with + | None -> Error (Dsl.Unexpected_failure "Joined child is unfinished") + | Some result -> + let* () = Dsl.Producer.forget_child t.producer scope.handle child in + control.consumed <- true; + control.release_capacity (); + scope.children <- List.filter (fun c -> c != control) scope.children; + t.tasks <- t.tasks - 1; + capacity_changed t; + result) + in + (* Join returns the child's domain result as data. *) + task_continue t token scope result continuation (steps + 1) + | Dsl.Task_cancel child -> + (match child_control scope child with + | None -> task_unwind t continuation Dsl.Stale_context + | Some c -> + c.cancel_child (); + task_continue t token scope () continuation (steps + 1)) + with + | Eio.Cancel.Cancelled _ -> + Eio.Cancel.protect (fun () -> task_unwind t continuation Dsl.Cancelled) + | Forced error -> Eio.Cancel.protect (fun () -> task_unwind t continuation error) + | _ -> + Eio.Cancel.protect (fun () -> + task_unwind t continuation (Dsl.Unexpected_failure "Task continuation raised"))) + +and task_continue + : type a b. + t -> unit ref -> task_scope -> a -> (a, b) task_continuation -> int -> b task_step + = + fun t token scope value continuation steps -> + match continuation with + | Task_done -> Task_result (Ok value) + | Task_then (next, parent, rest) -> task_step t token parent (next value) rest steps + | Task_exit (inner, rest) -> + (match cleanup_scope t inner with + | Ok () -> task_continue t token scope value rest steps + | Error e -> task_unwind t rest e) + +and task_unwind : type a b. t -> (a, b) task_continuation -> Dsl.error -> b task_step = + fun t continuation error -> + match continuation with + | Task_done -> Task_result (Error error) + | Task_then (_, _, rest) -> task_unwind t rest error + | Task_exit (scope, rest) -> + let error = + match cleanup_scope t scope with + | Ok () -> error + | Error e -> cleanup_error ~primary:error [ e ] + in + task_unwind t rest error + +and spawn_task + : type a. + t + -> unit ref + -> task_scope + -> Dsl.scope + -> Dsl.demand_receipt option + -> a Dsl.task + -> (a Dsl.child, Dsl.error) result + = + fun t token scope handle receipt task -> + let* () = Dsl.Producer.check_scope t.producer handle in + if handle != scope.handle + then Error Dsl.Stale_context + else + let* background = + match receipt with + | None -> Ok false + | Some receipt -> + let* d = current_demand t receipt in + Ok (background d.change) + in + let pending_ceiling = + t.limits.maximum_pending - if Option.is_some receipt then 3 else 0 + in + let ceiling = + min t.limits.maximum_tasks pending_ceiling + - if background then t.limits.foreground_reserve else 0 + in + if t.tasks >= ceiling + then Error (Dsl.Capacity_exceeded "Child slots") + else + let* reservation = + match receipt with + | None -> Ok None + | Some receipt -> + let* demand = current_demand t receipt in + (match demand.change with + | Dsl.Withdraw _ -> Ok None + | Dsl.Demand d -> + let* reservation = reserve_asset t d.asset (Some demand.change) in + Ok (Some reservation)) + in + let release_capacity () = + Option.iter (fun (_, release) -> release ()) reservation + in + let result = ref None in + let finished, resolve = Eio.Promise.create () in + let cancelled = ref false + and cc = ref None in + let* child = + match + Dsl.Producer.child t.producer scope.handle task ~poll:(fun () -> !result) + with + | Ok child -> Ok child + | Error error -> + release_capacity (); + Error error + in + let control = + { key = Obj.repr child + ; consumed = false + ; release_capacity + ; cancel_child = + (fun () -> + cancelled := true; + Option.iter (fun cc -> Eio.Cancel.cancel cc Cancel_execution) !cc) + ; join_child = + (fun () -> + Eio.Promise.await finished; + match !result with + | Some (Error (Dsl.Cancelled | Dsl.Network_closed | Dsl.Stale_context)) + | Some (Ok _) -> None + | Some (Error e) -> Some e + | None -> Some (Dsl.Unexpected_failure "Child failed to resolve")) + } + in + t.tasks <- t.tasks + 1; + scope.children <- control :: scope.children; + Eio.Fiber.fork_daemon ~sw:t.sw (fun () -> + let outcome = + try + Eio.Cancel.sub (fun context -> + cc := Some context; + if !cancelled + then Error Dsl.Cancelled + else ( + match reservation with + | None -> task_interpret t token scope task Task_done 0 + | Some (work, _) -> + Eio.Fiber.with_binding asset_binding (t, work) (fun () -> + task_interpret t token scope task Task_done 0))) + with + | Eio.Cancel.Cancelled _ -> Error Dsl.Cancelled + | Forced e -> Error e + | _ -> Error (Dsl.Unexpected_failure "Task continuation raised") + in + cc := None; + result := Some outcome; + Eio.Promise.resolve resolve (); + pulse t; + `Stop_daemon); + Ok child + +and task_interpret + : type a b. + t + -> unit ref + -> task_scope + -> a Dsl.task + -> (a, b) task_continuation + -> int + -> (b, Dsl.error) result + = + fun t token scope task continuation steps -> + let rec drive = function + | Task_result result -> result + | Task_step next -> drive (next ()) + in + drive (task_step t token scope task continuation steps) +;; + +let run_task (t : t) token task = + let root = make_task_scope t None in + t.active <- Some root; + t.stop <- false; + t.network_input <- None; + t.unlock <- None; + mark_dirty t; + let result = + try task_interpret t token root task Task_done 0 with + | exn -> + Eio.Cancel.protect (fun () -> + List.iter + (fun scope -> ignore (cleanup_scope t scope : (unit, Dsl.error) result)) + (List.filter (fun s -> s != root) t.scopes); + ignore (cleanup_scope t root : (unit, Dsl.error) result); + t.active <- None; + t.unlock <- None; + t.outbox_check <- None; + pulse t); + raise exn + in + let joined = cleanup_scope t root in + t.active <- None; + t.unlock <- None; + t.outbox_check <- None; + t.demands <- []; + pulse t; + match result, joined with + | result, Ok () -> result + | result, Error error -> + let error = + cleanup_error + ?primary: + (match result with + | Error e -> Some e + | Ok _ -> None) + [ error ] + in + record_cleanup t error; + raise (Forced error) +;; + +let drain (t : t) = + t.fenced <- true; + t.gate <- Dsl.Closed; + ignore (set_phase t Dsl.Draining); + let failures = ref [] in + let add = function + | Ok () -> () + | Error e -> failures := e :: !failures + in + let module W = (val t.caps.worker_owner) in + add (protect_result (fun () -> W.fence ~identity:t.id)); + add (protect_result (fun () -> t.caps.fence t.id)); + (match close_network t with + | Error e -> failures := e :: !failures + | Ok receipt -> add (await_network_retirement receipt)); + match List.rev !failures with + | [] -> Ok () + | xs -> + let e = cleanup_error xs in + record_cleanup t e; + Error e +;; + +let retire (t : t) = + let failures = ref [] in + let add = function + | Ok () -> () + | Error e -> failures := e :: !failures + in + List.iter + (fun e -> if (not e.delivery_owned) && not e.released then add (release_entry t e)) + t.resources; + let* () = + wait t (fun () -> + if t.cleanup <> [] + then Some (Error (cleanup_error (List.rev t.cleanup))) + else if List.exists (fun (d : delivery_entry) -> not d.released) t.deliveries + then None + else Some (Ok ())) + in + if t.cleanup <> [] + then Error (cleanup_error (List.rev t.cleanup)) + else + let module W = (val t.caps.worker_owner) in + add (protect_result (fun () -> W.await_quiescent ~identity:t.id)); + if !failures = [] then add (protect_result (fun () -> t.caps.retire t.id)); + match List.rev !failures with + | [] -> Ok () + | xs -> + let error = cleanup_error xs in + record_cleanup t error; + Error error +;; + +let operation + : type before after failed a. + t + -> unit ref + -> before Dsl.phase + -> (before, after, failed, a) Dsl.operation + -> (after Dsl.phase * a, failed Dsl.phase * Dsl.error) result + = + fun t token phase operation -> + let unchanged result = + match result with + | Ok a -> Ok (phase, a) + | Error e -> Error (phase, e) + in + match operation with + | Dsl.Restore -> + (match with_effect t token (fun scope -> t.caps.restore ~scope t.id) with + | Ok report -> + t.keys <- report.keys; + mark_dirty t; + ignore (set_phase t Dsl.Ready); + Ok (Dsl.Ready, report) + | Error e -> Error (Dsl.Cold, e)) + | Dsl.Install_initial_network gate -> + let result = + if not t.initial_eligible + then Ok () + else ( + t.initial_eligible <- false; + match gate with + | Dsl.Closed -> + let* receipt = close_network t in + await_network_retirement receipt + | Dsl.Open { authorization_epoch } -> open_network t ~authorization_epoch) + in + (match result with + | Ok () -> Ok (Dsl.Ready, ()) + | Error e -> Error (Dsl.Ready, e)) + | Dsl.Set_network gate -> + let result = + match gate with + | Dsl.Closed -> + let* receipt = close_network t in + await_network_retirement receipt + | Dsl.Open { authorization_epoch } -> open_network t ~authorization_epoch + in + (match result with + | Ok () -> Ok (Dsl.Ready, ()) + | Error e -> Error (Dsl.Ready, e)) + | Dsl.Run_task task -> + (match run_task t token task with + | Ok a -> Ok (Dsl.Ready, a) + | Error e -> Error (Dsl.Ready, e)) + | Dsl.Database op -> + (match perform_database t token op with + | Ok a -> Ok (Dsl.Ready, a) + | Error e -> Error (Dsl.Ready, e)) + | Dsl.Transport op -> + (match perform_transport t token op with + | Ok a -> Ok (Dsl.Ready, a) + | Error e -> Error (Dsl.Ready, e)) + | Dsl.Crypto op -> + (match perform_crypto t token op with + | Ok a -> Ok (Dsl.Ready, a) + | Error e -> Error (Dsl.Ready, e)) + | Dsl.Assets op -> + (match perform_assets t token op with + | Ok a -> Ok (Dsl.Ready, a) + | Error e -> Error (Dsl.Ready, e)) + | Dsl.Observe -> unchanged (observation_state t) + | Dsl.Release_file file -> unchanged (release_reference t (Dsl.File_reference file)) + | Dsl.Release_lease lease -> unchanged (release_reference t (Dsl.Lease_reference lease)) + | Dsl.Release_snapshot snapshot -> + unchanged (release_reference t (Dsl.Snapshot_reference snapshot)) + | Dsl.Begin_drain _ -> + (match drain t with + | Ok () -> Ok (Dsl.Draining, ()) + | Error e -> Error (Dsl.Draining, e)) + | Dsl.Finish_retirement -> + (match retire t with + | Ok () -> + ignore (set_phase t Dsl.Retired); + Ok (Dsl.Retired, ()) + | Error e -> Error (Dsl.Draining, e)) +;; + +let rec interpret + : type before after a. + t + -> unit ref + -> before Dsl.phase + -> (before, after, a) Dsl.program + -> (after, a) outcome + = + fun t token phase program -> + Eio.Fiber.check (); + match Dsl.view program with + | Dsl.Return value -> Completed (set_phase t phase, value) + | Dsl.Abort error -> Failed (actual t, error) + | Dsl.Perform (op, success, failure) -> + (match operation t token phase op with + | Ok (next, value) -> + ignore (set_phase t next); + interpret t token next (success value) + | Error (next, error) -> + t.last_failure <- Some error; + ignore (set_phase t next); + if t.cleanup <> [] + then Failed (actual t, cleanup_error ~primary:error (List.rev t.cleanup)) + else interpret t token next (failure error)) +;; + +let start (t : t) (context : _ graph_context) program = + if context.runtime != t || context.version != t.current_version || Option.is_some t.busy + then Error Dsl.Stale_context + else ( + let token = ref () in + let finished, resolve = Eio.Promise.create () in + let running = + { runtime = t + ; outcome_promise = finished + ; cancellation = None + ; cancelled = false + ; terminal = false + } + in + t.busy <- Some token; + t.current_version <- ref (); + ignore (set_phase t context.phase); + Eio.Fiber.fork_daemon ~sw:t.sw (fun () -> + let result = + try + Eio.Cancel.sub (fun cc -> + running.cancellation <- Some cc; + if running.cancelled then raise (Forced Dsl.Cancelled); + interpret t token context.phase program) + with + | Eio.Cancel.Cancelled _ -> + Eio.Cancel.protect (fun () -> ignore (drain t : (unit, Dsl.error) result)); + Failed + ( actual t + , if t.cleanup = [] + then Dsl.Cancelled + else cleanup_error ~primary:Dsl.Cancelled (List.rev t.cleanup) ) + | Forced error -> Failed (actual t, error) + | _ -> Failed (actual t, Dsl.Unexpected_failure "Program continuation raised") + in + running.terminal <- true; + t.busy <- None; + pulse t; + Eio.Promise.resolve resolve result; + `Stop_daemon); + Ok running) +;; + +let await running = Eio.Promise.await running.outcome_promise + +let cancel (t : t) running = + if running.runtime != t || (running.terminal && not running.cancelled) + then Error Dsl.Stale_context + else if running.cancelled + then Ok () + else ( + running.cancelled <- true; + t.fenced <- true; + t.gate <- Dsl.Closed; + ignore (set_phase t Dsl.Draining); + Option.iter (fun cc -> Eio.Cancel.cancel cc Cancel_execution) running.cancellation; + Ok ()) +;; + +let current (t : t) = + if Option.is_some t.busy then Error Dsl.Stale_context else Ok (actual t) +;; + +let create ~sw ~clock ~database_owner:owner ~identity ~limits caps = + let* () = protect_result (fun () -> caps.validate_identity identity) in + let* () = protect_result (fun () -> caps.validate_limits limits) in + let d = caps.network_deadlines in + let valid = + List.for_all + finite_positive + [ caps.cleanup_seconds + ; d.setup_seconds + ; d.tls_handshake_seconds + ; d.http_response_seconds + ; d.download_seconds + ] + in + let valid = + valid + && + match caps.websocket_liveness with + | Peer_close_only -> true + | Ping_pong { interval_seconds; timeout_seconds } -> + finite_positive interval_seconds && finite_positive timeout_seconds + in + if not valid + then Error (Dsl.Invalid_input "Network timings") + else if owner.bound + then Error (Dsl.Identity_mismatch "Database owner already bound") + else + let* () = protect_result (fun () -> caps.acquire identity) in + owner.bound <- true; + let t = + { sw + ; now = + (fun () -> + Mtime.to_uint64_ns (Eio.Time.Mono.now clock) + |> Int64.to_float + |> fun n -> n /. 1e9) + ; sleep = (fun seconds -> Eio.Time.Mono.sleep clock seconds) + ; id = identity + ; limits + ; caps + ; owner + ; producer = Dsl.Producer.create identity + ; changed = Eio.Condition.create () + ; actual = None + ; revision = 0L + ; current_version = ref () + ; busy = None + ; fenced = false + ; gate = Dsl.Closed + ; generation = 0L + ; retirement = None + ; initial_eligible = true + ; network_calls = [] + ; producers = [] + ; resources = [] + ; reserved_resources = 0 + ; cleanup = [] + ; active = None + ; scopes = [] + ; tasks = 0 + ; asset_work = [] + ; capacity = 0L + ; stop = false + ; network_input = None + ; unlock = None + ; demands = [] + ; deliveries = [] + ; receiver = false + ; input_reader = false + ; outbox_epoch = 1L + ; outbox_acked = 0L + ; outbox_check = None + ; keys = Dsl.Key_locked + ; status = Dsl.Service_idle + ; last_failure = None + ; observation = None + ; observation_reader = false + } + in + let cold = set_phase t Dsl.Cold in + let module W = (val caps.worker_owner) in + let network = + { with_admission = + (fun callback -> + with_network + t + (fun () -> if t.fenced then Error Dsl.Stale_context else Ok ()) + callback) + } + in + Eio.Switch.on_release sw (fun () -> + Eio.Cancel.protect (fun () -> + match actual t with + | Context (Dsl.Retired, _) -> () + | _ -> + ignore (drain t : (unit, Dsl.error) result); + ignore (retire t : (unit, Dsl.error) result))); + match + protect_result (fun () -> + W.bind_runtime + ~identity + ~database_owner:owner + ~limits + ~cleanup_seconds:caps.cleanup_seconds + ~network) + with + | Ok () -> Ok (t, cold) + | Error primary -> + Eio.Cancel.protect (fun () -> + let errors = + List.filter_map + (function + | Ok () -> None + | Error e -> Some e) + [ drain t; retire t ] + in + match errors with + | [] -> Error primary + | xs -> Error (cleanup_error ~primary xs)) +;; diff --git a/logseq_sync/lib/execute/database_owner.ml b/logseq_sync/lib/execute/database_owner.ml new file mode 100644 index 00000000..04154eb0 --- /dev/null +++ b/logseq_sync/lib/execute/database_owner.ml @@ -0,0 +1,155 @@ +module Dsl = Sync_dsl.Core +module Overlay = Logseq_overlay_db +module Database = Overlay.Database +module Types = Overlay.Types + +let ( let* ) = Result.bind + +type activation_input = + { snapshot : Dsl.snapshot + ; path : string + ; expected_rows : int + ; maximum_crypto_items : int + ; maximum_crypto_bytes : int + ; unprotect : + Database.unprotection_request + -> ((Types.crypto_item_id * string) list, Dsl.error) result + } + +type preparation = + { staged : Database.prepared_snapshot_activation + ; maximum_crypto_items : int + ; maximum_crypto_bytes : int + ; check_current : unit -> (unit, Dsl.error) result + ; unprotect : + Database.unprotection_request + -> ((Types.crypto_item_id * string) list, Dsl.error) result + } + +type t = + ( Database.t + , Database.mirror_inspection + , activation_input * (unit -> (unit, Dsl.error) result) + , preparation ) + Owner_binding.t + +let database_owner = Owner_binding.active +let database_if_open = Owner_binding.database_if_open +let database = Owner_binding.database +let with_database = Owner_binding.with_database + +let activation_result = function + | Ok value -> Ok value + | Error error -> Error (Dsl.Activation_failure error) +;; + +let rec unprotect preparation = + let* () = preparation.check_current () in + let* request = + activation_result + (Database.next_snapshot_unprotection_batch + ~maximum_items:preparation.maximum_crypto_items + ~maximum_bytes:preparation.maximum_crypto_bytes + preparation.staged) + in + match request with + | None -> Ok () + | Some request -> + let* plaintexts = preparation.unprotect request in + let* () = preparation.check_current () in + let* () = + activation_result + (Database.supply_snapshot_unprotection_batch + preparation.staged + ~request + ~plaintexts) + in + unprotect preparation +;; + +let open_error = function + | Types.Mirror_absent -> "Mirror absent after activation" + | Invalid_graph_name -> "Invalid graph name" + | Ownership_conflict -> "Database ownership conflict" + | Attachment_stale -> "Committed inspection is stale" + | Restore_failed message -> "Restore failed: " ^ message + | Corrupt_authoritative_store message -> "Corrupt authoritative store: " ^ message + | Corrupt_outbox message -> "Corrupt outbox: " ^ message + | Corrupt_mutation_receipt message -> "Corrupt mutation receipt: " ^ message + | Open_resource_failed message -> "Database open failed: " ^ message +;; + +let activation_owner ~sw ~dependencies ~inspection ~graph_name = + let backend = + { Owner_binding.same_input = + (fun (left, _) (right, _) -> + left.snapshot = right.snapshot + && left.path = right.path + && left.expected_rows = right.expected_rows + && left.maximum_crypto_items = right.maximum_crypto_items + && left.maximum_crypto_bytes = right.maximum_crypto_bytes) + ; prepare = + (fun inspection ((input : activation_input), check_current) -> + let* () = check_current () in + let* checksum = + match Types.Checksum.of_string input.snapshot.checksum with + | Ok checksum -> Ok checksum + | Error message -> Error (Dsl.Invalid_input message) + in + let* staged = + activation_result + (Database.prepare_snapshot_activation + dependencies + inspection + ~path:input.path + ~applied_server_cursor:input.snapshot.through + ~expected_checksum:(Some checksum) + ~expected_rows:input.expected_rows) + in + let preparation = + { staged + ; check_current + ; unprotect = input.unprotect + ; maximum_crypto_items = input.maximum_crypto_items + ; maximum_crypto_bytes = input.maximum_crypto_bytes + } + in + let transferred = ref false in + Fun.protect + ~finally:(fun () -> + if not !transferred + then + Eio.Cancel.protect (fun () -> Database.cancel_snapshot_activation staged)) + (fun () -> + let* () = unprotect preparation in + transferred := true; + Ok preparation)) + ; commit = + (fun preparation -> + let* () = preparation.check_current () in + activation_result (Database.commit_snapshot_activation preparation.staged)) + ; cancel = (fun preparation -> Database.cancel_snapshot_activation preparation.staged) + ; open_committed = + (fun inspection -> + match Database.open_ ~sw dependencies inspection ~graph_name with + | Ok database -> Ok database + | Error error -> Error (Dsl.Storage_failure (open_error error))) + } + in + Owner_binding.absent inspection backend +;; + +let activate_snapshot owner ~check_current input = + let* database = Owner_binding.activate owner ~check_current (input, check_current) in + let* snapshot = + match Database.current_snapshot database with + | Ok snapshot -> Ok snapshot + | Error _ -> + Error (Dsl.Storage_failure "Cannot inspect activated database generation") + in + Fun.protect + ~finally:(fun () -> Database.release_snapshot snapshot) + (fun () -> + let version = Database.snapshot_version snapshot in + Ok (Dsl.Activated version.generation)) +;; diff --git a/logseq_sync/lib/execute/database_owner.mli b/logseq_sync/lib/execute/database_owner.mli new file mode 100644 index 00000000..12d8ea85 --- /dev/null +++ b/logseq_sync/lib/execute/database_owner.mli @@ -0,0 +1,42 @@ +module Dsl = Sync_dsl.Core +module Overlay = Logseq_overlay_db + +type t + +type activation_input = + { snapshot : Dsl.snapshot + ; path : string + ; expected_rows : int + ; maximum_crypto_items : int + ; maximum_crypto_bytes : int + ; unprotect : + Overlay.Database.unprotection_request + -> ((Overlay.Types.crypto_item_id * string) list, Dsl.error) result + } + +val database_owner : Overlay.Database.t -> t + +val activation_owner + : sw:Eio.Switch.t + -> dependencies:Overlay.Database.dependencies + -> inspection:Overlay.Database.mirror_inspection + -> graph_name:string + -> t + +val database_if_open : t -> (Overlay.Database.t option, Dsl.error) result +val database : t -> (Overlay.Database.t, Dsl.error) result + +val with_database + : t + -> check_current:(unit -> (unit, Dsl.error) result) + -> (Overlay.Database.t -> ('a, Dsl.error) result) + -> ('a, Dsl.error) result + +(** Input paths and metadata must come from the original snapshot owner's ledger. + Preparation, commit, recovery and open retain the original inspection and queue. + The caller owns database close; the switch must outlive retirement. *) +val activate_snapshot + : t + -> check_current:(unit -> (unit, Dsl.error) result) + -> activation_input + -> (Dsl.activation, Dsl.error) result diff --git a/logseq_sync/lib/execute/dune b/logseq_sync/lib/execute/dune new file mode 100644 index 00000000..506b9693 --- /dev/null +++ b/logseq_sync/lib/execute/dune @@ -0,0 +1,12 @@ +(library + (name logseq_sync_execute_owners) + (public_name logseq_sync.internal.execute_owners) + (modules owner_binding database_owner) + (libraries eio sync_dsl logseq_overlay_db)) + +(library + (name logseq_sync_execute_impl) + (public_name logseq_sync.execute.impl) + (implements execute) + (modules core) + (libraries eio mtime sync_dsl logseq_sync_execute_owners logseq_overlay_db logseq_db_types rrbvec)) diff --git a/logseq_sync/lib/execute/owner_binding.ml b/logseq_sync/lib/execute/owner_binding.ml new file mode 100644 index 00000000..f91ad9b2 --- /dev/null +++ b/logseq_sync/lib/execute/owner_binding.ml @@ -0,0 +1,88 @@ +module Dsl = Sync_dsl.Core + +type ('database, 'inspection, 'input, 'prepared) backend = + { same_input : 'input -> 'input -> bool + ; prepare : 'inspection -> 'input -> ('prepared, Dsl.error) result + ; commit : 'prepared -> ('inspection, Dsl.error) result + ; cancel : 'prepared -> unit + ; open_committed : 'inspection -> ('database, Dsl.error) result + } + +type ('database, 'inspection, 'input, 'prepared) state = + | Absent of 'inspection * ('database, 'inspection, 'input, 'prepared) backend + | Committed of + 'inspection * ('database, 'inspection, 'input, 'prepared) backend * 'input + | Active of + 'database * ('input * ('database, 'inspection, 'input, 'prepared) backend) option + +type ('database, 'inspection, 'input, 'prepared) t = + { queue : Eio.Mutex.t + ; mutable state : ('database, 'inspection, 'input, 'prepared) state + } + +let active database = { queue = Eio.Mutex.create (); state = Active (database, None) } + +let absent inspection backend = + { queue = Eio.Mutex.create (); state = Absent (inspection, backend) } +;; + +let database_if_open owner = + Eio.Mutex.use_rw ~protect:false owner.queue (fun () -> + match owner.state with + | Active (database, _) -> Ok (Some database) + | Absent _ -> Ok None + | Committed _ -> Error (Dsl.Storage_failure "Committed mirror requires open recovery")) +;; + +let database owner = + match owner.state with + | Active (database, _) -> Ok database + | Absent _ -> Error (Dsl.Storage_failure "Mirror activation required") + | Committed _ -> Error (Dsl.Storage_failure "Committed mirror requires open recovery") +;; + +let ( let* ) = Result.bind + +let with_database owner ~check_current callback = + let* () = check_current () in + Eio.Mutex.use_rw ~protect:false owner.queue (fun () -> + let* () = check_current () in + let* database = database owner in + callback database) +;; + +let activate owner ~check_current input = + let* () = check_current () in + Eio.Mutex.use_rw ~protect:false owner.queue (fun () -> + let* () = check_current () in + let open_committed backend inspection = + Eio.Cancel.protect (fun () -> + let* database = backend.open_committed inspection in + owner.state <- Active (database, Some (input, backend)); + let* () = check_current () in + Ok database) + in + match owner.state with + | Active (database, Some (original, backend)) when backend.same_input original input + -> Ok database + | Active _ -> Error Dsl.Stale_context + | Committed (inspection, backend, original) -> + if backend.same_input original input + then open_committed backend inspection + else Error Dsl.Stale_context + | Absent (inspection, backend) -> + let* prepared = backend.prepare inspection input in + Fun.protect + ~finally:(fun () -> Eio.Cancel.protect (fun () -> backend.cancel prepared)) + (fun () -> + let* () = check_current () in + let* committed = + Eio.Cancel.protect (fun () -> + let* () = check_current () in + let* committed = backend.commit prepared in + owner.state <- Committed (committed, backend, input); + Ok committed) + in + let* () = check_current () in + open_committed backend committed)) +;; diff --git a/logseq_sync/lib/execute/owner_binding.mli b/logseq_sync/lib/execute/owner_binding.mli new file mode 100644 index 00000000..cd3dab37 --- /dev/null +++ b/logseq_sync/lib/execute/owner_binding.mli @@ -0,0 +1,41 @@ +module Dsl = Sync_dsl.Core + +type ('database, 'inspection, 'input, 'prepared) backend = + { same_input : 'input -> 'input -> bool + ; prepare : 'inspection -> 'input -> ('prepared, Dsl.error) result + ; commit : 'prepared -> ('inspection, Dsl.error) result + ; cancel : 'prepared -> unit + ; open_committed : 'inspection -> ('database, Dsl.error) result + } + +type ('database, 'inspection, 'input, 'prepared) t + +val active : 'database -> ('database, 'inspection, 'input, 'prepared) t + +val absent + : 'inspection + -> ('database, 'inspection, 'input, 'prepared) backend + -> ('database, 'inspection, 'input, 'prepared) t + +val database_if_open + : ('database, 'inspection, 'input, 'prepared) t + -> ('database option, Dsl.error) result + +val database + : ('database, 'inspection, 'input, 'prepared) t + -> ('database, Dsl.error) result + +val with_database + : ('database, 'inspection, 'input, 'prepared) t + -> check_current:(unit -> (unit, Dsl.error) result) + -> ('database -> ('value, Dsl.error) result) + -> ('value, Dsl.error) result + +(** Prepare may suspend. Commit follows a fresh check under the sole queue. + Committed inspection and opened binding survive cancellation; recovery never prepares or commits again. + The original owner supplies every backend operation; callers cannot replace it during recovery. *) +val activate + : ('database, 'inspection, 'input, 'prepared) t + -> check_current:(unit -> (unit, Dsl.error) result) + -> 'input + -> ('database, Dsl.error) result diff --git a/logseq_sync/lib/indexed_program/dune b/logseq_sync/lib/indexed_program/dune new file mode 100644 index 00000000..0dfb9b00 --- /dev/null +++ b/logseq_sync/lib/indexed_program/dune @@ -0,0 +1,4 @@ +(library + (name logseq_sync_indexed_program) + (public_name logseq_sync.internal.indexed_program) + (modules logseq_sync_indexed_program)) diff --git a/logseq_sync/lib/indexed_program/logseq_sync_indexed_program.ml b/logseq_sync/lib/indexed_program/logseq_sync_indexed_program.ml new file mode 100644 index 00000000..cc21ee04 --- /dev/null +++ b/logseq_sync/lib/indexed_program/logseq_sync_indexed_program.ml @@ -0,0 +1,74 @@ +module type EFFECTS = sig + type error + type (_, _, _, _) operation +end + +module Make (Effects : EFFECTS) = struct + type (_, _, _) t = + | Pure : 'a -> ('s, 's, 'a) t + | Failed : Effects.error -> ('s, 'final, 'a) t + | Effect : + ('before, 'after, 'failed, 'a) Effects.operation + * ('a -> ('after, 'final, 'b) t) + * (Effects.error -> ('failed, 'final, 'b) t) + -> ('before, 'final, 'b) t + | Bind : + ('before, 'middle, 'a) t * ('a -> ('middle, 'after, 'b) t) + -> ('before, 'after, 'b) t + + let return value = Pure value + let abort error = Failed error + let bind program continuation = Bind (program, continuation) + let perform operation ~success ~failure = Effect (operation, success, failure) + + type (_, _, _) node = + | Return : 'a -> ('s, 's, 'a) node + | Abort : Effects.error -> ('s, 'final, 'a) node + | Perform : + ('before, 'after, 'failed, 'a) Effects.operation + * ('a -> ('after, 'final, 'b) t) + * (Effects.error -> ('failed, 'final, 'b) t) + -> ('before, 'final, 'b) node + + type (_, _, _, _) continuations = + | Done : ('s, 's, 'a, 'a) continuations + | Next : + ('a -> ('middle, 'next, 'b) t) * ('next, 'final, 'b, 'result) continuations + -> ('middle, 'final, 'a, 'result) continuations + + let rec resume + : type before middle after value result. + (before, middle, value) t + -> (middle, after, value, result) continuations + -> (before, after, result) t + = + fun program continuations -> + match continuations with + | Done -> program + | Next (continuation, remaining) -> + Bind (program, fun value -> resume (continuation value) remaining) + ;; + + let rec normalize + : type before middle after value result. + (before, middle, value) t + -> (middle, after, value, result) continuations + -> (before, after, result) node + = + fun program continuations -> + match program with + | Pure value -> + (match continuations with + | Done -> Return value + | Next (continuation, remaining) -> normalize (continuation value) remaining) + | Failed error -> Abort error + | Effect (operation, success, failure) -> + Perform + ( operation + , (fun value -> resume (success value) continuations) + , fun error -> resume (failure error) continuations ) + | Bind (first, continuation) -> normalize first (Next (continuation, continuations)) + ;; + + let view program = normalize program Done +end diff --git a/logseq_sync/lib/indexed_program/logseq_sync_indexed_program.mli b/logseq_sync/lib/indexed_program/logseq_sync_indexed_program.mli new file mode 100644 index 00000000..57ef4ba1 --- /dev/null +++ b/logseq_sync/lib/indexed_program/logseq_sync_indexed_program.mli @@ -0,0 +1,33 @@ +module type EFFECTS = sig + type error + type (_, _, _, _) operation +end + +module Make (Effects : EFFECTS) : sig + type ('before, 'after, 'value) t + + val return : 'a -> ('s, 's, 'a) t + val abort : Effects.error -> ('s, 'final, 'a) t + + val bind + : ('before, 'middle, 'a) t + -> ('a -> ('middle, 'after, 'b) t) + -> ('before, 'after, 'b) t + + val perform + : ('before, 'after, 'failed, 'a) Effects.operation + -> success:('a -> ('after, 'final, 'b) t) + -> failure:(Effects.error -> ('failed, 'final, 'b) t) + -> ('before, 'final, 'b) t + + type (_, _, _) node = + | Return : 'a -> ('s, 's, 'a) node + | Abort : Effects.error -> ('s, 'final, 'a) node + | Perform : + ('before, 'after, 'failed, 'a) Effects.operation + * ('a -> ('after, 'final, 'b) t) + * (Effects.error -> ('failed, 'final, 'b) t) + -> ('before, 'final, 'b) node + + val view : ('before, 'after, 'a) t -> ('before, 'after, 'a) node +end diff --git a/logseq_sync/lib/effect_runner/protocol/asset_codec.ml b/logseq_sync/lib/native_crypto/asset_codec.ml similarity index 97% rename from logseq_sync/lib/effect_runner/protocol/asset_codec.ml rename to logseq_sync/lib/native_crypto/asset_codec.ml index 72c6eafa..d0708de1 100644 --- a/logseq_sync/lib/effect_runner/protocol/asset_codec.ml +++ b/logseq_sync/lib/native_crypto/asset_codec.ml @@ -132,9 +132,18 @@ let decode_error_message = function (* These roles stay inside the decoder: a failed candidate has no raw constructor path, and ciphertext cannot be fed directly to verification. *) type raw = Raw of string -type envelope = Envelope of { iv : string; ciphertext : string } + +type envelope = + | Envelope of + { iv : string + ; ciphertext : string + } + type plaintext = Plaintext of string -type classified = Clear of raw | Protected of envelope + +type classified = + | Clear of raw + | Protected of envelope type verified = { bytes : string @@ -182,7 +191,7 @@ let decode ~maximum_plaintext_bytes ~crypto ~key ~expected wire = then Error Wire_too_large else let* classified = classify ~maximum_plaintext_bytes ~key wire in - let* Plaintext plaintext = + let* (Plaintext plaintext) = match classified, key with | Clear (Raw bytes), _ -> Ok (Plaintext bytes) | Protected (Envelope { iv; ciphertext }), Some key -> diff --git a/logseq_sync/lib/native_crypto/dune b/logseq_sync/lib/native_crypto/dune new file mode 100644 index 00000000..5a6793b0 --- /dev/null +++ b/logseq_sync/lib/native_crypto/dune @@ -0,0 +1,8 @@ +(library + (name logseq_sync_native_crypto) + (public_name logseq_sync.native_crypto) + (modules platform_crypto e2ee asset_codec) + (libraries eio uri digestif logseq_db_types melange-transit-core melange-transit-native ppx_deriving_yojson.runtime yojson) + (flags (:standard -open Logseq_db_types)) + (preprocess (pps ppx_deriving_yojson)) + (foreign_stubs (language c) (names platform_crypto_stubs))) diff --git a/logseq_sync/lib/effect_runner/protocol/e2ee.ml b/logseq_sync/lib/native_crypto/e2ee.ml similarity index 65% rename from logseq_sync/lib/effect_runner/protocol/e2ee.ml rename to logseq_sync/lib/native_crypto/e2ee.ml index 0ee069cd..25dbff01 100644 --- a/logseq_sync/lib/effect_runner/protocol/e2ee.ml +++ b/logseq_sync/lib/native_crypto/e2ee.ml @@ -19,19 +19,32 @@ let bind result f = ;; let protect parse_operation f = - let operation_name = match parse_operation with + let operation_name = + match parse_operation with | "decode protected value" | "decode decrypted value" -> "decryptValue" - | _ -> "platformCrypto" in + | _ -> "platformCrypto" + in let parse_failure () = - Error (Logseq_db_types.Failure_diagnostic.( - create ~operation:operation_name ~reason:"parseFailed" - ~source:(parse_operation ^ " failed") () |> message)) in + Error + Logseq_db_types.Failure_diagnostic.( + create + ~operation:operation_name + ~reason:"parseFailed" + ~source:(parse_operation ^ " failed") + () + |> message) + in try Ok (f ()) with | Transit.Decode_error _ | Yojson.Json_error _ | Failure _ | Invalid_argument _ -> parse_failure () - | error -> Error (Logseq_db_types.Failure_diagnostic.( - of_source ~operation:operation_name ~reason:"parseFailed" - (Printexc.to_string error) |> message)) + | error -> + Error + Logseq_db_types.Failure_diagnostic.( + of_source + ~operation:operation_name + ~reason:"parseFailed" + (Printexc.to_string error) + |> message) ;; let bounded_string field value = @@ -64,10 +77,17 @@ let graph_key_response source = graph_key_response_dto_of_yojson (Yojson.Safe.from_string source) |> Result.map_error (fun message -> - let fields = List.map (fun (field, wire) -> "E2ee.graph_key_response_dto." ^ field, - "field " ^ wire ^ " is missing or has an invalid type") - ["encrypted_aes_key", "encrypted-aes-key"; "encrypted_aes_key_base64", "encrypted-aes-key-base64"] in - "E2EE graph key response: " ^ Logseq_db_types.Json_codec.error_context ~fields message)) + let fields = + List.map + (fun (field, wire) -> + ( "E2ee.graph_key_response_dto." ^ field + , "field " ^ wire ^ " is missing or has an invalid type" )) + [ "encrypted_aes_key", "encrypted-aes-key" + ; "encrypted_aes_key_base64", "encrypted-aes-key-base64" + ] + in + "E2EE graph key response: " + ^ Logseq_db_types.Json_codec.error_context ~fields message)) (fun dto -> (* The optional compatibility field is validated by PPX but not consumed. *) let _ = dto.encrypted_aes_key_base64 in @@ -83,10 +103,17 @@ let user_keys_response source = user_keys_response_dto_of_yojson (Yojson.Safe.from_string source) |> Result.map_error (fun message -> - let fields = List.map (fun (field, wire) -> "E2ee.user_keys_response_dto." ^ field, - "field " ^ wire ^ " is missing or has an invalid type") - ["public_key", "public-key"; "encrypted_private_key", "encrypted-private-key"] in - "E2EE user keys response: " ^ Logseq_db_types.Json_codec.error_context ~fields message)) + let fields = + List.map + (fun (field, wire) -> + ( "E2ee.user_keys_response_dto." ^ field + , "field " ^ wire ^ " is missing or has an invalid type" )) + [ "public_key", "public-key" + ; "encrypted_private_key", "encrypted-private-key" + ] + in + "E2EE user keys response: " + ^ Logseq_db_types.Json_codec.error_context ~fields message)) (fun dto -> bind (bounded_string "public-key" dto.public_key) (fun _ -> bounded_string "encrypted-private-key" dto.encrypted_private_key)) @@ -121,14 +148,20 @@ let decrypt_value ~crypto ~graph_key source = | Ok (Transit.Array [ Transit.Binary iv; Transit.Binary ciphertext ]) -> (match crypto.decrypt_aes_gcm ~key:graph_key ~iv ~ciphertext with | Error source -> - Error (Logseq_db_types.Failure_diagnostic.( - of_source ~operation:"decryptValue" ~reason:"cryptoInvalid" source |> encode)) + Error + Logseq_db_types.Failure_diagnostic.( + of_source ~operation:"decryptValue" ~reason:"cryptoInvalid" source |> encode) | Ok plaintext -> (match protect "decode decrypted value" (fun () -> Codec.of_string plaintext) with | Ok value -> Ok value | Error _ -> - Error (Logseq_db_types.Failure_diagnostic.( - create ~operation:"decryptValue" ~reason:"parseFailed" - ~source:"decrypted Transit is invalid" () |> encode)))) + Error + Logseq_db_types.Failure_diagnostic.( + create + ~operation:"decryptValue" + ~reason:"parseFailed" + ~source:"decrypted Transit is invalid" + () + |> encode))) | Ok plaintext -> Ok plaintext ;; diff --git a/logseq_sync/lib/effect_runner/protocol/e2ee.mli b/logseq_sync/lib/native_crypto/e2ee.mli similarity index 100% rename from logseq_sync/lib/effect_runner/protocol/e2ee.mli rename to logseq_sync/lib/native_crypto/e2ee.mli diff --git a/logseq_sync/lib/effect_runner/platform/platform_crypto.ml b/logseq_sync/lib/native_crypto/platform_crypto.ml similarity index 71% rename from logseq_sync/lib/effect_runner/platform/platform_crypto.ml rename to logseq_sync/lib/native_crypto/platform_crypto.ml index eedead6b..bcedf176 100644 --- a/logseq_sync/lib/effect_runner/platform/platform_crypto.ml +++ b/logseq_sync/lib/native_crypto/platform_crypto.ml @@ -66,13 +66,26 @@ type response = [@@deriving yojson] let crypto_error_codes = - [ "cryptoInvalidRequest"; "cryptoAuthenticationFailed"; "cryptoInvalidPrivateKey" - ; "cryptoRSADecryptionFailed"; "cryptoInvalidGraphKeyLength"; "cryptoKeyStoreLocked" - ; "cryptoKeyStorePermissionDenied"; "cryptoKeyStoreUnavailable"; "cryptoKeyStoreDecodeFailed" - ; "cryptoKeyStoreFailed"; "cryptoKeyStoreInvalidType"; "cryptoOperationFailed"; "wrappedGraphKeyUnavailable" - ; "cryptoInvalidAESInput"; "cryptoEnginePrivateKeyUnavailable"; "cryptoAllocationFailed" - ; "cryptoNativeUnavailable"; "cryptoNativeResponseUnavailable" - ; "platform crypto unavailable" ] + [ "cryptoInvalidRequest" + ; "cryptoAuthenticationFailed" + ; "cryptoInvalidPrivateKey" + ; "cryptoRSADecryptionFailed" + ; "cryptoInvalidGraphKeyLength" + ; "cryptoKeyStoreLocked" + ; "cryptoKeyStorePermissionDenied" + ; "cryptoKeyStoreUnavailable" + ; "cryptoKeyStoreDecodeFailed" + ; "cryptoKeyStoreFailed" + ; "cryptoKeyStoreInvalidType" + ; "cryptoOperationFailed" + ; "wrappedGraphKeyUnavailable" + ; "cryptoInvalidAESInput" + ; "cryptoEnginePrivateKeyUnavailable" + ; "cryptoAllocationFailed" + ; "cryptoNativeUnavailable" + ; "cryptoNativeResponseUnavailable" + ; "platform crypto unavailable" + ] ;; let diagnostic_message ~reason:reason_name source = @@ -81,51 +94,94 @@ let diagnostic_message ~reason:reason_name source = ;; let response_context fields = - let reason = match fields.reason with - | Some ("wrapped-key-absent" | "wrapped-key-cache-invalid" as reason) -> Some reason - | Some value when List.exists (fun code -> - value = "wrapped-key-verification-failed:" ^ code) crypto_error_codes -> Some value - | _ -> None in - let context = Option.fold ~none:"" ~some:(fun value -> " Context: " ^ value ^ ".") reason in - context ^ if fields.cleanup_pending = Some true then " Wrapped-key cleanup remains pending." else "" + let reason = + match fields.reason with + | Some (("wrapped-key-absent" | "wrapped-key-cache-invalid") as reason) -> Some reason + | Some value + when List.exists + (fun code -> value = "wrapped-key-verification-failed:" ^ code) + crypto_error_codes -> Some value + | _ -> None + in + let context = + Option.fold ~none:"" ~some:(fun value -> " Context: " ^ value ^ ".") reason + in + context + ^ + if fields.cleanup_pending = Some true + then " Wrapped-key cleanup remains pending." + else "" ;; -type invocation_failure = { recovery_code : string option; diagnostic : string } +type invocation_failure = + { recovery_code : string option + ; diagnostic : string + } let failure ?recovery_code diagnostic = Error { recovery_code; diagnostic } let invoke_response request = try - let request = { request with diagnostic_operation = - (if Sys.getenv_opt "JOURNAL_UNLOCK_DIAGNOSTICS" = Some "1" - then Eio.Fiber.get diagnostic_operation else None) } in + let request = + { request with + diagnostic_operation = + (if Sys.getenv_opt "JOURNAL_UNLOCK_DIAGNOSTICS" = Some "1" + then Eio.Fiber.get diagnostic_operation + else None) + } + in let source = call_raw (request_to_yojson request |> Yojson.Safe.to_string) in - let decoded = try Ok (Yojson.Safe.from_string source) with - | Yojson.Json_error _ -> failure (diagnostic_message ~reason:"parseFailed" - "platform crypto response JSON is invalid") in + let decoded = + try Ok (Yojson.Safe.from_string source) with + | Yojson.Json_error _ -> + failure + (diagnostic_message + ~reason:"parseFailed" + "platform crypto response JSON is invalid") + in Result.bind decoded (fun json -> match Logseq_db_types.Json_codec.decode response_of_yojson json with | Ok { ok = true; _ } as result -> result (* This exact error selects the existing local-key recovery branch. *) | Ok ({ ok = false; error = Some "localPrivateKeyUnavailable"; _ } as fields) -> - failure ~recovery_code:"localPrivateKeyUnavailable" + failure + ~recovery_code:"localPrivateKeyUnavailable" ("local private key is unavailable" ^ response_context fields) | Ok ({ ok = false; error = Some error; _ } as fields) -> - let source = if List.mem error crypto_error_codes then error - else "platform crypto provider failed" in - let reason = if List.mem error ["platform crypto unavailable"; "cryptoNativeUnavailable"; - "cryptoNativeResponseUnavailable"] then "unavailable" else "cryptoInvalid" in + let source = + if List.mem error crypto_error_codes + then error + else "platform crypto provider failed" + in + let reason = + if + List.mem + error + [ "platform crypto unavailable" + ; "cryptoNativeUnavailable" + ; "cryptoNativeResponseUnavailable" + ] + then "unavailable" + else "cryptoInvalid" + in failure (diagnostic_message ~reason source ^ response_context fields) - | Ok _ -> failure (diagnostic_message ~reason:"cryptoInvalid" - "platform crypto response error is missing") - | Error _ -> failure (diagnostic_message ~reason:"parseFailed" - "platform crypto response shape is invalid")) + | Ok _ -> + failure + (diagnostic_message + ~reason:"cryptoInvalid" + "platform crypto response error is missing") + | Error _ -> + failure + (diagnostic_message + ~reason:"parseFailed" + "platform crypto response shape is invalid")) with | error -> failure (diagnostic_message ~reason:"unavailable" (Printexc.to_string error)) ;; let invoke request = - invoke_response request |> Result.map_error (fun failure -> + invoke_response request + |> Result.map_error (fun failure -> Option.value failure.recovery_code ~default:failure.diagnostic) ;; @@ -261,16 +317,23 @@ let verify_and_save_wrapped_graph_key ;; let delete_account_secrets ~managed_sync_origin ~user_id = - let result = bind - (invoke (identity_request "deleteAccountSecrets" ~managed_sync_origin ~user_id)) - (fun _ -> Ok ()) in + let result = + bind + (invoke (identity_request "deleteAccountSecrets" ~managed_sync_origin ~user_id)) + (fun _ -> Ok ()) + in (match result with | Ok () -> () | Error diagnostic -> (* [invoke] has already reduced native data to a bounded safe message. Secondary account cleanup retains its original completion strategy. *) - (try Printf.eprintf - "journal: phase=cleanup operation=deleteAccountSecrets code=cleanupPending detail=%s\n%!" - diagnostic with _ -> ())); + (try + Printf.eprintf + "journal: phase=cleanup operation=deleteAccountSecrets code=cleanupPending \ + detail=%s\n\ + %!" + diagnostic + with + | _ -> ())); result ;; diff --git a/logseq_sync/lib/effect_runner/platform/platform_crypto.mli b/logseq_sync/lib/native_crypto/platform_crypto.mli similarity index 100% rename from logseq_sync/lib/effect_runner/platform/platform_crypto.mli rename to logseq_sync/lib/native_crypto/platform_crypto.mli diff --git a/logseq_sync/lib/effect_runner/platform/platform_crypto_stubs.c b/logseq_sync/lib/native_crypto/platform_crypto_stubs.c similarity index 73% rename from logseq_sync/lib/effect_runner/platform/platform_crypto_stubs.c rename to logseq_sync/lib/native_crypto/platform_crypto_stubs.c index 6fbaa9fc..d63ad03e 100644 --- a/logseq_sync/lib/effect_runner/platform/platform_crypto_stubs.c +++ b/logseq_sync/lib/native_crypto/platform_crypto_stubs.c @@ -1,11 +1,11 @@ #ifdef __linux__ -# define _GNU_SOURCE +#define _GNU_SOURCE #endif +#include #include #include #include -#include #include #include @@ -20,20 +20,22 @@ CAMLprim value logseq_journal_crypto_call(value request) { char *request_copy = strdup(String_val(request)); char *response = NULL; if (request_copy == NULL) { - CAMLreturn(caml_copy_string("{\"ok\":false,\"error\":\"cryptoAllocationFailed\"}")); + CAMLreturn(caml_copy_string( + "{\"ok\":false,\"error\":\"cryptoAllocationFailed\"}")); } caml_enter_blocking_section(); - logseq_journal_crypto_fn crypto = - (logseq_journal_crypto_fn)dlsym(RTLD_DEFAULT, "logseq_journal_crypto_json"); + logseq_journal_crypto_fn crypto = (logseq_journal_crypto_fn)dlsym( + RTLD_DEFAULT, "logseq_journal_crypto_json"); if (crypto != NULL) { response = crypto(request_copy); } caml_leave_blocking_section(); free(request_copy); if (response == NULL) { - CAMLreturn(caml_copy_string(crypto == NULL - ? "{\"ok\":false,\"error\":\"cryptoNativeUnavailable\"}" - : "{\"ok\":false,\"error\":\"cryptoNativeResponseUnavailable\"}")); + CAMLreturn(caml_copy_string( + crypto == NULL + ? "{\"ok\":false,\"error\":\"cryptoNativeUnavailable\"}" + : "{\"ok\":false,\"error\":\"cryptoNativeResponseUnavailable\"}")); } result = caml_copy_string(response); free(response); @@ -46,8 +48,8 @@ typedef unsigned char *(*logseq_journal_binary_fn)( int, const unsigned char *, size_t, const unsigned char *, size_t, const unsigned char *, size_t, intptr_t *); -CAMLprim value logseq_journal_crypto_binary_call( - value encrypt, value key, value iv, value input) { +CAMLprim value logseq_journal_crypto_binary_call(value encrypt, value key, + value iv, value input) { CAMLparam4(encrypt, key, iv, input); CAMLlocal2(result, payload); const size_t limit = 8 * 1024 * 1024; @@ -68,8 +70,8 @@ CAMLprim value logseq_journal_crypto_binary_call( memcpy(copy + key_length, String_val(iv), iv_length); memcpy(copy + key_length + iv_length, String_val(input), input_length); caml_enter_blocking_section(); - logseq_journal_binary_fn crypto = (logseq_journal_binary_fn) - dlsym(RTLD_DEFAULT, "logseq_journal_crypto_binary"); + logseq_journal_binary_fn crypto = (logseq_journal_binary_fn)dlsym( + RTLD_DEFAULT, "logseq_journal_crypto_binary"); failure = "platform binary crypto unavailable"; if (crypto != NULL) { response = crypto(sealing, copy, key_length, copy + key_length, @@ -78,10 +80,17 @@ CAMLprim value logseq_journal_crypto_binary_call( failure = "platform binary crypto operation failed"; if (response == NULL) { switch (response_length) { - case -1: failure = "platform binary crypto invalid request"; break; - case -2: failure = "platform binary crypto authentication failed"; break; - case -3: failure = "platform binary crypto allocation failed"; break; - default: break; /* Includes older native implementations returning zero. */ + case -1: + failure = "platform binary crypto invalid request"; + break; + case -2: + failure = "platform binary crypto authentication failed"; + break; + case -3: + failure = "platform binary crypto allocation failed"; + break; + default: + break; /* Includes older native implementations returning zero. */ } } } @@ -90,17 +99,20 @@ CAMLprim value logseq_journal_crypto_binary_call( free(copy); } } - /* Check signed sentinels before converting lengths or touching native bytes. */ + /* Check signed sentinels before converting lengths or touching native bytes. + */ if (response == NULL) { payload = caml_copy_string(failure); result = caml_alloc(1, 1); /* Error */ } else if (response_length < 0 || - (size_t)response_length != (sealing ? input_length + 28 : input_length - 16)) { + (size_t)response_length != + (sealing ? input_length + 28 : input_length - 16)) { free(response); payload = caml_copy_string("platform binary crypto invalid response"); result = caml_alloc(1, 1); } else { - payload = caml_alloc_initialized_string((size_t)response_length, (const char *)response); + payload = caml_alloc_initialized_string((size_t)response_length, + (const char *)response); free(response); result = caml_alloc(1, 0); /* Ok */ } @@ -110,21 +122,24 @@ CAMLprim value logseq_journal_crypto_binary_call( #include "transport_dns_stubs.h" -/* Restore a retired directory without overwriting a concurrently created copy. */ +/* Restore a retired directory without overwriting a concurrently created copy. + */ #include -#include #include +#include #ifdef __linux__ -#include #include +#include #endif CAMLprim value logseq_journal_rename_exclusive(value source, value target) { CAMLparam2(source, target); int rc; #ifdef __APPLE__ - rc = renameatx_np(AT_FDCWD, String_val(source), AT_FDCWD, String_val(target), RENAME_EXCL); + rc = renameatx_np(AT_FDCWD, String_val(source), AT_FDCWD, String_val(target), + RENAME_EXCL); #elif defined(__linux__) && defined(SYS_renameat2) - rc = syscall(SYS_renameat2, AT_FDCWD, String_val(source), AT_FDCWD, String_val(target), 1); + rc = syscall(SYS_renameat2, AT_FDCWD, String_val(source), AT_FDCWD, + String_val(target), 1); #else errno = ENOTSUP; rc = -1; diff --git a/logseq_sync/lib/effect_runner/platform/transport_dns_stubs.h b/logseq_sync/lib/native_crypto/transport_dns_stubs.h similarity index 63% rename from logseq_sync/lib/effect_runner/platform/transport_dns_stubs.h rename to logseq_sync/lib/native_crypto/transport_dns_stubs.h index 4f6b95bc..6c880da1 100644 --- a/logseq_sync/lib/effect_runner/platform/transport_dns_stubs.h +++ b/logseq_sync/lib/native_crypto/transport_dns_stubs.h @@ -1,5 +1,6 @@ -/* Cancellable Apple DNS-SD queries. All callbacks run synchronously from process; - neither OCaml callbacks nor background resolver threads retain this owner. */ +/* Cancellable Apple DNS-SD queries. All callbacks run synchronously from + process; neither OCaml callbacks nor background resolver threads retain this + owner. */ #include #include #ifdef __APPLE__ @@ -16,30 +17,39 @@ struct logseq_dns_query { }; static void logseq_dns_release(value handle) { - struct logseq_dns_query *query = *(struct logseq_dns_query **)Data_custom_val(handle); + struct logseq_dns_query *query = + *(struct logseq_dns_query **)Data_custom_val(handle); if (query != NULL) { - if (query->service != NULL) DNSServiceRefDeallocate(query->service); + if (query->service != NULL) + DNSServiceRefDeallocate(query->service); free(query); *(struct logseq_dns_query **)Data_custom_val(handle) = NULL; } } static struct custom_operations logseq_dns_operations = { - "logseq.transport.dns", logseq_dns_release, custom_compare_default, - custom_hash_default, custom_serialize_default, custom_deserialize_default, - custom_compare_ext_default, custom_fixed_length_default -}; + "logseq.transport.dns", logseq_dns_release, + custom_compare_default, custom_hash_default, + custom_serialize_default, custom_deserialize_default, + custom_compare_ext_default, custom_fixed_length_default}; static struct logseq_dns_query *logseq_dns_get(value handle) { - struct logseq_dns_query *query = *(struct logseq_dns_query **)Data_custom_val(handle); - if (query == NULL) caml_invalid_argument("closed DNS query"); + struct logseq_dns_query *query = + *(struct logseq_dns_query **)Data_custom_val(handle); + if (query == NULL) + caml_invalid_argument("closed DNS query"); return query; } static void logseq_dns_reply(DNSServiceRef service, DNSServiceFlags flags, - uint32_t interface_index, DNSServiceErrorType error, const char *hostname, - const struct sockaddr *address, uint32_t ttl, void *context) { - (void)service; (void)interface_index; (void)hostname; (void)ttl; + uint32_t interface_index, + DNSServiceErrorType error, const char *hostname, + const struct sockaddr *address, uint32_t ttl, + void *context) { + (void)service; + (void)interface_index; + (void)hostname; + (void)ttl; struct logseq_dns_query *query = context; if (error != kDNSServiceErr_NoError) { query->error = error == kDNSServiceErr_NoSuchRecord ? 0 : error; @@ -50,19 +60,24 @@ static void logseq_dns_reply(DNSServiceRef service, DNSServiceFlags flags, const void *bytes = NULL; int length = 0; if (address->sa_family == AF_INET) { - bytes = &((const struct sockaddr_in *)address)->sin_addr; length = 4; + bytes = &((const struct sockaddr_in *)address)->sin_addr; + length = 4; } else if (address->sa_family == AF_INET6) { const struct sockaddr_in6 *ipv6 = (const struct sockaddr_in6 *)address; /* Eio's TCP address has no interface scope field. Do not erase a required link-local scope and accidentally connect through another interface. */ - if (ipv6->sin6_scope_id == 0) { bytes = &ipv6->sin6_addr; length = 16; } + if (ipv6->sin6_scope_id == 0) { + bytes = &ipv6->sin6_addr; + length = 16; + } } if (bytes != NULL) { query->lengths[query->count] = length; memcpy(query->addresses[query->count++], bytes, length); } } - if (!(flags & kDNSServiceFlagsMoreComing)) query->done = 1; + if (!(flags & kDNSServiceFlagsMoreComing)) + query->done = 1; } CAMLprim value logseq_transport_dns_start(value hostname, value ipv6) { @@ -71,12 +86,13 @@ CAMLprim value logseq_transport_dns_start(value hostname, value ipv6) { handle = caml_alloc_custom(&logseq_dns_operations, sizeof(void *), 0, 1); *(struct logseq_dns_query **)Data_custom_val(handle) = NULL; struct logseq_dns_query *query = calloc(1, sizeof(*query)); - if (query == NULL) caml_raise_out_of_memory(); + if (query == NULL) + caml_raise_out_of_memory(); *(struct logseq_dns_query **)Data_custom_val(handle) = query; /* Complete an absent family through NoSuchRecord instead of waiting out the shared DNS/TCP deadline while the other family already has addresses. */ - DNSServiceErrorType error = DNSServiceGetAddrInfo(&query->service, - kDNSServiceFlagsReturnIntermediates, 0, + DNSServiceErrorType error = DNSServiceGetAddrInfo( + &query->service, kDNSServiceFlagsReturnIntermediates, 0, Bool_val(ipv6) ? kDNSServiceProtocol_IPv6 : kDNSServiceProtocol_IPv4, String_val(hostname), logseq_dns_reply, query); if (error != kDNSServiceErr_NoError) { @@ -89,7 +105,8 @@ CAMLprim value logseq_transport_dns_start(value hostname, value ipv6) { CAMLprim value logseq_transport_dns_fd(value handle) { CAMLparam1(handle); int fd = DNSServiceRefSockFD(logseq_dns_get(handle)->service); - if (fd < 0) caml_failwith("DNS query has no descriptor"); + if (fd < 0) + caml_failwith("DNS query has no descriptor"); CAMLreturn(Val_int(fd)); } @@ -98,7 +115,8 @@ CAMLprim value logseq_transport_dns_process(value handle) { struct logseq_dns_query *query = logseq_dns_get(handle); if (DNSServiceProcessResult(query->service) != kDNSServiceErr_NoError) caml_failwith("DNS query processing failed"); - if (query->error) caml_failwith("DNS query failed"); + if (query->error) + caml_failwith("DNS query failed"); CAMLreturn(Val_bool(query->done)); } @@ -108,9 +126,12 @@ CAMLprim value logseq_transport_dns_results(value handle) { struct logseq_dns_query *query = logseq_dns_get(handle); list = Val_emptylist; for (int i = query->count - 1; i >= 0; --i) { - address = caml_alloc_initialized_string(query->lengths[i], (char *)query->addresses[i]); + address = caml_alloc_initialized_string(query->lengths[i], + (char *)query->addresses[i]); cell = caml_alloc(2, 0); - Store_field(cell, 0, address); Store_field(cell, 1, list); list = cell; + Store_field(cell, 0, address); + Store_field(cell, 1, list); + list = cell; } CAMLreturn(list); } @@ -121,20 +142,27 @@ CAMLprim value logseq_transport_dns_close(value handle) { CAMLreturn(Val_unit); } #else -/* The application targets Apple platforms; unsupported platforms fail explicitly. */ +/* The application targets Apple platforms; unsupported platforms fail + * explicitly. */ CAMLprim value logseq_transport_dns_start(value host, value ipv6) { - (void)host; (void)ipv6; caml_failwith("Apple DNS-SD is required"); + (void)host; + (void)ipv6; + caml_failwith("Apple DNS-SD is required"); } CAMLprim value logseq_transport_dns_fd(value handle) { - (void)handle; caml_failwith("Apple DNS-SD is required"); + (void)handle; + caml_failwith("Apple DNS-SD is required"); } CAMLprim value logseq_transport_dns_process(value handle) { - (void)handle; caml_failwith("Apple DNS-SD is required"); + (void)handle; + caml_failwith("Apple DNS-SD is required"); } CAMLprim value logseq_transport_dns_results(value handle) { - (void)handle; caml_failwith("Apple DNS-SD is required"); + (void)handle; + caml_failwith("Apple DNS-SD is required"); } CAMLprim value logseq_transport_dns_close(value handle) { - (void)handle; return Val_unit; + (void)handle; + return Val_unit; } #endif diff --git a/logseq_sync/lib/effect_runner/platform/artifact_decoder.ml b/logseq_sync/lib/native_transport/artifact_decoder.ml similarity index 63% rename from logseq_sync/lib/effect_runner/platform/artifact_decoder.ml rename to logseq_sync/lib/native_transport/artifact_decoder.ml index d920220b..9d7738ed 100644 --- a/logseq_sync/lib/effect_runner/platform/artifact_decoder.ml +++ b/logseq_sync/lib/native_transport/artifact_decoder.ml @@ -3,4 +3,4 @@ external decompress_gzip -> string -> int -> int - = "logseq_journal_gzip_decompress" + = "logseq_transport_gzip_decompress" diff --git a/logseq_sync/lib/effect_runner/platform/artifact_decoder.mli b/logseq_sync/lib/native_transport/artifact_decoder.mli similarity index 100% rename from logseq_sync/lib/effect_runner/platform/artifact_decoder.mli rename to logseq_sync/lib/native_transport/artifact_decoder.mli diff --git a/logseq_sync/lib/native_transport/dune b/logseq_sync/lib/native_transport/dune new file mode 100644 index 00000000..46449ad5 --- /dev/null +++ b/logseq_sync/lib/native_transport/dune @@ -0,0 +1,7 @@ +(library + (name logseq_sync_native_transport) + (public_name logseq_sync.native_transport) + (modules transport_domain transport_time tls_client_eio websocket_eio http http_eio snapshot_artifact artifact_decoder snapshot_parser snapshot_checksum snapshot_verify) + (foreign_stubs (language c) (names gzip_stubs)) + (libraries yojson logseq_overlay_db melange-transit-core melange-transit-native sqlite3 bigstringaf cstruct digestif domain-name eio eio.unix faraday httpun httpun-eio httpun-ws mtime logseq_db_types logseq_sync.native_crypto mirage-crypto-rng mirage-crypto-rng.unix tls tls-eio unix uri x509) + (flags (:standard -open Logseq_db_types))) diff --git a/logseq_sync/lib/effect_runner/platform/gzip_stubs.c b/logseq_sync/lib/native_transport/gzip_stubs.c similarity index 73% rename from logseq_sync/lib/effect_runner/platform/gzip_stubs.c rename to logseq_sync/lib/native_transport/gzip_stubs.c index d7e46088..45b02660 100644 --- a/logseq_sync/lib/effect_runner/platform/gzip_stubs.c +++ b/logseq_sync/lib/native_transport/gzip_stubs.c @@ -1,12 +1,12 @@ +#include +#include #include #include -#include #include #include -#include -#include #include +#include /* zlib is resolved at runtime (same pattern as platform_crypto_stubs.c) so the stubs archive carries no link-time -lz dependency on platforms without @@ -33,32 +33,34 @@ static int zlib_resolve(void) { zlib_inflate = (zlib_inflate_fn)dlsym(handle, "inflate"); zlib_inflate_end = (zlib_inflate_end_fn)dlsym(handle, "inflateEnd"); } - resolved = - (zlib_inflate_init2 != NULL && zlib_inflate != NULL && - zlib_inflate_end != NULL) - ? 1 - : 0; + resolved = (zlib_inflate_init2 != NULL && zlib_inflate != NULL && + zlib_inflate_end != NULL) + ? 1 + : 0; } return resolved; } -CAMLprim value logseq_journal_gzip_decompress( - value source_value, - value destination_value, - value maximum_bytes_value) { +CAMLprim value logseq_transport_gzip_decompress(value source_value, + value destination_value, + value maximum_bytes_value) { CAMLparam3(source_value, destination_value, maximum_bytes_value); const char *source = String_val(source_value); const char *destination = String_val(destination_value); intnat maximum_bytes = Long_val(maximum_bytes_value); - if (maximum_bytes <= 0) CAMLreturn(Val_int(5)); - if (!zlib_resolve()) CAMLreturn(Val_int(3)); + if (maximum_bytes <= 0) + CAMLreturn(Val_int(5)); + if (!zlib_resolve()) + CAMLreturn(Val_int(3)); FILE *input = fopen(source, "rb"); - if (input == NULL) CAMLreturn(Val_int(1)); + if (input == NULL) + CAMLreturn(Val_int(1)); int output_fd = open(destination, O_WRONLY | O_CREAT | O_EXCL, 0600); FILE *output = output_fd < 0 ? NULL : fdopen(output_fd, "wb"); if (output == NULL) { - if (output_fd >= 0) close(output_fd); + if (output_fd >= 0) + close(output_fd); fclose(input); CAMLreturn(Val_int(2)); } @@ -66,8 +68,8 @@ CAMLprim value logseq_journal_gzip_decompress( z_stream stream; memset(&stream, 0, sizeof(stream)); int status = 0; - if (zlib_inflate_init2( - &stream, 15 + 16, ZLIB_VERSION, (int)sizeof(z_stream)) != Z_OK) { + if (zlib_inflate_init2(&stream, 15 + 16, ZLIB_VERSION, + (int)sizeof(z_stream)) != Z_OK) { status = 3; } else { unsigned char input_buffer[16384]; @@ -75,7 +77,8 @@ CAMLprim value logseq_journal_gzip_decompress( intnat produced_total = 0; int finished = 0; while (!finished && status == 0) { - stream.avail_in = (uInt)fread(input_buffer, 1, sizeof(input_buffer), input); + stream.avail_in = + (uInt)fread(input_buffer, 1, sizeof(input_buffer), input); if (ferror(input)) { status = 3; break; @@ -98,7 +101,8 @@ CAMLprim value logseq_journal_gzip_decompress( status = 5; break; } - if (produced > 0 && fwrite(output_buffer, 1, produced, output) != produced) { + if (produced > 0 && + fwrite(output_buffer, 1, produced, output) != produced) { status = 4; break; } @@ -111,8 +115,11 @@ CAMLprim value logseq_journal_gzip_decompress( } zlib_inflate_end(&stream); } - if (fclose(input) != 0 && status == 0) status = 3; - if (fclose(output) != 0 && status == 0) status = 4; - if (status != 0) remove(destination); + if (fclose(input) != 0 && status == 0) + status = 3; + if (fclose(output) != 0 && status == 0) + status = 4; + if (status != 0) + remove(destination); CAMLreturn(Val_int(status)); } diff --git a/logseq_sync/lib/effect_runner/eio/http.ml b/logseq_sync/lib/native_transport/http.ml similarity index 96% rename from logseq_sync/lib/effect_runner/eio/http.ml rename to logseq_sync/lib/native_transport/http.ml index b217b18f..cd9dc117 100644 --- a/logseq_sync/lib/effect_runner/eio/http.ml +++ b/logseq_sync/lib/native_transport/http.ml @@ -32,7 +32,10 @@ let require_base_url base_url = ;; let authorization token = - if String.length token = 0 || String.length token > 1024 * 1024 + if + String.length token = 0 + || String.length token > 1024 * 1024 + || String.exists (fun character -> character = '\r' || character = '\n') token then invalid_arg "ID token must be bounded and non-empty"; [ "authorization", "Bearer " ^ token; "accept", "application/transit+json" ] ;; diff --git a/logseq_sync/lib/effect_runner/eio/http.mli b/logseq_sync/lib/native_transport/http.mli similarity index 92% rename from logseq_sync/lib/effect_runner/eio/http.mli rename to logseq_sync/lib/native_transport/http.mli index e7bfc557..e0beac4d 100644 --- a/logseq_sync/lib/effect_runner/eio/http.mli +++ b/logseq_sync/lib/native_transport/http.mli @@ -36,7 +36,9 @@ val e2ee_graph_key -> request val e2ee_user_keys : base_url:Uri.t -> token:string -> request +val authorization : string -> (string * string) list val same_origin : Uri.t -> Uri.t -> bool +val valid_artifact : Uri.t -> bool val artifact : base_url:Uri.t -> uri:Uri.t -> token:string option -> request val validate_base_url : Uri.t -> (unit, string) result diff --git a/logseq_sync/lib/effect_runner/eio/http_eio.ml b/logseq_sync/lib/native_transport/http_eio.ml similarity index 63% rename from logseq_sync/lib/effect_runner/eio/http_eio.ml rename to logseq_sync/lib/native_transport/http_eio.ml index 42692f32..4508b99c 100644 --- a/logseq_sync/lib/effect_runner/eio/http_eio.ml +++ b/logseq_sync/lib/native_transport/http_eio.ml @@ -1,13 +1,27 @@ +type progress = + { received_bytes : int + ; total_bytes : int option + ; datom_count : int option + } + type response = { status : int ; headers : (string * string) list ; body : string } -exception Transport_error of string +type error = + | Network of string + | Protocol of string + | Authentication of string + | Storage of string + | Crypto of string + | Invalid_input of string + +exception Transport_error of error exception Retire_attempt -let fail message = raise (Transport_error message) +let fail message = raise (Transport_error (Protocol message)) let port uri = Option.value (Uri.port uri) ~default:443 let successful_status status = status >= 200 && status < 300 let next_attempt = Atomic.make 0 @@ -20,10 +34,21 @@ let header name headers = ;; let exception_message = function - | Transport_error message -> message - | End_of_file -> "premature HTTP EOF" - | Eio.Time.Timeout -> "HTTP transport timed out" - | error -> Tls_client_eio.transport_failure ~operation:"httpTransport" error + | Transport_error cause -> cause + | End_of_file -> Protocol "premature HTTP EOF" + | Eio.Time.Timeout -> Network "HTTP transport timed out" + | Tls_eio.Tls_failure (`Error (`AuthenticationFailure _)) -> + Authentication "TLS authentication failed" + | Tls_eio.Tls_failure _ | Tls_eio.Tls_alert _ -> Crypto "TLS protocol failed" + | Sys_error _ -> Storage "HTTP artifact I/O failed" + | Unix.Unix_error ((Unix.ENOSPC | Unix.EROFS | Unix.EACCES | Unix.EPERM), _, _) -> + Storage "HTTP I/O unavailable" + | Invalid_argument _ -> Invalid_input "Invalid HTTP operation" + | error -> + (match Tls_client_eio.transient_failure error with + | Some _ -> + Network (Tls_client_eio.transport_failure ~operation:"httpTransport" error) + | None -> Protocol "HTTP transport failed") ;; let tls_stream_socket flow = @@ -49,7 +74,18 @@ let tls_stream_socket flow = Eio.Resource.T (flow, Eio.Net.Pi.stream_socket (module Socket)) ;; -let with_attempt ~sw ~authenticator ~network ~clock ~response_budget request ~consume = +let with_attempt + ~setup_seconds + ~tls_handshake_seconds + ~sw + ~authenticator + ~network + ~clock + ~response_budget + ~download_budget + request + ~consume + = Eio.Switch.check sw; let id = Atomic.fetch_and_add next_attempt 1 in let stage = ref "setup" @@ -100,11 +136,27 @@ let with_attempt ~sw ~authenticator ~network ~clock ~response_budget request ~co | Some host -> host | None -> fail "HTTPS host is missing" in - let dns_failure = ref None in + let setup_failure = ref None in + let on_failure exn = + let cause = exception_message exn in + match !setup_failure, cause with + | ( Some + ( Protocol _ + | Authentication _ + | Crypto _ + | Storage _ + | Invalid_input _ ) + , Network _ ) -> () + | _ -> setup_failure := Some cause + in let flow = match - Tls_client_eio.connect_diagnosed - ~on_diagnostic:(fun source -> dns_failure := Some source) + Tls_client_eio.connect_with_failure + ~setup_seconds + ~tls_handshake_seconds + ~on_failure + ~on_transient_failure:(fun _ -> ()) + ~on_diagnostic:(fun _ -> ()) ~sw:owner ~authenticator ~network @@ -116,28 +168,56 @@ let with_attempt ~sw ~authenticator ~network ~clock ~response_budget request ~co | Error Tls_client_eio.Invalid_dns_host -> fail "TLS setup failed: invalid DNS host" | Error Tls_client_eio.No_network_address -> - fail ("TLS setup failed: no network address" - ^ Option.fold ~none:"" ~some:(fun source -> " (" ^ source ^ ")") !dns_failure) + raise + (Transport_error + (Option.value + !setup_failure + ~default:(Network "DNS returned no network address"))) | Error Tls_client_eio.Authentication_failed -> - fail "TLS authentication failed" - | Error (Tls_client_eio.Setup_failed "invalid TLS configuration") -> - fail "TLS setup failed: invalid TLS configuration" - | Error (Tls_client_eio.Setup_failed "all resolved addresses failed") -> - fail "TLS setup failed: all resolved addresses failed" - | Error (Tls_client_eio.Setup_failed "DNS/TCP dial timed out") -> - fail "TLS setup failed: DNS/TCP dial timed out" - | Error (Tls_client_eio.Setup_failed source) -> - (match Tls_client_eio.safe_setup_detail source with - | Some detail -> fail ("TLS setup failed: " ^ detail) - | None -> fail "TLS connection setup failed") + raise (Transport_error (Authentication "TLS authentication failed")) + | Error (Tls_client_eio.Setup_failed _) -> + raise + (Transport_error + (Option.value + !setup_failure + ~default:(Protocol "Invalid TLS setup"))) + in + let started = Transport_time.now clock in + let headers_ready, mark_headers = Eio.Promise.create () in + let headers_at = ref None in + let remaining budget since = + max 0. (!budget -. (Transport_time.now clock -. since)) + in + let timed callback = + match download_budget with + | None -> Transport_time.with_timeout clock !response_budget callback + | Some budget -> + Eio.Fiber.first callback (fun () -> + Transport_time.with_timeout + clock + (remaining response_budget started) + (fun () -> Eio.Promise.await headers_ready); + let since = Option.get !headers_at in + Transport_time.with_timeout clock (remaining budget since) (fun () -> + Eio.Promise.await completed); + Eio.Fiber.await_cancel ()) in - let started = Eio.Time.now clock in Fun.protect ~finally:(fun () -> - response_budget - := max 0. (!response_budget -. (Eio.Time.now clock -. started))) + let finished = Transport_time.now clock in + let response_end = Option.value !headers_at ~default:finished in + let response_end = + if download_budget = None then finished else response_end + in + response_budget := max 0. (!response_budget -. (response_end -. started)); + Option.iter + (fun budget -> + Option.iter + (fun since -> budget := max 0. (!budget -. (finished -. since))) + !headers_at) + download_budget) (fun () -> - Eio.Time.with_timeout_exn clock !response_budget (fun () -> + timed (fun () -> let client = Httpun_eio.Client.create_connection ~sw:owner @@ -146,9 +226,9 @@ let with_attempt ~sw ~authenticator ~network ~clock ~response_budget request ~co let error_handler error = let message = match error with - | `Malformed_response _ -> "malformed HTTP response" + | `Malformed_response _ -> Protocol "malformed HTTP response" | `Invalid_response_body_length _ -> - "invalid HTTP response body length" + Protocol "invalid HTTP response body length" | `Exn exn -> exception_message exn in settle (Error message) @@ -160,6 +240,8 @@ let with_attempt ~sw ~authenticator ~network ~clock ~response_budget request ~co in let response_handler response body = guard (fun () -> + headers_at := Some (Transport_time.now clock); + ignore (Eio.Promise.try_resolve mark_headers () : bool); let status = Httpun.Status.to_code response.Httpun.Response.status in @@ -187,9 +269,7 @@ let with_attempt ~sw ~authenticator ~network ~clock ~response_budget request ~co :: request.headers in let headers = - Httpun.Headers.of_list - (("host", host_header) - :: request_headers) + Httpun.Headers.of_list (("host", host_header) :: request_headers) in let target = Uri.path_and_query request.uri in let descriptor = @@ -220,7 +300,9 @@ let with_attempt ~sw ~authenticator ~network ~clock ~response_budget request ~co | Retire_attempt -> ()); Eio.Fiber.check (); log "retired"; - Option.value !outcome ~default:(Error "HTTP driver retired without an outcome") + Option.value + !outcome + ~default:(Error (Protocol "HTTP driver retired without an outcome")) with | Eio.Cancel.Cancelled _ as cancelled -> log "cancelled-retired"; @@ -239,7 +321,7 @@ let read_body ~maximum_bytes ~received ~guard ~settle ~on_chunk ~on_eof body = guard (fun () -> if len > maximum_bytes - !received then ( - settle (Error "HTTP body exceeds its configured bound"); + settle (Error (Protocol "HTTP body exceeds its configured bound")); Httpun.Body.Reader.close body) else ( on_chunk (Bigstringaf.substring chunk ~off ~len); @@ -249,13 +331,25 @@ let read_body ~maximum_bytes ~received ~guard ~settle ~on_chunk ~on_eof body = read () ;; -let request_once ~sw ~authenticator ~network ~clock ~response_budget request = +let request_once + ~setup_seconds + ~tls_handshake_seconds + ~sw + ~authenticator + ~network + ~clock + ~response_budget + request + = with_attempt + ~setup_seconds + ~tls_handshake_seconds ~sw ~authenticator ~network ~clock ~response_budget + ~download_budget:None request ~consume:(fun ~status ~headers ~received ~guard ~settle body -> let buffer = Buffer.create 4096 in @@ -276,14 +370,14 @@ let redirect_status = function let redirect request headers = match header "location" headers with - | None -> Error "HTTP redirect is missing Location" + | None -> Error (Protocol "HTTP redirect is missing Location") | Some location -> let next = Uri.resolve "" request.Http.uri (Uri.of_string location) in if Uri.userinfo next <> None || Uri.fragment next <> None || not (Http.same_origin request.uri next) - then Error "HTTP redirect changed authority or contains unsafe credentials" + then Error (Protocol "HTTP redirect changed authority or contains unsafe credentials") else Ok { request with uri = next } ;; @@ -296,9 +390,9 @@ let follow_redirects attempt request = match request.Http.operation with | Put _ -> true | Get -> false - then Error "Asset uploads cannot redirect" + then Error (Protocol "Asset uploads cannot redirect") else if remaining = 0 - then Error "HTTP redirect limit exceeded" + then Error (Protocol "HTTP redirect limit exceeded") else Result.bind (redirect request response.headers) (follow (remaining - 1)) else Ok (request, response)) in @@ -308,34 +402,67 @@ let follow_redirects attempt request = let protected f = try f () with | Eio.Cancel.Cancelled _ as cancelled -> raise cancelled - | Eio.Time.Timeout -> Error "HTTP transport timed out" + | Eio.Time.Timeout -> Error (Network "HTTP transport timed out") | exception_ -> Error (exception_message exception_) ;; -let perform ~sw ~authenticator ~network ~clock request = +let perform_with_deadlines + ~setup_seconds + ~tls_handshake_seconds + ~http_response_seconds + ~sw + ~authenticator + ~network + ~clock + request + = Tls_client_eio.initialize_rng (); - let response_budget = ref 30. in + let response_budget = ref http_response_seconds in protected (fun () -> Result.bind (follow_redirects - (request_once ~sw ~authenticator ~network ~clock ~response_budget) + (request_once + ~setup_seconds + ~tls_handshake_seconds + ~sw + ~authenticator + ~network + ~clock + ~response_budget) request) (fun (request, response) -> if successful_status response.status then Result.map (fun () -> response) - (Http.validate_response_content_type request response.headers) + (Http.validate_response_content_type request response.headers + |> Result.map_error (fun _ -> Protocol "Invalid HTTP response Content-Type") + ) else Ok response)) ;; +let perform ~sw ~authenticator ~network ~clock request = + perform_with_deadlines + ~setup_seconds:30. + ~tls_handshake_seconds:10. + ~http_response_seconds:30. + ~sw + ~authenticator + ~network + ~clock + request +;; + let remove_partial path = try Sys.remove path with | Sys_error _ -> () ;; let download_once + ~setup_seconds + ~tls_handshake_seconds ~response_budget + ~download_budget ~sw ~authenticator ~network @@ -354,11 +481,14 @@ let download_once (fun () -> let result = with_attempt + ~setup_seconds + ~tls_handshake_seconds ~sw ~authenticator ~network ~clock ~response_budget + ~download_budget:(Some download_budget) request ~consume:(fun ~status ~headers ~received ~guard ~settle body -> if not (successful_status status) @@ -366,7 +496,7 @@ let download_once else ( (match Http.validate_response_content_type request headers with | Ok () -> () - | Error message -> fail message); + | Error _ -> fail "Invalid HTTP response Content-Type"); let total_bytes = Option.bind (header "content-length" headers) int_of_string_opt |> fun count -> @@ -386,8 +516,7 @@ let download_once partial := Some path; channel := Some output; let progress count = - on_progress - Bootstrap.{ received_bytes = count; total_bytes; datom_count = None } + on_progress { received_bytes = count; total_bytes; datom_count = None } in progress 0; read_body @@ -415,10 +544,14 @@ let download_once with | exn -> Error (exception_message exn)) | None, None -> Ok response - | _ -> Error "incomplete artifact output state")) + | _ -> Error (Protocol "incomplete artifact output state"))) ;; -let download +let download_with_deadlines + ~setup_seconds + ~tls_handshake_seconds + ~download_seconds + ~http_response_seconds ~sw ~authenticator ~network @@ -430,16 +563,20 @@ let download = Tls_client_eio.initialize_rng (); if maximum_bytes <= 0 - then Error "snapshot artifact bound must be positive" + then Error (Protocol "snapshot artifact bound must be positive") else ( - let response_budget = ref 120. in + let response_budget = ref http_response_seconds in + let download_budget = ref download_seconds in protected (fun () -> Result.map snd (follow_redirects (fun request -> download_once + ~setup_seconds + ~tls_handshake_seconds ~response_budget + ~download_budget ~sw ~authenticator ~network @@ -450,3 +587,28 @@ let download ~on_progress) request))) ;; + +let download + ~sw + ~authenticator + ~network + ~clock + ~request + ~destination + ~maximum_bytes + ~on_progress + = + download_with_deadlines + ~setup_seconds:30. + ~tls_handshake_seconds:10. + ~download_seconds:120. + ~http_response_seconds:30. + ~sw + ~authenticator + ~network + ~clock + ~request + ~destination + ~maximum_bytes + ~on_progress +;; diff --git a/logseq_sync/lib/native_transport/http_eio.mli b/logseq_sync/lib/native_transport/http_eio.mli new file mode 100644 index 00000000..deed6df9 --- /dev/null +++ b/logseq_sync/lib/native_transport/http_eio.mli @@ -0,0 +1,65 @@ +type progress = + { received_bytes : int + ; total_bytes : int option + ; datom_count : int option + } + +type response = + { status : int + ; headers : (string * string) list + ; body : string + } + +type error = + | Network of string + | Protocol of string + | Authentication of string + | Storage of string + | Crypto of string + | Invalid_input of string + +(** DNS/TCP, TLS and HTTP have separate budgets. Redirects share HTTP budgets. *) +val perform_with_deadlines + : setup_seconds:float + -> tls_handshake_seconds:float + -> http_response_seconds:float + -> sw:Eio.Switch.t + -> authenticator:X509.Authenticator.t + -> network:_ Eio.Net.t + -> clock:Transport_time.t + -> Http.request + -> (response, error) result + +val download_with_deadlines + : setup_seconds:float + -> tls_handshake_seconds:float + -> download_seconds:float + -> http_response_seconds:float + -> sw:Eio.Switch.t + -> authenticator:X509.Authenticator.t + -> network:_ Eio.Net.t + -> clock:Transport_time.t + -> request:Http.request + -> destination:string + -> maximum_bytes:int + -> on_progress:(progress -> unit) + -> (response, error) result + +val perform + : sw:Eio.Switch.t + -> authenticator:X509.Authenticator.t + -> network:_ Eio.Net.t + -> clock:Transport_time.t + -> Http.request + -> (response, error) result + +val download + : sw:Eio.Switch.t + -> authenticator:X509.Authenticator.t + -> network:_ Eio.Net.t + -> clock:Transport_time.t + -> request:Http.request + -> destination:string + -> maximum_bytes:int + -> on_progress:(progress -> unit) + -> (response, error) result diff --git a/logseq_sync/lib/native_transport/snapshot_artifact.ml b/logseq_sync/lib/native_transport/snapshot_artifact.ml new file mode 100644 index 00000000..6b2b6824 --- /dev/null +++ b/logseq_sync/lib/native_transport/snapshot_artifact.ml @@ -0,0 +1,177 @@ +module Failure_diagnostic = Logseq_db_types.Failure_diagnostic + +let failure_message ~reason:reason_name source = + Failure_diagnostic.( + of_source ~operation:"decodeSnapshot" ~reason:reason_name source |> message) +;; + +let artifact_row_count headers = + let value = + List.find_map + (fun (name, value) -> + if String.equal (String.lowercase_ascii name) "x-snapshot-row-count" + then Some value + else None) + headers + in + match Option.bind value int_of_string_opt with + | Some value when value > 0 -> Ok value + | _ -> Error "snapshot row-count header is invalid" +;; + +let remove_if_exists path = + try Sys.remove path with + | Sys_error _ as error -> + (* Missing artifacts are normal idempotent cleanup. Preserve the original + unit completion if an artifact remains, but retain a safe support cause. *) + if Sys.file_exists path + then ( + let detail = + Failure_diagnostic.of_source + ~operation:"decodeSnapshot" + ~reason:"cleanupFailed" + (Printexc.to_string error) + in + try + Printf.eprintf + "journal: phase=cleanup operation=removeSnapshotArtifact code=cleanupPending \ + source=%s\n\ + %!" + (Failure_diagnostic.source_class detail) + with + | _ -> ()) +;; + +let cleanup = List.iter remove_if_exists + +let gzip_signature path = + try + let channel = open_in_bin path in + Fun.protect + ~finally:(fun () -> close_in_noerr channel) + (fun () -> + try + let first = input_char channel in + let second = input_char channel in + Ok (first = '\x1f' && second = '\x8b') + with + | End_of_file -> Ok false) + with + | Sys_error _ as error -> + Error (failure_message ~reason:"resourceFailed" (Printexc.to_string error)) +;; + +let copy_file ~maximum_bytes source destination = + try + let input_channel = open_in_bin source in + Fun.protect + ~finally:(fun () -> close_in_noerr input_channel) + (fun () -> + let output_channel = + open_out_gen + [ Open_wronly; Open_creat; Open_excl; Open_binary ] + 0o600 + destination + in + Fun.protect + ~finally:(fun () -> close_out_noerr output_channel) + (fun () -> + let buffer = Bytes.create 16_384 in + let copied = ref 0 in + let rec loop () = + let count = input input_channel buffer 0 (Bytes.length buffer) in + if count > 0 + then ( + if !copied + count > maximum_bytes + then failwith "snapshot artifact exceeds its decompressed bound"; + output output_channel buffer 0 count; + copied := !copied + count; + loop ()) + in + loop ())); + Ok () + with + | Failure source when source = "snapshot artifact exceeds its decompressed bound" -> + remove_if_exists destination; + Error "snapshot artifact exceeds its decompressed bound" + | exception_ -> + remove_if_exists destination; + Error (failure_message ~reason:"resourceFailed" (Printexc.to_string exception_)) +;; + +let nonempty path = + try (Unix.stat path).st_size > 0 with + | Unix.Unix_error _ -> false +;; + +let decompress ~decompress_gzip ~maximum_bytes source destination = + remove_if_exists destination; + match decompress_gzip source destination maximum_bytes with + | 0 when nonempty destination -> Ok () + | 0 -> + remove_if_exists destination; + Error "snapshot gzip layer is empty" + | status -> + remove_if_exists destination; + Error + (match status with + | 1 -> "snapshot gzip input could not be opened" + | 2 -> "snapshot gzip output could not be opened" + | 3 -> "snapshot gzip decode failed" + | 4 -> "snapshot gzip output write failed" + | 5 -> "snapshot gzip decompressed limit exceeded" + | _ -> "snapshot gzip layer is invalid") +;; + +let peel_gzip_layers ~decompress_gzip ~maximum_bytes ~source ~destination ~temporary_paths + = + if maximum_bytes <= 0 + then Error "snapshot decompressed bound must be positive" + else ( + match temporary_paths with + | first :: second :: _ -> + cleanup [ destination; first; second ]; + let fail message = + cleanup [ destination; first; second ]; + Error message + in + (match gzip_signature source with + | Error message -> fail message + | Ok false -> copy_file ~maximum_bytes source destination + | Ok true -> + (match decompress ~decompress_gzip ~maximum_bytes source first with + | Error message -> fail message + | Ok () -> + (match gzip_signature first with + | Error message -> fail message + | Ok false -> + (try + Sys.rename first destination; + cleanup [ second ]; + Ok () + with + | Sys_error _ as error -> + fail + (failure_message + ~reason:"persistenceFailed" + (Printexc.to_string error))) + | Ok true -> + (match decompress ~decompress_gzip ~maximum_bytes first second with + | Error message -> fail message + | Ok () -> + (match gzip_signature second with + | Error message -> fail message + | Ok true -> fail "snapshot exceeds the supported gzip layer count" + | Ok false -> + (try + Sys.rename second destination; + cleanup [ first ]; + Ok () + with + | Sys_error _ as error -> + fail + (failure_message + ~reason:"persistenceFailed" + (Printexc.to_string error)))))))) + | [] | [ _ ] -> Error "two private gzip staging paths are required") +;; diff --git a/logseq_sync/lib/native_transport/snapshot_artifact.mli b/logseq_sync/lib/native_transport/snapshot_artifact.mli new file mode 100644 index 00000000..efb36007 --- /dev/null +++ b/logseq_sync/lib/native_transport/snapshot_artifact.mli @@ -0,0 +1,9 @@ +val artifact_row_count : (string * string) list -> (int, string) result + +val peel_gzip_layers + : decompress_gzip:(string -> string -> int -> int) + -> maximum_bytes:int + -> source:string + -> destination:string + -> temporary_paths:string list + -> (unit, string) result diff --git a/logseq_sync/lib/native_transport/snapshot_checksum.ml b/logseq_sync/lib/native_transport/snapshot_checksum.ml new file mode 100644 index 00000000..e385f150 --- /dev/null +++ b/logseq_sync/lib/native_transport/snapshot_checksum.ml @@ -0,0 +1,6 @@ +type verifier = + check_current:(unit -> unit) + -> scratch_bytes:int64 + -> encrypted:bool + -> path:string + -> (Logseq_overlay_db.Types.Checksum.t, string) result diff --git a/logseq_sync/lib/native_transport/snapshot_checksum.mli b/logseq_sync/lib/native_transport/snapshot_checksum.mli new file mode 100644 index 00000000..e385f150 --- /dev/null +++ b/logseq_sync/lib/native_transport/snapshot_checksum.mli @@ -0,0 +1,6 @@ +type verifier = + check_current:(unit -> unit) + -> scratch_bytes:int64 + -> encrypted:bool + -> path:string + -> (Logseq_overlay_db.Types.Checksum.t, string) result diff --git a/logseq_sync/lib/native_transport/snapshot_parser.ml b/logseq_sync/lib/native_transport/snapshot_parser.ml new file mode 100644 index 00000000..1b7c5200 --- /dev/null +++ b/logseq_sync/lib/native_transport/snapshot_parser.ml @@ -0,0 +1,163 @@ +module Value = Transit_core.Json +module Codec = Transit_native.Transit.Json + +type row = + { addr : int + ; content : string + ; addresses : string option + } + +type parser = + { max_frame_bytes : int + ; mutable buffer : string + } + +type import = + { expected_rows : int + ; mutable accepted_rows : int + ; mutable last_addr : int option + ; mutable has_root : bool + ; mutable has_tail : bool + } + +type completed_import = { row_count : int } + +let bind result f = + match result with + | Ok value -> f value + | Error _ as error -> error +;; + +let create_parser ~max_frame_bytes = + if max_frame_bytes <= 0 then invalid_arg "max_frame_bytes must be positive"; + { max_frame_bytes; buffer = "" } +;; + +let uint32_be value offset = + let byte index = Char.code value.[offset + index] in + Int64.( + logor + (shift_left (of_int (byte 0)) 24) + (logor + (shift_left (of_int (byte 1)) 16) + (logor (shift_left (of_int (byte 2)) 8) (of_int (byte 3))))) +;; + +let int_of_value = function + | Value.Int value when value >= 0 -> Ok value + | Value.Int64 value when value >= 0L && value <= Int64.of_int max_int -> + Ok (Int64.to_int value) + | _ -> Error "snapshot row address must be a non-negative Transit integer" +;; + +let optional_string = function + | Value.Null -> Ok None + | Value.String value -> Ok (Some value) + | _ -> Error "snapshot row addresses must be JSON text or nil" +;; + +let decode_row = function + | Value.Array [ addr; Value.String content; addresses ] -> + bind (int_of_value addr) (fun addr -> + bind (optional_string addresses) (fun addresses -> Ok { addr; content; addresses })) + | _ -> Error "snapshot row must be [addr, content, addresses]" +;; + +let decode_rows payload = + let rec decode_all decoded = function + | [] -> Ok (List.rev decoded) + | value :: rest -> + bind (decode_row value) (fun row -> decode_all (row :: decoded) rest) + in + try + match Codec.of_string payload with + | Value.Array values -> decode_all [] values + | _ -> Error "snapshot frame payload must be a Transit row array" + with + | Value.Decode_error message + | Yojson.Json_error message + | Failure message + | Invalid_argument message -> Error message +;; + +let feed parser chunk = + parser.buffer <- parser.buffer ^ chunk; + let rec consume offset rows = + let remaining = String.length parser.buffer - offset in + if remaining < 4 + then Ok (offset, List.rev rows) + else ( + let length64 = uint32_be parser.buffer offset in + if length64 > Int64.of_int parser.max_frame_bytes + then Error "snapshot frame exceeds configured size limit" + else ( + let length = Int64.to_int length64 in + if remaining - 4 < length + then Ok (offset, List.rev rows) + else ( + let payload = String.sub parser.buffer (offset + 4) length in + bind (decode_rows payload) (fun decoded -> + consume (offset + 4 + length) (List.rev_append decoded rows))))) + in + bind (consume 0 []) (fun (consumed, rows) -> + if consumed > 0 + then + parser.buffer + <- String.sub parser.buffer consumed (String.length parser.buffer - consumed); + Ok rows) +;; + +let finish parser = + if String.length parser.buffer = 0 + then Ok () + else Error "incomplete framed snapshot stream" +;; + +let create_import ~expected_rows = + { expected_rows + ; accepted_rows = 0 + ; last_addr = None + ; has_root = false + ; has_tail = false + } +;; + +let accept_rows state rows = + let rec validate last_addr has_root has_tail count = function + | [] -> Ok (last_addr, has_root, has_tail, count) + | row :: rest -> + (match last_addr with + | Some previous when row.addr <= previous -> + Error "snapshot row addresses must be strictly increasing" + | None | Some _ -> + validate + (Some row.addr) + (has_root || row.addr = 0) + (has_tail || row.addr = 1) + (count + 1) + rest) + in + bind + (validate state.last_addr state.has_root state.has_tail state.accepted_rows rows) + (fun (last_addr, has_root, has_tail, accepted_rows) -> + if accepted_rows > state.expected_rows + then Error "snapshot contains more rows than advertised" + else ( + state.last_addr <- last_addr; + state.has_root <- has_root; + state.has_tail <- has_tail; + state.accepted_rows <- accepted_rows; + Ok ())) +;; + +let finish_import state = + if state.expected_rows < 0 + then Error "snapshot expected row count must be non-negative" + else if state.accepted_rows <> state.expected_rows + then Error "snapshot row count does not match server metadata" + else if not state.has_root + then Error "snapshot is missing DataScript root row 0" + else if not state.has_tail + then Error "snapshot is missing DataScript tail row 1" + else Ok { row_count = state.accepted_rows } +;; diff --git a/logseq_sync/lib/native_transport/snapshot_parser.mli b/logseq_sync/lib/native_transport/snapshot_parser.mli new file mode 100644 index 00000000..72f0e299 --- /dev/null +++ b/logseq_sync/lib/native_transport/snapshot_parser.mli @@ -0,0 +1,16 @@ +type row = + { addr : int + ; content : string + ; addresses : string option + } + +type parser +type import +type completed_import = { row_count : int } + +val create_parser : max_frame_bytes:int -> parser +val feed : parser -> string -> (row list, string) result +val finish : parser -> (unit, string) result +val create_import : expected_rows:int -> import +val accept_rows : import -> row list -> (unit, string) result +val finish_import : import -> (completed_import, string) result diff --git a/logseq_sync/lib/native_transport/snapshot_verify.ml b/logseq_sync/lib/native_transport/snapshot_verify.ml new file mode 100644 index 00000000..160dfd8c --- /dev/null +++ b/logseq_sync/lib/native_transport/snapshot_verify.ml @@ -0,0 +1,138 @@ +let rec remove_tree path = + match Unix.lstat path with + | { st_kind = Unix.S_DIR; _ } -> + Sys.readdir path |> Array.iter (fun name -> remove_tree (Filename.concat path name)); + Unix.rmdir path + | _ -> Unix.unlink path + | exception Unix.Unix_error (Unix.ENOENT, _, _) -> () +;; + +let import_snapshot_rows + ~check_current + ~maximum_frame_bytes + ~scratch_bytes + ~snapshot_path + ~expected_rows + database_path + = + let sqlite = Sqlite3.db_open database_path in + let closed = ref false in + let result = + Fun.protect + ~finally:(fun () -> + ignore (Sqlite3.exec sqlite "ROLLBACK"); + closed := Sqlite3.db_close sqlite) + (fun () -> + try + if scratch_bytes < 8192L + then Error "Snapshot scratch bound" + else ( + Sqlite3.Rc.check (Sqlite3.exec sqlite "PRAGMA page_size=4096"); + Sqlite3.Rc.check + (Sqlite3.exec + sqlite + ("PRAGMA max_page_count=" + ^ Int64.to_string (Int64.div scratch_bytes 4096L))); + Sqlite3.Rc.check (Sqlite3.exec sqlite "PRAGMA journal_mode=DELETE"); + Sqlite3.Rc.check + (Sqlite3.exec + sqlite + "CREATE TABLE kvs (addr INTEGER PRIMARY KEY, content TEXT, addresses \ + JSON)"); + Sqlite3.Rc.check (Sqlite3.exec sqlite "BEGIN IMMEDIATE"); + let statement = + Sqlite3.prepare + sqlite + "INSERT INTO kvs(addr, content, addresses) VALUES(?, ?, ?)" + in + let imported = + Fun.protect + ~finally:(fun () -> ignore (Sqlite3.finalize statement)) + (fun () -> + let channel = open_in_bin snapshot_path in + Fun.protect + ~finally:(fun () -> close_in_noerr channel) + (fun () -> + let parser = + Snapshot_parser.create_parser + ~max_frame_bytes:maximum_frame_bytes + in + let import = Snapshot_parser.create_import ~expected_rows in + let buffer = Bytes.create 65_536 in + let insert (row : Snapshot_parser.row) = + check_current (); + Sqlite3.Rc.check (Sqlite3.reset statement); + Sqlite3.Rc.check (Sqlite3.bind_int statement 1 row.addr); + Sqlite3.Rc.check (Sqlite3.bind_text statement 2 row.content); + Sqlite3.Rc.check + (Sqlite3.bind + statement + 3 + (match row.addresses with + | None -> Sqlite3.Data.NULL + | Some addresses -> TEXT addresses)); + match Sqlite3.step statement with + | Sqlite3.Rc.DONE -> () + | _ -> failwith "Snapshot storage write failed" + in + let rec read () = + check_current (); + match input channel buffer 0 (Bytes.length buffer) with + | 0 -> + Result.bind (Snapshot_parser.finish parser) (fun () -> + Result.map + (fun _ -> ()) + (Snapshot_parser.finish_import import)) + | count -> + Result.bind + (Snapshot_parser.feed + parser + (Bytes.sub_string buffer 0 count)) + (fun rows -> + Result.bind + (Snapshot_parser.accept_rows import rows) + (fun () -> + List.iter insert rows; + read ())) + in + read ())) + in + Result.bind imported (fun () -> + Sqlite3.Rc.check (Sqlite3.exec sqlite "COMMIT"); + if Int64.of_int (Unix.stat database_path).st_size > scratch_bytes + then Error "Snapshot scratch bound" + else Ok ())) + with + | Eio.Cancel.Cancelled _ as cancelled -> raise cancelled + | _ -> Error "Snapshot framed storage import failed") + in + if !closed then result else Error "Snapshot verification SQLite close failed" +;; + +let inspect + ~verify + ~check_current + ~scratch_bytes + ~maximum_frame_bytes + ~directory + ~encrypted + ~path + ~expected_rows + = + let scratch = Filename.concat directory "verify.sqlite" in + Fun.protect + ~finally:(fun () -> + List.iter + (fun suffix -> remove_tree (scratch ^ suffix)) + [ ""; "-journal"; "-wal"; "-shm" ]) + (fun () -> + Result.bind + (import_snapshot_rows + ~check_current + ~scratch_bytes + ~maximum_frame_bytes + ~snapshot_path:path + ~expected_rows + scratch) + (fun () -> verify ~check_current ~scratch_bytes ~encrypted ~path:scratch)) +;; diff --git a/logseq_sync/lib/native_transport/snapshot_verify.mli b/logseq_sync/lib/native_transport/snapshot_verify.mli new file mode 100644 index 00000000..15af0d26 --- /dev/null +++ b/logseq_sync/lib/native_transport/snapshot_verify.mli @@ -0,0 +1,12 @@ +val remove_tree : string -> unit + +val inspect + : verify:Snapshot_checksum.verifier + -> check_current:(unit -> unit) + -> scratch_bytes:int64 + -> maximum_frame_bytes:int + -> directory:string + -> encrypted:bool + -> path:string + -> expected_rows:int + -> (Logseq_overlay_db.Types.Checksum.t, string) result diff --git a/logseq_sync/lib/effect_runner/eio/tls_client_eio.ml b/logseq_sync/lib/native_transport/tls_client_eio.ml similarity index 55% rename from logseq_sync/lib/effect_runner/eio/tls_client_eio.ml rename to logseq_sync/lib/native_transport/tls_client_eio.ml index 0928ff15..bc997a9a 100644 --- a/logseq_sync/lib/effect_runner/eio/tls_client_eio.ml +++ b/logseq_sync/lib/native_transport/tls_client_eio.ml @@ -4,16 +4,25 @@ type error = | Setup_failed of string | Authentication_failed -type transient_failure = Connection_refused | Connection_timeout | Network_unreachable | Connection_reset +type transient_failure = + | Connection_refused + | Connection_timeout + | Network_unreachable + | Connection_reset let transient_failure = function | Unix.Unix_error (Unix.ECONNREFUSED, _, _) - | Eio.Io (Eio.Net.E (Eio.Net.Connection_failure (Eio.Net.Refused _)), _) -> Some Connection_refused - | Eio.Time.Timeout | Unix.Unix_error (Unix.ETIMEDOUT, _, _) - | Eio.Io (Eio.Net.E (Eio.Net.Connection_failure Eio.Net.Timeout), _) -> Some Connection_timeout - | Unix.Unix_error ((Unix.ENETUNREACH | Unix.EHOSTUNREACH), _, _) -> Some Network_unreachable + | Eio.Io (Eio.Net.E (Eio.Net.Connection_failure (Eio.Net.Refused _)), _) -> + Some Connection_refused + | Eio.Time.Timeout + | Unix.Unix_error (Unix.ETIMEDOUT, _, _) + | Eio.Io (Eio.Net.E (Eio.Net.Connection_failure Eio.Net.Timeout), _) -> + Some Connection_timeout + | Unix.Unix_error ((Unix.ENETUNREACH | Unix.EHOSTUNREACH), _, _) -> + Some Network_unreachable | Unix.Unix_error ((Unix.ECONNRESET | Unix.EPIPE), _, _) - | Eio.Io (Eio.Net.E (Eio.Net.Connection_reset _), _) | End_of_file -> Some Connection_reset + | Eio.Io (Eio.Net.E (Eio.Net.Connection_reset _), _) + | End_of_file -> Some Connection_reset | _ -> None ;; @@ -41,56 +50,87 @@ let host_name host = | Ok host -> Ok host) ;; -let resolve ~clock ~deadline host port = +let resolve ~on_failure ~clock ~deadline host port = let family ipv6 = let query = dns_start host ipv6 in Fun.protect ~finally:(fun () -> dns_close query) (fun () -> try - let addresses = Eio.Time.with_timeout_exn - clock - (max 0. (deadline -. Eio.Time.now clock)) - (fun () -> - let rec receive () = - Eio_unix.await_readable (dns_fd query); - if dns_process query then dns_results query else receive () - in - receive ()) in + let addresses = + Transport_time.with_timeout + clock + (max 0. (deadline -. Transport_time.now clock)) + (fun () -> + let rec receive () = + Eio_unix.await_readable (dns_fd query); + if dns_process query then dns_results query else receive () + in + receive ()) + in addresses, None with | Eio.Cancel.Cancelled _ as cancelled -> raise cancelled - | Eio.Time.Timeout -> [], Some ("DNS resolution timed out", Some Connection_timeout) - | Unix.Unix_error ((Unix.EACCES | Unix.EPERM), _, _) -> + | Eio.Time.Timeout -> + on_failure Eio.Time.Timeout; + [], Some ("DNS resolution timed out", Some Connection_timeout) + | Unix.Unix_error ((Unix.EACCES | Unix.EPERM), _, _) as error -> + on_failure error; [], Some ("DNS resolver permission denied", None) - | Unix.Unix_error _ -> [], Some ("DNS resolver I/O failed", None) - | _ -> [], Some ("DNS resolver operation failed", None)) + | Unix.Unix_error _ as error -> + on_failure error; + [], Some ("DNS resolver I/O failed", None) + | error -> + on_failure error; + [], Some ("DNS resolver operation failed", None)) + in + let (ipv6, ipv6_error), (ipv4, ipv4_error) = + Eio.Fiber.pair (fun () -> family true) (fun () -> family false) in - let (ipv6, ipv6_error), (ipv4, ipv4_error) = Eio.Fiber.pair (fun () -> family true) (fun () -> family false) in - let addresses = List.sort_uniq String.compare (ipv6 @ ipv4) - |> List.map (fun address -> `Tcp (Eio.Net.Ipaddr.of_raw address, port)) in - let transient = match ipv4_error, ipv6_error with + let addresses = + List.sort_uniq String.compare (ipv6 @ ipv4) + |> List.map (fun address -> `Tcp (Eio.Net.Ipaddr.of_raw address, port)) + in + let transient = + match ipv4_error, ipv6_error with | Some (_, Some a), Some (_, Some _) -> Some a - | _ -> None in - addresses, Option.map fst (match ipv4_error with Some _ -> ipv4_error | None -> ipv6_error), transient + | _ -> None + in + ( addresses + , Option.map + fst + (match ipv4_error with + | Some _ -> ipv4_error + | None -> ipv6_error) + , transient ) ;; (* Eio may allocate a socket before connect raises. Keep each candidate in its own child owner, and report failure only after that owner has released it. A successful candidate remains owned until the operation switch retires. *) -let connect_address ~sw ~network ~clock ~deadline ~tcp_seconds config peer_name address = +let connect_address + ~sw + ~network + ~clock + ~deadline + ~tcp_seconds + ~tls_handshake_seconds + config + peer_name + address + = let result, resolve = Eio.Promise.create () in Eio.Fiber.fork ~sw (fun () -> try Eio.Switch.run (fun address_owner -> let connected = - Eio.Time.with_timeout_exn + Transport_time.with_timeout clock - (max 0. (min tcp_seconds (deadline -. Eio.Time.now clock))) + (max 0. (min tcp_seconds (deadline -. Transport_time.now clock))) (fun () -> Eio.Net.connect ~sw:address_owner network address) in let flow = - Eio.Time.with_timeout_exn clock 10. (fun () -> + Transport_time.with_timeout clock tls_handshake_seconds (fun () -> Tls_eio.client_of_flow config ~host:peer_name connected) in ignore (Eio.Promise.try_resolve resolve (Ok flow) : bool); @@ -110,7 +150,8 @@ let setup_class = function | Eio.Time.Timeout | Unix.Unix_error (Unix.ETIMEDOUT, _, _) | Eio.Io (Eio.Net.E (Eio.Net.Connection_failure Eio.Net.Timeout), _) -> "dial timed out" - | Unix.Unix_error ((Unix.ENETUNREACH | Unix.EHOSTUNREACH), _, _) -> "network unreachable" + | Unix.Unix_error ((Unix.ENETUNREACH | Unix.EHOSTUNREACH), _, _) -> + "network unreachable" | Unix.Unix_error ((Unix.EACCES | Unix.EPERM), _, _) -> "permission denied" | Unix.Unix_error ((Unix.ECONNRESET | Unix.EPIPE), _, _) | Eio.Io (Eio.Net.E (Eio.Net.Connection_reset _), _) -> "connection reset" @@ -122,15 +163,41 @@ let setup_class = function ;; let safe_setup_detail source = - let prefixes = ["all resolved addresses failed: "; "TLS connection setup failed: "] in - let classes = ["connection refused"; "dial timed out"; "network unreachable"; - "permission denied"; "connection reset"; "TLS peer alert"; "TLS handshake failed"; - "operation raised Failure"; "operation raised Invalid_argument"; "external setup failure"] in - if List.exists (fun prefix -> List.exists (fun value -> source = prefix ^ value) classes) - prefixes then Some source else None + let prefixes = [ "all resolved addresses failed: "; "TLS connection setup failed: " ] in + let classes = + [ "connection refused" + ; "dial timed out" + ; "network unreachable" + ; "permission denied" + ; "connection reset" + ; "TLS peer alert" + ; "TLS handshake failed" + ; "operation raised Failure" + ; "operation raised Invalid_argument" + ; "external setup failure" + ] + in + if + List.exists + (fun prefix -> List.exists (fun value -> source = prefix ^ value) classes) + prefixes + then Some source + else None ;; -let connect_diagnosed_with_attention ~on_transient_failure ~on_diagnostic ~sw ~authenticator ~network ~clock ~host ~port = +let connect_with_failure + ~on_failure + ~setup_seconds + ~tls_handshake_seconds + ~on_transient_failure + ~on_diagnostic + ~sw + ~authenticator + ~network + ~clock + ~host + ~port + = match host_name host with | Error _ as error -> error | Ok peer_name -> @@ -139,19 +206,28 @@ let connect_diagnosed_with_attention ~on_transient_failure ~on_diagnostic ~sw ~a with | Error (`Msg _) -> Error (Setup_failed "invalid TLS configuration") | Ok config -> - let deadline = Eio.Time.now clock +. 30. in + let deadline = Transport_time.now clock +. setup_seconds in (try - let addresses, dns_failure, dns_transient = resolve ~clock ~deadline host port in - let report transient = Option.iter (fun cause -> - try on_transient_failure cause with _ -> ()) transient in + let addresses, dns_failure, dns_transient = + resolve ~on_failure ~clock ~deadline host port + in + let report transient = + Option.iter + (fun cause -> + try on_transient_failure cause with + | _ -> ()) + transient + in let rec attempt last_failure transient attention_required = function | [] -> if not attention_required then report transient; Error (Setup_failed ("all resolved addresses failed: " ^ last_failure)) | address :: rest -> - let remaining = deadline -. Eio.Time.now clock in + let remaining = deadline -. Transport_time.now clock in if remaining <= 0. - then (if not attention_required then report (Some Connection_timeout); + then ( + on_failure Eio.Time.Timeout; + if not attention_required then report (Some Connection_timeout); Error (Setup_failed "DNS/TCP dial timed out")) else ( match @@ -160,6 +236,7 @@ let connect_diagnosed_with_attention ~on_transient_failure ~on_diagnostic ~sw ~a ~network ~clock ~deadline + ~tls_handshake_seconds ~tcp_seconds:(remaining /. float_of_int (1 + List.length rest)) config peer_name @@ -170,28 +247,85 @@ let connect_diagnosed_with_attention ~on_transient_failure ~on_diagnostic ~sw ~a | Error (Tls_eio.Tls_failure (`Error (`AuthenticationFailure _))) -> Error Authentication_failed | Error error -> + on_failure error; let transient = transient_failure error in - attempt (setup_class error) transient (attention_required || transient = None) rest) + attempt + (setup_class error) + transient + (attention_required || transient = None) + rest) in - if addresses = [] then ( - Option.iter (fun cause -> try on_diagnostic cause with _ -> ()) dns_failure; + if addresses = [] + then ( + Option.iter + (fun cause -> + try on_diagnostic cause with + | _ -> ()) + dns_failure; report dns_transient; Error No_network_address) else attempt "external setup failure" None false addresses with | Eio.Cancel.Cancelled _ as cancelled -> raise cancelled | error -> - Option.iter (fun cause -> try on_transient_failure cause with _ -> ()) (transient_failure error); + on_failure error; + Option.iter + (fun cause -> + try on_transient_failure cause with + | _ -> ()) + (transient_failure error); Error (Setup_failed ("TLS connection setup failed: " ^ setup_class error)))) ;; +let connect_diagnosed_with_attention + ~setup_seconds + ~tls_handshake_seconds + ~on_transient_failure + ~on_diagnostic + ~sw + ~authenticator + ~network + ~clock + ~host + ~port + = + connect_with_failure + ~on_failure:(fun _ -> ()) + ~setup_seconds + ~tls_handshake_seconds + ~on_transient_failure + ~on_diagnostic + ~sw + ~authenticator + ~network + ~clock + ~host + ~port +;; + let connect_diagnosed ~on_diagnostic ~sw ~authenticator ~network ~clock ~host ~port = - connect_diagnosed_with_attention ~on_transient_failure:(fun _ -> ()) ~on_diagnostic - ~sw ~authenticator ~network ~clock ~host ~port + connect_diagnosed_with_attention + ~setup_seconds:30. + ~tls_handshake_seconds:10. + ~on_transient_failure:(fun _ -> ()) + ~on_diagnostic + ~sw + ~authenticator + ~network + ~clock + ~host + ~port ;; let connect ~sw ~authenticator ~network ~clock ~host ~port = - connect_diagnosed ~on_diagnostic:(fun _ -> ()) ~sw ~authenticator ~network ~clock ~host ~port + connect_diagnosed + ~on_diagnostic:(fun _ -> ()) + ~sw + ~authenticator + ~network + ~clock + ~host + ~port ;; (* Only finite TLS alert constructors and classified exception identities cross @@ -199,8 +333,10 @@ let connect ~sw ~authenticator ~network ~clock ~host ~port = let transport_failure ~operation:operation_name error = let alert_name = function | Tls.Packet.UNKNOWN _ -> "unknown" - | alert -> Tls.Packet.alert_type_to_string alert in - let source = match error with + | alert -> Tls.Packet.alert_type_to_string alert + in + let source = + match error with | Tls_eio.Tls_alert alert -> "TLS alert: " ^ alert_name alert | Tls_eio.Tls_failure failure -> let _, alert = Tls.Engine.alert_of_failure failure in @@ -211,7 +347,8 @@ let transport_failure ~operation:operation_name error = "Unix.Unix_error(Unix.ETIMEDOUT,,)" | Eio.Io (Eio.Net.E (Eio.Net.Connection_reset _), _) -> "Unix.Unix_error(Unix.ECONNRESET,,)" - | error -> Printexc.to_string error in + | error -> Printexc.to_string error + in Logseq_db_types.Failure_diagnostic.( of_source ~operation:operation_name ~reason:"networkFailed" source |> encode) ;; diff --git a/logseq_sync/lib/effect_runner/eio/tls_client_eio.mli b/logseq_sync/lib/native_transport/tls_client_eio.mli similarity index 66% rename from logseq_sync/lib/effect_runner/eio/tls_client_eio.mli rename to logseq_sync/lib/native_transport/tls_client_eio.mli index 168782d2..9dfe4434 100644 --- a/logseq_sync/lib/effect_runner/eio/tls_client_eio.mli +++ b/logseq_sync/lib/native_transport/tls_client_eio.mli @@ -4,17 +4,20 @@ type error = | Setup_failed of string | Authentication_failed -type transient_failure = Connection_refused | Connection_timeout | Network_unreachable | Connection_reset +type transient_failure = + | Connection_refused + | Connection_timeout + | Network_unreachable + | Connection_reset val transient_failure : exn -> transient_failure option - val initialize_rng : unit -> unit val connect : sw:Eio.Switch.t -> authenticator:X509.Authenticator.t -> network:_ Eio.Net.t - -> clock:_ Eio.Time.clock + -> clock:Transport_time.t -> host:string -> port:int -> (Tls_eio.t, error) result @@ -28,7 +31,7 @@ val connect_diagnosed -> sw:Eio.Switch.t -> authenticator:X509.Authenticator.t -> network:_ Eio.Net.t - -> clock:_ Eio.Time.clock + -> clock:Transport_time.t -> host:string -> port:int -> (Tls_eio.t, error) result @@ -36,13 +39,29 @@ val connect_diagnosed (** Presentation metadata only. Report a finite transient cause only after every failed address is known transient. Unknown, permission and TLS/security failures remain attention-worthy; transport results and retry policy do not change. *) +val connect_with_failure + : on_failure:(exn -> unit) + -> setup_seconds:float + -> tls_handshake_seconds:float + -> on_transient_failure:(transient_failure -> unit) + -> on_diagnostic:(string -> unit) + -> sw:Eio.Switch.t + -> authenticator:X509.Authenticator.t + -> network:_ Eio.Net.t + -> clock:Transport_time.t + -> host:string + -> port:int + -> (Tls_eio.t, error) result + val connect_diagnosed_with_attention - : on_transient_failure:(transient_failure -> unit) + : setup_seconds:float + -> tls_handshake_seconds:float + -> on_transient_failure:(transient_failure -> unit) -> on_diagnostic:(string -> unit) -> sw:Eio.Switch.t -> authenticator:X509.Authenticator.t -> network:_ Eio.Net.t - -> clock:_ Eio.Time.clock + -> clock:Transport_time.t -> host:string -> port:int -> (Tls_eio.t, error) result diff --git a/logseq_sync/lib/native_transport/transport_domain.ml b/logseq_sync/lib/native_transport/transport_domain.ml new file mode 100644 index 00000000..058d4103 --- /dev/null +++ b/logseq_sync/lib/native_transport/transport_domain.ml @@ -0,0 +1,12 @@ +type error_attention = + | Transient_connection + | Requires_attention + +type websocket_termination = + | Recoverable_close of + { message : string option + ; attention : error_attention + } + | Local_close + | Authentication_failure of string + | Protocol_failure of string diff --git a/logseq_sync/lib/native_transport/transport_domain.mli b/logseq_sync/lib/native_transport/transport_domain.mli new file mode 100644 index 00000000..058d4103 --- /dev/null +++ b/logseq_sync/lib/native_transport/transport_domain.mli @@ -0,0 +1,12 @@ +type error_attention = + | Transient_connection + | Requires_attention + +type websocket_termination = + | Recoverable_close of + { message : string option + ; attention : error_attention + } + | Local_close + | Authentication_failure of string + | Protocol_failure of string diff --git a/logseq_sync/lib/native_transport/transport_time.ml b/logseq_sync/lib/native_transport/transport_time.ml new file mode 100644 index 00000000..be154815 --- /dev/null +++ b/logseq_sync/lib/native_transport/transport_time.ml @@ -0,0 +1,25 @@ +type t = + { now : unit -> float + ; sleep : float -> unit + ; timeout : 'a. float -> (unit -> 'a) -> 'a + } + +let monotonic clock = + { now = (fun () -> Int64.to_float (Mtime.to_uint64_ns (Eio.Time.Mono.now clock)) /. 1e9) + ; sleep = Eio.Time.Mono.sleep clock + ; timeout = + (fun seconds callback -> + Eio.Time.Timeout.run_exn (Eio.Time.Timeout.seconds clock seconds) callback) + } +;; + +let realtime clock = + { now = (fun () -> Eio.Time.now clock) + ; sleep = Eio.Time.sleep clock + ; timeout = (fun seconds callback -> Eio.Time.with_timeout_exn clock seconds callback) + } +;; + +let now clock = clock.now () +let sleep clock seconds = clock.sleep seconds +let with_timeout clock seconds callback = clock.timeout seconds callback diff --git a/logseq_sync/lib/native_transport/transport_time.mli b/logseq_sync/lib/native_transport/transport_time.mli new file mode 100644 index 00000000..250d3d86 --- /dev/null +++ b/logseq_sync/lib/native_transport/transport_time.mli @@ -0,0 +1,7 @@ +type t + +val monotonic : _ Eio.Time.Mono.t -> t +val realtime : _ Eio.Time.clock -> t +val now : t -> float +val sleep : t -> float -> unit +val with_timeout : t -> float -> (unit -> 'a) -> 'a diff --git a/logseq_sync/lib/effect_runner/eio/websocket_eio.ml b/logseq_sync/lib/native_transport/websocket_eio.ml similarity index 82% rename from logseq_sync/lib/effect_runner/eio/websocket_eio.ml rename to logseq_sync/lib/native_transport/websocket_eio.ml index 730f4c8c..ce8c9723 100644 --- a/logseq_sync/lib/effect_runner/eio/websocket_eio.ml +++ b/logseq_sync/lib/native_transport/websocket_eio.ml @@ -1,9 +1,12 @@ -module Core = Logseq_sync_pure_reducer.Core +module Core = Transport_domain type connect_error = | Unauthorized | Forbidden - | Connection_failed of { message : string; attention : Logseq_sync_pure_reducer.Core.error_attention } + | Connection_failed of + { message : string + ; attention : Transport_domain.error_attention + } | Protocol_failed of string | Authentication_failed of string @@ -56,17 +59,21 @@ let close_timeout = 1. let protocol_error message = raise (Protocol_error message) let termination_message = function - | Core.Recoverable_close { message = reason; _ } -> Option.value reason ~default:"WebSocket closed" + | Core.Recoverable_close { message = reason; _ } -> + Option.value reason ~default:"WebSocket closed" | Core.Local_close -> "WebSocket closed locally" | Core.Authentication_failure message | Core.Protocol_failure message -> message ;; -let connection_failed ?(attention = Core.Requires_attention) message = Connection_failed { message; attention } +let connection_failed ?(attention = Core.Requires_attention) message = + Connection_failed { message; attention } +;; let connection_error = function | Core.Authentication_failure message -> Authentication_failed message | Core.Protocol_failure message -> Protocol_failed message - | Core.Recoverable_close { message; attention } -> connection_failed ~attention (Option.value message ~default:"WebSocket closed") + | Core.Recoverable_close { message; attention } -> + connection_failed ~attention (Option.value message ~default:"WebSocket closed") | reason -> connection_failed (termination_message reason) ;; @@ -78,13 +85,17 @@ let peer_termination payload = let message = Printf.sprintf "WebSocket peer closed with status %d" status in match status with | 1002 | 1003 | 1007 | 1008 | 1009 | 1010 -> Core.Protocol_failure message - | _ -> Core.Recoverable_close { message = Some message; attention = Requires_attention }) + | _ -> + Core.Recoverable_close { message = Some message; attention = Requires_attention }) ;; let closed_reason t = if t.local_closing then Core.Local_close - else Option.value t.peer_termination ~default:(Core.Recoverable_close { message = None; attention = Requires_attention }) + else + Option.value + t.peer_termination + ~default:(Core.Recoverable_close { message = None; attention = Requires_attention }) ;; let stop t reason = @@ -147,7 +158,8 @@ let close t = request_close t "\x03\xe8" ;; -let failure_attention t = match t.terminal with +let failure_attention t = + match t.terminal with | Some (Core.Recoverable_close { attention; _ }) -> attention | _ -> Core.Requires_attention ;; @@ -245,7 +257,9 @@ let input_handlers t maximum = (match error with | Some _ -> Core.Protocol_failure "WebSocket protocol failed" | None when t.peer_closed && t.close_sent -> closed_reason t - | None -> Core.Recoverable_close { message = Some "WebSocket peer EOF"; attention = Transient_connection }) + | None -> + Core.Recoverable_close + { message = Some "WebSocket peer EOF"; attention = Transient_connection }) in { Httpun_ws.Websocket_connection.frame; eof } ;; @@ -269,7 +283,13 @@ let read_loop t ~maximum_frame_bytes flow protocol = done; let before = t.read_progress in match Client.next_read_operation protocol with - | `Close -> stop t (Core.Recoverable_close { message = Some "WebSocket reader closed"; attention = Transient_connection }) + | `Close -> + stop + t + (Core.Recoverable_close + { message = Some "WebSocket reader closed" + ; attention = Transient_connection + }) | `Yield -> let resumed, resume = Eio.Promise.create () in Client.yield_reader protocol (fun () -> @@ -301,7 +321,11 @@ let read_loop t ~maximum_frame_bytes flow protocol = t (if t.peer_closed && t.close_sent then closed_reason t - else Core.Recoverable_close { message = Some "WebSocket peer EOF"; attention = Transient_connection }))) + else + Core.Recoverable_close + { message = Some "WebSocket peer EOF" + ; attention = Transient_connection + }))) in loop 0 false ;; @@ -379,7 +403,13 @@ let write_loop t flow protocol = | `Close _ -> drained (); if not t.closing - then stop t (Core.Recoverable_close { message = Some "WebSocket writer closed"; attention = Transient_connection })) + then + stop + t + (Core.Recoverable_close + { message = Some "WebSocket writer closed" + ; attention = Transient_connection + })) and wait () = let resumed, resume = Eio.Promise.create () in let wake () = ignore (Eio.Promise.try_resolve resume () : bool) in @@ -413,7 +443,7 @@ let heartbeat t clock = function | Ping_pong { interval_seconds; timeout_seconds } -> let sequence = ref 0L in while (not t.closing) && t.terminal = None do - Eio.Time.sleep clock interval_seconds; + Transport_time.sleep clock interval_seconds; if (not t.closing) && t.terminal = None then ( sequence := Int64.succ !sequence; @@ -424,12 +454,18 @@ let heartbeat t clock = function t.pong <- Some (value, resolve); enqueue_control t 9 value; (try - Eio.Time.with_timeout_exn clock timeout_seconds (fun () -> + Transport_time.with_timeout clock timeout_seconds (fun () -> Eio.Promise.await pong) with | Eio.Time.Timeout -> if not t.closing - then stop t (Core.Recoverable_close { message = Some "WebSocket Pong deadline expired"; attention = Transient_connection })); + then + stop + t + (Core.Recoverable_close + { message = Some "WebSocket Pong deadline expired" + ; attention = Transient_connection + })); t.pong <- None) done ;; @@ -445,12 +481,24 @@ let failure_reason = function | Protocol_error message -> Core.Protocol_failure message | Tls_eio.Tls_failure (`Error (`AuthenticationFailure _)) -> Core.Authentication_failure "WebSocket TLS authentication failed" - | exn -> Core.Recoverable_close { message = Some (failure_message exn); attention = - (if (match exn with End_of_file -> true | _ -> false) || Option.is_some (Tls_client_eio.transient_failure exn) - then Transient_connection else Requires_attention) } + | exn -> + Core.Recoverable_close + { message = Some (failure_message exn) + ; attention = + (if + (match exn with + | End_of_file -> true + | _ -> false) + || Option.is_some (Tls_client_eio.transient_failure exn) + then Transient_connection + else Requires_attention) + } ;; let connect + ~setup_seconds + ~tls_handshake_seconds + ~http_response_seconds ~sw ~authenticator ~network @@ -470,6 +518,24 @@ let connect in if (not valid_uri) + || (not + (List.for_all + (fun value -> Float.is_finite value && value > 0.) + [ setup_seconds; tls_handshake_seconds; http_response_seconds ])) + || (match liveness with + | Disabled -> false + | Ping_pong { interval_seconds; timeout_seconds } -> + not + (Float.is_finite interval_seconds + && interval_seconds > 0. + && Float.is_finite timeout_seconds + && timeout_seconds > 0.)) + || String.length token = 0 + || String.length token > 1024 * 1024 + || (not + (String.for_all + (fun character -> Char.code character > 32 && Char.code character < 127) + token)) || maximum_frame_bytes <= 0 || maximum_frame_bytes > maximum_queue_bytes then Error (connection_failed "invalid WebSocket connection parameters") @@ -520,7 +586,10 @@ let connect let tls_attention = ref Core.Requires_attention in match Tls_client_eio.connect_diagnosed_with_attention - ~on_transient_failure:(fun _ -> tls_attention := Core.Transient_connection) + ~setup_seconds + ~tls_handshake_seconds + ~on_transient_failure:(fun _ -> + tls_attention := Core.Transient_connection) ~on_diagnostic:(fun source -> dns_failure := Some source) ~sw:owner ~authenticator @@ -532,27 +601,43 @@ let connect | Error Tls_client_eio.Authentication_failed -> Error (Authentication_failed "WebSocket TLS authentication failed") | Error Tls_client_eio.Invalid_dns_host -> - Error (connection_failed ~attention:!tls_attention "WebSocket TLS setup failed: invalid DNS host") + Error + (connection_failed + ~attention:!tls_attention + "WebSocket TLS setup failed: invalid DNS host") | Error Tls_client_eio.No_network_address -> - Error (connection_failed ~attention:!tls_attention ("WebSocket TLS setup failed: no network address" - ^ Option.fold ~none:"" ~some:(fun source -> " (" ^ source ^ ")") !dns_failure)) + Error + (connection_failed + ~attention:!tls_attention + ("WebSocket TLS setup failed: no network address" + ^ Option.fold + ~none:"" + ~some:(fun source -> " (" ^ source ^ ")") + !dns_failure)) | Error (Tls_client_eio.Setup_failed "invalid TLS configuration") -> Error - (connection_failed ~attention:!tls_attention + (connection_failed + ~attention:!tls_attention "WebSocket TLS setup failed: invalid TLS configuration") | Error (Tls_client_eio.Setup_failed "all resolved addresses failed") -> Error - (connection_failed ~attention:!tls_attention + (connection_failed + ~attention:!tls_attention "WebSocket TLS setup failed: all resolved addresses failed") | Error (Tls_client_eio.Setup_failed "DNS/TCP dial timed out") -> Error - (connection_failed ~attention:!tls_attention "WebSocket TLS setup failed: DNS/TCP dial timed out") + (connection_failed + ~attention:!tls_attention + "WebSocket TLS setup failed: DNS/TCP dial timed out") | Error (Tls_client_eio.Setup_failed source) -> - Error (connection_failed ~attention:!tls_attention (match Tls_client_eio.safe_setup_detail source with - | Some detail -> "WebSocket TLS setup failed: " ^ detail - | None -> "WebSocket TLS setup failed")) + Error + (connection_failed + ~attention:!tls_attention + (match Tls_client_eio.safe_setup_detail source with + | Some detail -> "WebSocket TLS setup failed: " ^ detail + | None -> "WebSocket TLS setup failed")) | Ok flow -> - Eio.Time.with_timeout_exn clock 30. (fun () -> + Transport_time.with_timeout clock http_response_seconds (fun () -> let opened, resolve_opened = Eio.Promise.create () in let websocket_handler descriptor = t.descriptor <- Some descriptor; @@ -621,8 +706,13 @@ let connect ~sw:owner (guarded (fun () -> Eio.Promise.await close_requested; - Eio.Time.sleep clock close_timeout; - stop t (Core.Recoverable_close { message = Some "WebSocket close deadline expired"; attention = Transient_connection }))); + Transport_time.sleep clock close_timeout; + stop + t + (Core.Recoverable_close + { message = Some "WebSocket close deadline expired" + ; attention = Transient_connection + }))); (* Setup and both pumps share this owner from before DNS through final retirement, including a failure before a handle can be returned. *) let setup_result = @@ -640,7 +730,8 @@ let connect | Forbidden -> Core.Authentication_failure "Authorization failed." | Authentication_failed message -> Core.Authentication_failure message | Protocol_failed message -> Core.Protocol_failure message - | Connection_failed { message; attention } -> Core.Recoverable_close { message = Some message; attention }) + | Connection_failed { message; attention } -> + Core.Recoverable_close { message = Some message; attention }) | Ok () -> ignore (Eio.Promise.try_resolve resolve_ready (Ok t) : bool)); Eio.Promise.await stopped; raise Stop_owner) @@ -658,7 +749,11 @@ let connect let reason = Option.value t.terminal - ~default:(Core.Recoverable_close { message = Some "WebSocket owner stopped"; attention = Requires_attention }) + ~default: + (Core.Recoverable_close + { message = Some "WebSocket owner stopped" + ; attention = Requires_attention + }) in ignore (Eio.Promise.try_resolve resolve_ready (Error (connection_error reason)) : bool); diff --git a/logseq_sync/lib/effect_runner/eio/websocket_eio.mli b/logseq_sync/lib/native_transport/websocket_eio.mli similarity index 70% rename from logseq_sync/lib/effect_runner/eio/websocket_eio.mli rename to logseq_sync/lib/native_transport/websocket_eio.mli index 91df5278..fc3d6936 100644 --- a/logseq_sync/lib/effect_runner/eio/websocket_eio.mli +++ b/logseq_sync/lib/native_transport/websocket_eio.mli @@ -3,7 +3,10 @@ type t type connect_error = | Unauthorized | Forbidden - | Connection_failed of { message : string; attention : Logseq_sync_pure_reducer.Core.error_attention } + | Connection_failed of + { message : string + ; attention : Transport_domain.error_attention + } | Protocol_failed of string | Authentication_failed of string @@ -15,26 +18,28 @@ type liveness = } val connect - : sw:Eio.Switch.t + : setup_seconds:float + -> tls_handshake_seconds:float + -> http_response_seconds:float + -> sw:Eio.Switch.t -> authenticator:X509.Authenticator.t -> network:_ Eio.Net.t - -> clock:_ Eio.Time.clock + -> clock:Transport_time.t -> liveness:liveness -> uri:Uri.t -> token:string -> maximum_frame_bytes:int -> on_message:(string -> unit) - -> on_close:(Logseq_sync_pure_reducer.Core.websocket_termination -> unit) + -> on_close:(Transport_domain.websocket_termination -> unit) -> (t, connect_error) result (** Register and publish open inside [on_open] before enabling message delivery. A terminated pending connection fails activation without invoking [on_open]. *) val activate : t -> on_open:(unit -> unit) -> (unit, connect_error) result -(** Success means local queue admission. Driver failures use the terminal callback. *) -(** Presentation classification of an already observed terminal cause. *) -val failure_attention : t -> Logseq_sync_pure_reducer.Core.error_attention +val failure_attention : t -> Transport_domain.error_attention +(** Success means local queue admission, not delivery. *) val send : t -> string -> (unit, string) result (** Initiate a bounded close exchange without waiting for the owner. *) diff --git a/logseq_sync/lib/protocol/dune b/logseq_sync/lib/protocol/dune new file mode 100644 index 00000000..4c29e453 --- /dev/null +++ b/logseq_sync/lib/protocol/dune @@ -0,0 +1,8 @@ +(library + (name logseq_sync_protocol_impl) + (public_name logseq_sync.protocol.impl) + (implements logseq_sync_protocol) + (modules sync_protocol) + (libraries logseq_db_types ppx_deriving_yojson.runtime yojson) + (preprocess + (pps ppx_deriving_yojson))) diff --git a/logseq_sync/lib/pure_reducer/sync_protocol.ml b/logseq_sync/lib/protocol/sync_protocol.ml similarity index 99% rename from logseq_sync/lib/pure_reducer/sync_protocol.ml rename to logseq_sync/lib/protocol/sync_protocol.ml index 7d9ed4a9..0ca5ffa5 100644 --- a/logseq_sync/lib/pure_reducer/sync_protocol.ml +++ b/logseq_sync/lib/protocol/sync_protocol.ml @@ -577,8 +577,12 @@ let validate_rejection context (rejection : rejection) = match rejection.reason with | Stale | Snapshot_upload_in_progress -> Option.is_some rejection.t && operational_fields_absent - | Empty_tx_data | Invalid_tx | Invalid_t_before -> + | Empty_tx_data | Invalid_t_before -> Option.is_none rejection.t && operational_fields_absent + | Invalid_tx -> + (Option.is_none rejection.t && operational_fields_absent) + || (Option.is_some rejection.t && Option.is_some rejection.failed_tx_id + && rejection.missing_block_uuids = []) | Db_transact_failed -> Option.is_some rejection.t in if valid diff --git a/logseq_sync/lib/pure_reducer/core.ml b/logseq_sync/lib/pure_reducer/core.ml deleted file mode 100644 index ebfd6f08..00000000 --- a/logseq_sync/lib/pure_reducer/core.ml +++ /dev/null @@ -1,4706 +0,0 @@ -module Overlay = Logseq_overlay_db.Types -module Server_cursor = Overlay.Server_cursor - -type graph_id = Logseq_db_types.Graph_types.Uuid.t [@@deriving yojson] - -type cache_schema = Logseq_db_types.Managed_graph.schema = - { major : int - ; minor : int - ; exact : bool - } -[@@deriving yojson] - -type graph = Logseq_db_types.Managed_graph.t = - { graph_id : graph_id - ; name : string - ; schema : cache_schema - ; encrypted : bool - } -[@@deriving yojson] - -type account_generation = int -type graph_generation = int -type connection_generation = int -type presentation_generation = int -type lifecycle_generation = int64 - -type account_scope = - { managed_sync_origin : Uri.t - ; user_id : string - ; account_generation : account_generation - ; presentation_generation : presentation_generation - ; lifecycle_generation : lifecycle_generation - } - -type graph_scope = - { account : account_scope - ; graph_id : graph_id - ; graph_generation : graph_generation - } - -(** Execution admission gate owned by a pending host handoff. It grants no - deletion authority. Release is exact-ID and cannot unfreeze a later round. *) -module Host_cleanup_scopes=Set.Make(struct type t=graph_scope let compare=Stdlib.compare end) -type host_fence_target=One_graph of graph_id|All_graphs -type host_fence={id:Logseq_db_types.Graph_types.Uuid.t;owner:account_scope; - target:host_fence_target;captured_scopes:graph_scope Rrbvec.t;previous:Logseq_db_types.Graph_types.Uuid.t option} - -type connection_scope = - { graph : graph_scope - ; connection_generation : connection_generation - } - -type sync_phase = - | Offline - | Connecting - | Pulling - | Submitting - | Current - | Paused - | Failed - -type startup_failure_stage = - | During_authentication - | During_catalog - | During_local_restore - | During_bootstrap - | During_e2ee - -type local_unavailability = - | No_local_account - | Local_catalog_unavailable - | Local_mirror_unavailable - | Local_key_unavailable - | Local_restore_failed - -type local_access = - | Local_pending - | Local_available - | Local_unavailable of local_unavailability - -type startup_network_phase = - | Local_only - | Online_after_timeline - | Online_by_user - -type startup_facts = - { authenticated : bool - ; local_access : local_access - ; network_phase : startup_network_phase - ; catalog_loading : bool - ; awaiting_selection : bool - ; restoring_local : bool - ; bootstrapping : bool - ; awaiting_e2ee_password : bool - ; failure : startup_failure_stage option - ; account_generation : account_generation - ; graph_generation : graph_generation - ; presentation_generation : presentation_generation - } - -type local_deletion_stage = - | Closing_graph - | Deleting_mirror - | Clearing_selection - -type local_deletion = - | Deletion_in_progress of local_deletion_stage - | Deletion_failed of local_deletion_stage - -type unlock_challenge = - { challenge_scope : graph_scope - ; challenge_id : int - ; encrypted_graph_key : string - ; private_key_package : string - } - -type unlock_stage = Private_key of int | Graph_key of int | Attachment - -type unlock_submission = - { challenge : unlock_challenge - ; operation_id : int - ; stage : unlock_stage - } - -type unlock_attached = { submission : unlock_submission; attached_scope : graph_scope } - -type unlock_recovery = - | Retry_password of unlock_challenge * string - | Retry_preparation of string - -type unlock_phase = - | Unlock_inactive - | Unlock_challenge of unlock_challenge - | Unlock_running of unlock_submission - | Unlock_bootstrapping of unlock_submission - | Unlock_presentable of unlock_attached - | Unlock_failed of unlock_recovery - | Unlock_cancelled of unlock_submission - -let unlock_challenge = function - | Unlock_challenge challenge | Unlock_failed (Retry_password (challenge, _)) -> Some challenge - | _ -> None -;; - -let equal_unlock_challenge a b = - a.challenge_id = b.challenge_id && a.challenge_scope = b.challenge_scope -let unlock_challenge_scope challenge = challenge.challenge_scope -let unlock_submission_challenge submission = submission.challenge -let equal_unlock_submission a b = - a.operation_id = b.operation_id && equal_unlock_challenge a.challenge b.challenge -;; -let unlock_attached_submission receipt = receipt.submission -let unlock_attached_scope receipt = receipt.attached_scope - -(** Presentation only: retry eligibility is still determined by the original outcome. - Unknown and security failures require attention even when transport retries them. *) -type error_attention = Transient_connection | Requires_attention - -type snapshot = - { sync_phase : sync_phase - ; catalog : graph list - ; selected_graph : graph_id option - ; resident_graphs : graph_id list - ; applied_server_t : Server_cursor.t option - ; timeline_presentation_pending : bool - ; unlock : unlock_phase - ; startup : startup_facts - ; last_error : string option - ; last_error_attention : error_attention option - ; local_deletion : local_deletion option - } - -type diagnostic_group = - { title : string - ; entries : (string * string) list - } - -type diagnostics = { groups : diagnostic_group list } - -type state = - { snapshot : snapshot - ; diagnostics : diagnostics - } - -type limits = - { maximum_response_bytes : int - ; maximum_artifact_bytes : int - ; submission_batch_size : int - } -[@@warning "-69"] - -type config = - { managed_sync_origin : Uri.t - ; limits : limits - } - -type config_error = Invalid_config of string - -let limits ~maximum_response_bytes ~maximum_artifact_bytes ~submission_batch_size = - if - maximum_response_bytes <= 0 - || maximum_response_bytes > Logseq_db_types.Limits.maximum_response_bytes - then Error (Invalid_config "maximum response bytes are outside the supported bound") - else if maximum_artifact_bytes <= 0 - then Error (Invalid_config "maximum artifact bytes must be positive") - else if submission_batch_size <= 0 || submission_batch_size > 4096 - then Error (Invalid_config "submission batch size is outside the supported bound") - else Ok { maximum_response_bytes; maximum_artifact_bytes; submission_batch_size } -;; - -let config ~managed_sync_origin ~limits = - match Uri.scheme managed_sync_origin, Uri.host managed_sync_origin with - | Some "https", Some host when String.length host > 0 -> - Ok { managed_sync_origin; limits } - | _ -> Error (Invalid_config "managed sync origin must be an absolute HTTPS URI") -;; - -type bootstrap_progress = - { graph_id : graph_id - ; received_bytes : int64 - ; total_bytes : int64 option - } - -type invalidation = - { basis : int64 - ; changed_uuids : graph_id list - ; changed_uuids_truncated : bool - } - -type effect_scope = - { account_generation : account_generation option - ; graph_generation : graph_generation option - ; connection_generation : connection_generation option - ; presentation_generation : presentation_generation option - ; lifecycle_generation : lifecycle_generation option - } - -type effect_id = int - -type crypto_failure_kind = - | Invalid_key_material - | Crypto_provider_unavailable - -type effect_error = - | Effect_failed of string - | Crypto_failed of crypto_failure_kind * string - -type graph_key_handle = - { handle_id : string - ; handle_scope : graph_scope - } - -type snapshot_download = - { scope : graph_scope - ; uri : Uri.t - ; expected_bytes : int64 option - ; maximum_bytes : int - ; metadata_cursor : Server_cursor.t - ; attempt_id : effect_id - ; download_id : effect_id - } - -type staged_artifact = - { download : snapshot_download - ; artifact_path : string - ; artifact_expected_rows : int - } - -type catalog_cache = - { cache_user_id : string - ; cache_graphs : graph list - ; cache_selected_graph : graph_id option - ; cache_resident_graphs : graph_id list option [@default None] - } -[@@deriving yojson] - -let graph_key_handle ~id ~scope = { handle_id = id; handle_scope = scope } -let graph_key_handle_id handle = handle.handle_id -let graph_key_handle_scope handle = handle.handle_scope - -let stage_snapshot_download download ~path ~expected_rows = - if path = "" || expected_rows < 0 then Error "Invalid staged snapshot artifact" - else Ok { download; artifact_path = path; artifact_expected_rows = expected_rows } -;; - -let snapshot_download_scope request = request.scope -let snapshot_download_uri request = request.uri -let snapshot_download_expected_bytes request = request.expected_bytes -let snapshot_download_maximum_bytes request = request.maximum_bytes - -let staged_artifact_path artifact = artifact.artifact_path -let staged_artifact_expected_rows artifact = artifact.artifact_expected_rows - -let effect_scope_of_account (account : account_scope) = - { account_generation = Some account.account_generation - ; graph_generation = None - ; connection_generation = None - ; presentation_generation = Some account.presentation_generation - ; lifecycle_generation = Some account.lifecycle_generation - } -;; - -let effect_scope_of_graph (graph : graph_scope) = - { (effect_scope_of_account graph.account) with - graph_generation = Some graph.graph_generation - } -;; - -let effect_scope_of_connection (connection : connection_scope) = - { (effect_scope_of_graph connection.graph) with - connection_generation = Some connection.connection_generation - } -;; - -let catalog_cache ~user_id ~graphs ~selected_graph = - { cache_user_id = user_id - ; cache_graphs = graphs - ; cache_selected_graph = selected_graph - ; cache_resident_graphs = None - } -;; - -let catalog_cache_user_id cache = cache.cache_user_id -let catalog_cache_graphs cache = cache.cache_graphs -let catalog_cache_selected_graph cache = cache.cache_selected_graph -let catalog_cache_resident_graphs cache = - Option.value cache.cache_resident_graphs ~default:(Option.to_list cache.cache_selected_graph) -let encode_catalog_cache cache = catalog_cache_to_yojson cache |> Yojson.Safe.to_string - -let decode_catalog_cache source = - let bounded_text value = - String.length value > 0 - && String.length value <= 512 - && String.is_valid_utf_8 value - && not (String.contains value '\000') - in - try - match - Logseq_db_types.Json_codec.decode - catalog_cache_of_yojson - (Yojson.Safe.from_string source) - with - | Error message -> - let fields = List.concat_map (fun (prefix, names) -> - List.map (fun name -> prefix ^ name, - "field " ^ name ^ " is missing or has an invalid type") names) - [ "Core.catalog_cache.", ["cache_user_id"; "cache_graphs"; "cache_selected_graph"; "cache_resident_graphs"] - ; "Core.graph.", ["graph_id"; "name"; "schema"; "encrypted"] - ; "Core.cache_schema.", ["major"; "minor"; "exact"] ] in - Error ("catalog cache: " ^ Logseq_db_types.Json_codec.error_context ~fields message) - | Ok cache -> - if - (not (bounded_text cache.cache_user_id)) - || (match cache.cache_resident_graphs with - | None -> false - | Some ids -> List.length ids > 3 - || List.length (List.sort_uniq compare ids) <> List.length ids - || List.exists (fun id -> not (List.exists (fun (graph:graph) -> graph.graph_id=id) cache.cache_graphs)) ids - || (match cache.cache_selected_graph with - | Some id -> not (List.mem id ids) - | None -> ids <> [])) - || List.exists - (fun (graph : graph) -> - (not (bounded_text graph.name)) - || graph.schema.major < 0 - || graph.schema.minor < 0) - cache.cache_graphs - then Error "catalog cache contains invalid domain values" - else Ok cache - with - | Yojson.Json_error _ -> Error "catalog cache: invalid JSON syntax" - | Failure _ | Invalid_argument _ -> Error "catalog cache: invalid shape" -;; - -type encryption_batch = - { scope : graph_scope - ; key : graph_key_handle - ; plaintexts : string list - } - -type encrypted_values = (string * string) list - -type decryption_batch = - { scope : graph_scope - ; key : graph_key_handle - ; protected_values : (string * string) list - } - -type decrypted_values = string list -type snapshot_baseline = string -type snapshot_metadata = string - -type graph_key_request = - { scope : graph_scope - ; encrypted_graph_key : string - } - -type private_key_unlock = - { scope : account_scope - ; password : string - ; private_key_package : string - } - -type retirement_id = Logseq_db_types.Graph_types.Retirement_id.t -let retirement_id_of_string = Logseq_db_types.Graph_types.Retirement_id.of_string -let retirement_id_to_string = Logseq_db_types.Graph_types.Retirement_id.to_string - -type asset_deletion = { deletion_scope : graph_scope; retirement : retirement_id } -let asset_deletion_reference ~scope ~retirement = { deletion_scope = scope; retirement } -let asset_deletion_scope deletion = deletion.deletion_scope -let asset_deletion_retirement deletion = deletion.retirement - -type asset_context = - { scope : graph_scope - ; encrypted : bool - ; key : graph_key_handle option - } - -type asset_action = - | Check_asset_cache of graph_id * Logseq_db_types.Asset_descriptor.version - | Fetch_asset of - { asset : graph_id - ; version : Logseq_db_types.Asset_descriptor.version - ; maximum_plaintext_bytes : int - } - | Stage_asset_file of - { operation : graph_id - ; file_type : string - ; source_file : string - } - | Put_asset_file of - { asset : graph_id - ; version : Logseq_db_types.Asset_descriptor.version - ; file : string - ; maximum_plaintext_bytes : int - } - | Publish_staged_asset of - { asset : Logseq_db_types.Graph_types.Uuid.t - ; version : Logseq_db_types.Asset_descriptor.version - ; file : string - } - | Retain_asset_file of string - | Retain_staged_file of string - | Release_asset_file of string - | Release_staged_file of string - | Prune_asset_staging of string list - (** Join already admitted actual asset I/O tasks and their callback posting. - This read-only barrier neither closes caches nor grants a host permit. *) - | Await_asset_operations - | Close_asset_scope - | Prepare_graph_asset_deletion of retirement_id - | Commit_graph_asset_deletion of asset_deletion - | Cancel_graph_asset_deletion of asset_deletion - | Delete_account_assets - | Asset_retry_after of float - | Cancel_asset_operation of string - -type asset_failure = - | Asset_network - | Asset_not_found - | Asset_checksum_mismatch - | Asset_authentication - | Asset_locked - | Asset_storage_full - | Asset_missing_source - | Asset_size_rejected - | Asset_revoked_access - | Asset_cancelled - | Asset_invalid_content of string - | Asset_with_cause of asset_failure * Logseq_db_types.Failure_diagnostic.t - -let rec asset_failure_kind = function - | Asset_with_cause (failure, _) -> asset_failure_kind failure - | failure -> failure -;; - -let asset_failure_cause = function - | Asset_with_cause (_, cause) -> Some cause - | _ -> None -;; - -let asset_failure_with_cause failure cause = - Asset_with_cause (asset_failure_kind failure, cause) -;; - -type asset_value = - | Asset_unit - | Asset_cached of string option - | Asset_downloaded of string - | Asset_staged of - { file : string - ; checksum : string - ; size : int64 - } - | Asset_retained of (string * string) option - | Asset_deletion_prepared of asset_deletion - | Asset_pruned of int - | Asset_retry_elapsed - -type asset_result = (asset_value, asset_failure) result - -type asset_request = - { scope : graph_scope - ; operation : string - ; action : asset_action - } - -type asset_io_request = - { context : asset_context - ; operation : string - ; action : asset_action - } - -type asset_ticket = - { asset_serial : int - ; asset_request : asset_io_request - } - -let asset_ticket_id ticket = "asset:" ^ string_of_int ticket.asset_serial - -type asset_output = - { request : asset_request - ; result : asset_result - } - -type protected_action = - | Encrypt_values of graph_key_handle * string list - | Decrypt_value of graph_key_handle * string - -type protected_request = - { protected_operation : string - ; protected_scope : graph_scope - ; protected_action : protected_action - } - -type protected_value = - | Encrypted_values of (string * string) list - | Decrypted_value of string - -type protected_result = (protected_value, effect_error) result - -type protected_output = - { protected_request : protected_request - ; protected_result : protected_result - } - -type protected_ticket = - { protected_serial : int - ; protected_input : protected_request - } - -let protected_ticket_id ticket = "protected:" ^ string_of_int ticket.protected_serial - -type _ runner_request = - | Load_catalog : account_scope -> catalog_cache option runner_request - | Save_catalog : - { account : account_scope - ; cache : catalog_cache - } - -> unit runner_request - | Fetch_catalog : account_scope -> graph list runner_request - | Fetch_snapshot_baseline : graph_scope -> snapshot_baseline runner_request - | Fetch_snapshot_metadata : graph_scope -> snapshot_metadata runner_request - | Download_snapshot : snapshot_download -> staged_artifact runner_request - | Fetch_e2ee_graph_key : graph_scope -> string runner_request - | Fetch_e2ee_user_keys : account_scope -> string runner_request - | Load_and_unlock_graph_key : graph_scope -> graph_key_handle runner_request - | Fetch_and_unlock_graph_key : graph_key_request -> graph_key_handle runner_request - | Unlock_private_key : private_key_unlock -> unit runner_request - | Delete_account_secrets : account_scope -> unit runner_request - | Encrypt_protected_values : encryption_batch -> encrypted_values runner_request - | Decrypt_protected_values : decryption_batch -> decrypted_values runner_request - -type _ request_kind = - | Load_catalog_kind : catalog_cache option request_kind - | Save_catalog_kind : unit request_kind - | Fetch_catalog_kind : graph list request_kind - | Fetch_snapshot_baseline_kind : snapshot_baseline request_kind - | Fetch_snapshot_metadata_kind : snapshot_metadata request_kind - | Download_snapshot_kind : staged_artifact request_kind - | Fetch_e2ee_graph_key_kind : string request_kind - | Fetch_e2ee_user_keys_kind : string request_kind - | Load_and_unlock_graph_key_kind : graph_key_handle request_kind - | Fetch_and_unlock_graph_key_kind : graph_key_handle request_kind - | Unlock_private_key_kind : unit request_kind - | Delete_account_secrets_kind : unit request_kind - | Encrypt_protected_values_kind : encrypted_values request_kind - | Decrypt_protected_values_kind : decrypted_values request_kind - -type 'a effect_ticket = - { id : effect_id - ; scope : effect_scope - ; kind : 'a request_kind - } - -let effect_ticket_id ticket = ticket.id -let effect_ticket_scope ticket = ticket.scope -let effect_id_to_string = string_of_int - -type websocket_request = - { scope : connection_scope - ; uri : Uri.t - } - -type websocket_send = - { scope : connection_scope - ; message : Sync_protocol.Client.message - } - -type websocket_termination = - | Recoverable_close of { message : string option; attention : error_attention } - | Local_close - | Authentication_failure of string - | Protocol_failure of string - -type timer_id = int - -type timer_request = - { id : timer_id - ; scope : effect_scope - ; delay_seconds : float - } - -type runner_effect = - (* Snapshot actual I/O owners before fork; wait off the serial consumer and - report after their physical task and callback posting finish. No host, - filesystem or data-admission permission is granted. *) - | Await_io of int64 * effect_scope - | Asset_io of asset_ticket * asset_io_request - | Protected_io of protected_ticket * protected_request - | Request : 'a effect_ticket * 'a runner_request -> runner_effect - | Start_websocket of websocket_request - | Send_websocket of websocket_send - | Close_websocket of connection_scope - | Schedule_timer of timer_request - | Cancel_effects of effect_scope - | Cancel_network_effects of effect_scope - -type runner_completion = - | Asset_completion of asset_ticket * asset_result - | Protected_completion of protected_ticket * protected_result - | Completion : 'a effect_ticket * ('a, effect_error) result -> runner_completion - -let scope_of_request : type a. a runner_request -> effect_scope = function - | Load_catalog account | Save_catalog { account; _ } -> effect_scope_of_account account - | Fetch_catalog value -> effect_scope_of_account value - | Fetch_snapshot_baseline value | Fetch_snapshot_metadata value -> - effect_scope_of_graph value - | Download_snapshot value -> effect_scope_of_graph value.scope - | Fetch_e2ee_graph_key value -> effect_scope_of_graph value - | Fetch_e2ee_user_keys value -> effect_scope_of_account value - | Load_and_unlock_graph_key value -> effect_scope_of_graph value - | Fetch_and_unlock_graph_key value -> effect_scope_of_graph value.scope - | Unlock_private_key value -> effect_scope_of_account value.scope - | Delete_account_secrets value -> effect_scope_of_account value - | Encrypt_protected_values value -> effect_scope_of_graph value.scope - | Decrypt_protected_values value -> effect_scope_of_graph value.scope -;; - -let runner_effect_scope = function - | Await_io(_,scope)->scope - | Protected_io (_, request) -> effect_scope_of_graph request.protected_scope - | Asset_io (_, request) -> effect_scope_of_graph request.context.scope - | Request (ticket, _) -> ticket.scope - | Start_websocket value -> effect_scope_of_connection value.scope - | Send_websocket value -> effect_scope_of_connection value.scope - | Close_websocket value -> effect_scope_of_connection value - | Schedule_timer value -> value.scope - | Cancel_effects value | Cancel_network_effects value -> value -;; - -let runner_effect_diagnostic = function - | Await_io(id,_)->Printf.sprintf "io-fence:%Ld" id - | Protected_io (ticket, _) -> protected_ticket_id ticket - | Asset_io (ticket, _) -> asset_ticket_id ticket - | Request (ticket, _) -> Printf.sprintf "request:%d" ticket.id - | Start_websocket _ -> "start-websocket" - | Send_websocket _ -> "send-websocket" - | Close_websocket _ -> "close-websocket" - | Schedule_timer value -> Printf.sprintf "timer:%d" value.id - | Cancel_effects _ -> "cancel-effects" - | Cancel_network_effects _ -> "cancel-network-effects" -;; - -let equal_runner_effect left right = left = right - -type mirror_request = - { graph : graph - ; scope : graph_scope - } - -type snapshot_activation_request = - { artifact : staged_artifact - ; scope : graph_scope - ; applied_server_t : Server_cursor.t - ; key : graph_key_handle option - } - -let snapshot_activation_scope request = request.scope -let snapshot_activation_cursor request = request.applied_server_t -let snapshot_activation_artifact request = request.artifact -let snapshot_activation_key request = request.key - -(* Each phase owns all evidence needed by the next phase. Effect ids identify - attempts at runtime; sharing an OCaml type never establishes identity. *) -type bootstrap = - | No_bootstrap - | Preparing_snapshot of graph_scope - | Awaiting_snapshot_metadata of graph_scope * effect_id - | Downloading_snapshot of snapshot_download - | Activating_snapshot of snapshot_activation_request - -let bootstrap_scope = function - | No_bootstrap -> None - | Preparing_snapshot scope | Awaiting_snapshot_metadata (scope, _) -> Some scope - | Downloading_snapshot request -> Some request.scope - | Activating_snapshot request -> Some request.scope -;; - -let bootstrap_downloading = function Downloading_snapshot _ -> true | _ -> false -let bootstrap_activating = function Activating_snapshot _ -> true | _ -> false - -type mirror_deletion = - { account : account_scope - ; graph_id : graph_id - ; scope : effect_scope - } - -type mirror_inspection = - | Mirror_available of mirror_request - | Mirror_absent of graph_scope - -type outbox_transition_request = - { expected : Overlay.sync_token - ; key : graph_key_handle option - ; scope : graph_scope - ; transition : Overlay.outbox_transition - } - -type outbox_transition_failure = - { request : outbox_transition_request - ; message : string - } - -type authoritative_batch = - { input : Overlay.authoritative_batch - ; key : graph_key_handle option - ; scope : connection_scope - ; presentation_generation : presentation_generation - ; lifecycle_generation : lifecycle_generation - } - -type worker_effect = - | Inspect_mirror of mirror_request - | Activate_snapshot of snapshot_activation_request - | Delete_mirror of mirror_deletion - | Attach_graph of mirror_request - | Detach_graph of graph_scope - | Reset_managed_account of account_scope - | Inspect_sync of graph_scope - | Apply_outbox_transition of outbox_transition_request - | Apply_authoritative_batch of authoritative_batch - -type output = - | Asset_finished of asset_output - | Protected_finished of protected_output - | State_changed of state - | Resident_scopes_changed of graph_scope list - | Session_state_changed of graph_scope * state - | Bootstrap_progressed of bootstrap_progress - -type instruction = - | Run of runner_effect - | Delegate of worker_effect - | Publish of output - -let equal_instruction left right = left = right -let equal_instructions left right = left = right - -type scoped_error = - { scope : effect_scope - ; message : string - } - -type graph_attachment = - { scope : graph_scope - ; sync : Overlay.sync_view - } - -type sync_inspection = - { scope : graph_scope - ; sync : Overlay.sync_view - } - -type authoritative_commit_result = - { scope : graph_scope - ; commit : Overlay.authoritative_commit - ; sync : Overlay.sync_view - } - -type authoritative_deferred_result = - { scope : graph_scope - ; defer : Overlay.authoritative_defer - } - -type outbox_transition_result = - { scope : graph_scope - ; commit : Overlay.outbox_commit - ; sync : Overlay.sync_view - } - -type snapshot_activation = snapshot_activation_request -type snapshot_activation_error = { activation : snapshot_activation_request; message : string } -let snapshot_activation_completed (request : snapshot_activation_request) : snapshot_activation = request -let snapshot_activation_failed activation ~message = { activation; message } - -type startup_scope = - { account_generation : account_generation - ; presentation_generation : presentation_generation - } - -type timeline_scope = - { startup : startup_scope - ; graph_id : graph_id - ; graph_generation : graph_generation - } - -type event = - | Host_fence_started of host_fence - | Host_fence_released of Logseq_db_types.Graph_types.Uuid.t - | IO_fence_requested of int64 * effect_scope - | IO_fence_completed of int64 * effect_scope * (unit,string) result - | Asset_requested of asset_request - | Protected_requested of protected_request - | Restore_local_account of { user_id : string } - | Account_authenticated of { user_id : string option } - | Local_feed_acknowledged - | Local_outbox_changed - | Resident_graphs_changed of graph_id list - | Visible_graph_changed of graph_id - | Local_graph_outbox_changed of graph_scope - | Timeline_presented of timeline_scope - | Graph_selected of graph_id - | Graph_picker_requested - | Sign_in_requested of startup_scope - | Catalog_refresh_requested of startup_scope - | Online_recovery_requested of timeline_scope - | Online_cancelled of startup_scope - | Local_account_unavailable of startup_scope * local_unavailability - | E2ee_password_submitted of unlock_challenge * string - | Local_cache_deletion_requested of graph_id - | Foreground_changed of - { foreground : bool - ; lifecycle_generation : lifecycle_generation - } - | Graph_detached of graph_scope * (unit, string) result - | Mirror_deleted of mirror_deletion * (unit, string) result - | Mirror_inspected of mirror_inspection - | Graph_attached of graph_attachment - | Graph_attachment_failed of scoped_error - | Sync_inspected of sync_inspection - | Outbox_transition_applied of outbox_transition_result - (** An uncommitted preparation lost its token to a newer durable revision. - The exact pending request and a changed token fence recovery; this is not - evidence of a commit or permission to repeat an unchanged failed request. *) - | Outbox_transition_superseded of - { request : outbox_transition_request; sync : Logseq_overlay_db.Types.sync_view } - | Outbox_transition_failed of outbox_transition_failure - | Authoritative_batch_applied of authoritative_commit_result - | Authoritative_batch_deferred of authoritative_deferred_result - | Authoritative_batch_failed of scoped_error - | Snapshot_activated of snapshot_activation - | Snapshot_activation_failed of snapshot_activation_error - | Runner_completed of runner_completion - | Snapshot_download_progress of bootstrap_progress - | Websocket_opened of connection_scope - | Websocket_message of connection_scope * Sync_protocol.Server.message - | Websocket_protocol_error of connection_scope * Sync_protocol.codec_error - | Websocket_closed of connection_scope * websocket_termination - | Websocket_retired of connection_scope - | Timer_elapsed of timer_id - | Shutdown - -type pending_effect = Pending_effect : 'a effect_ticket -> pending_effect - -type submission_owner = - { batch_id : Overlay.submission_batch_id - ; mutation_ids : graph_id list - ; connection : connection_scope - ; accepted_through : Server_cursor.t option - ; response_timer : timer_id option - } - -type pull_stage = - | Awaiting_pull_response of timer_id - | Applying_pull - | Awaiting_pull_retry of timer_id - -type pull_owner = - { pull_connection : connection_scope - ; issued_since : Server_cursor.t - ; pull_stage : pull_stage - } - -type pending_asset = - { asset_ticket : asset_ticket - ; asset_cancelled : bool - ; asset_notify : bool - } - -type websocket_phase = - | Opening - | Open - | Retiring - -type websocket_owner = - { physical_scope : connection_scope - ; phase : websocket_phase - } - -module Asset_retirement_set = Map.Make (String) -module Pending_id_set = Set.Make (Int) -let asset_retirement_key (scope : graph_scope) retirement = - Printf.sprintf "%S:%S:%S:%S" (Uri.to_string scope.account.managed_sync_origin) - scope.account.user_id (Logseq_db_types.Graph_types.Uuid.to_string scope.graph_id) - (retirement_id_to_string retirement) -;; - -type unlock_preparation = - | No_unlock_preparation - | Fetching_graph_key of graph_scope * effect_id - | Fetching_user_keys of graph_scope * string * effect_id - -type graph_session = - { saved_identity : graph_scope - ; saved_snapshot : snapshot - ; saved_diagnostics : diagnostics - ; saved_pending_effects : pending_effect list - ; saved_asset_retirement_observed : unit Asset_retirement_set.t - ; saved_asset_known_scopes : graph_scope list - ; saved_protected_pending : protected_ticket list - ; saved_assets_pending : pending_asset list - ; saved_selected_graph_value : graph option - ; saved_current_graph_scope : graph_scope option - ; saved_connection_generation : connection_generation - ; saved_graph_key : graph_key_handle option - ; saved_pending_graph_open : mirror_request option - ; saved_pending_mirror_inspection : mirror_request option - ; saved_pending_selected_download : graph_scope option - ; saved_pending_attachment : mirror_request option - ; saved_unlock_preparation : unlock_preparation - ; saved_bootstrap : bootstrap - ; saved_sync_view : Overlay.sync_view option - ; saved_pending_sync_inspection : graph_scope option - ; saved_pending_outbox_transition : outbox_transition_request option - ; saved_submission_owner : submission_owner option - ; saved_pull_owner : pull_owner option - ; saved_pull_wanted : (graph_scope * Server_cursor.t) option - ; saved_force_pull_pending : bool - ; saved_diagnostic_remote_cursor : (graph_scope * Server_cursor.t) option - ; saved_pull_retry_delay : float - ; saved_active_authoritative_batch : authoritative_batch option - ; saved_queued_authoritative_batch : authoritative_batch option - ; saved_deferred_authoritative_owner : Overlay.submission_batch_id option - ; saved_websocket_live : bool - ; saved_foreground : bool - ; saved_foreground_generation : lifecycle_generation option - ; saved_websocket_owner : websocket_owner option - ; saved_wanted_websocket : websocket_request option - ; saved_reconnect_timer : (timer_id * connection_scope) option - ; saved_reconnect_pending : bool - ; saved_reconnect_delay : float - ; saved_closed : bool - } - -type t = - { host_fence : host_fence option - ; host_cleanup_scopes : Host_cleanup_scopes.t - ; resident_order : graph_id list - ; resident_sessions : graph_session list - ; resident_next_generation : int - ; resident_enabled : bool - ; restored_residents : graph_id list option - ; asset_retirement_observed : unit Asset_retirement_set.t - ; asset_known_scopes : graph_scope list - ; protected_pending : protected_ticket list - ; assets_pending : pending_asset list - ; config : config - ; public_state : state - ; user_id : string option - ; lifecycle_generation : lifecycle_generation - ; next_effect_id : int - ; pending_effects : pending_effect list - ; catalog_cache_loading : bool - ; deferred_catalog_reconciliation : bool - ; cached_selected_graph : graph_id option - ; selected_graph_value : graph option - ; deletion_scope : graph_scope option - ; current_graph_scope : graph_scope option - ; connection_generation : connection_generation - ; graph_key : graph_key_handle option - ; pending_graph_open : mirror_request option - ; pending_mirror_inspection : mirror_request option - ; pending_selected_download : graph_scope option - ; pending_attachment : mirror_request option - ; unlock_preparation : unlock_preparation - ; bootstrap : bootstrap - ; sync_view : Overlay.sync_view option - ; pending_sync_inspection : graph_scope option - ; pending_outbox_transition : outbox_transition_request option - ; submission_owner : submission_owner option - ; pull_owner : pull_owner option - ; pull_wanted : (graph_scope * Server_cursor.t) option - ; force_pull_pending : bool - ; diagnostic_remote_cursor : (graph_scope * Server_cursor.t) option - ; pull_retry_delay : float - ; active_authoritative_batch : authoritative_batch option - ; queued_authoritative_batch : authoritative_batch option - ; deferred_authoritative_owner : Overlay.submission_batch_id option - ; websocket_live : bool - ; foreground : bool - ; foreground_generation : lifecycle_generation option - ; websocket_owner : websocket_owner option - ; wanted_websocket : websocket_request option - ; reconnect_timer : (timer_id * connection_scope) option - ; reconnect_pending : bool - ; reconnect_delay : float - ; closed : bool - } - -type create_error = Invalid_create of string - -let initial_startup = - { authenticated = false - ; local_access = Local_pending - ; network_phase = Local_only - ; catalog_loading = false - ; awaiting_selection = false - ; restoring_local = false - ; bootstrapping = false - ; awaiting_e2ee_password = false - ; failure = None - ; account_generation = 0 - ; graph_generation = 0 - ; presentation_generation = 0 - } -;; - -let initial config = - let snapshot = - { local_deletion = None - ; sync_phase = Offline - ; catalog = [] - ; selected_graph = None - ; resident_graphs = [] - ; applied_server_t = None - ; timeline_presentation_pending = false - ; unlock = Unlock_inactive - ; startup = initial_startup - ; last_error = None - ; last_error_attention = None - } - in - Ok - { host_fence = None - ; host_cleanup_scopes = Host_cleanup_scopes.empty - ; resident_order = [] - ; resident_sessions = [] - ; resident_next_generation = 0 - ; resident_enabled = false - ; restored_residents = None - ; asset_retirement_observed = Asset_retirement_set.empty - ; asset_known_scopes = [] - ; protected_pending = [] - ; assets_pending = [] - ; config - ; public_state = { snapshot; diagnostics = { groups = [] } } - ; user_id = None - ; lifecycle_generation = 0L - ; next_effect_id = 1 - ; pending_effects = [] - ; catalog_cache_loading = false - ; deferred_catalog_reconciliation = false - ; cached_selected_graph = None - ; selected_graph_value = None - ; deletion_scope = None - ; current_graph_scope = None - ; pending_selected_download = None - ; connection_generation = 0 - ; graph_key = None - ; pending_graph_open = None - ; pending_mirror_inspection = None - ; pending_attachment = None - ; unlock_preparation = No_unlock_preparation - ; bootstrap = No_bootstrap - ; sync_view = None - ; pending_sync_inspection = None - ; pending_outbox_transition = None - ; pull_owner = None - ; pull_wanted = None; force_pull_pending = false - ; diagnostic_remote_cursor = None - ; pull_retry_delay = 1. - ; submission_owner = None - ; active_authoritative_batch = None - ; queued_authoritative_batch = None - ; deferred_authoritative_owner = None - ; websocket_live = false - ; foreground = true - ; foreground_generation = None - ; websocket_owner = None - ; wanted_websocket = None - ; reconnect_timer = None - ; reconnect_pending = false - ; reconnect_delay = 1. - ; closed = false - } -;; - -let state core = core.public_state -let admitted_graph_scope core = core.current_graph_scope - -let asset_context core = - match core.current_graph_scope, core.selected_graph_value with - | Some scope, Some graph -> - Some { scope; encrypted = graph.encrypted; key = core.graph_key } - | _ -> None -;; - -type transition = - { next : t - ; effects : instruction list - } - -let unchanged next = { next; effects = [] } -let publish core = Publish (State_changed core.public_state) - -let set_snapshot core (snapshot : snapshot) = - let previous = core.public_state.snapshot.startup in - let unlock = - if previous.account_generation <> snapshot.startup.account_generation - || previous.graph_generation <> snapshot.startup.graph_generation - then Unlock_inactive else snapshot.unlock - in - let last_error_attention = match snapshot.last_error with - | None -> None - | Some _ -> Some (Option.value snapshot.last_error_attention ~default:Requires_attention) in - let snapshot = { snapshot with unlock; last_error_attention; startup = { snapshot.startup with - awaiting_e2ee_password = Option.is_some (unlock_challenge unlock) } } in - let previous = core.public_state.snapshot.startup in - let diagnostics = - if previous.account_generation <> snapshot.startup.account_generation - || previous.graph_generation <> snapshot.startup.graph_generation - then { groups = [] } - else core.public_state.diagnostics - in - let diagnostic_remote_cursor = - if previous.account_generation <> snapshot.startup.account_generation - || previous.graph_generation <> snapshot.startup.graph_generation - then None - else core.diagnostic_remote_cursor - in - { core with public_state = { snapshot; diagnostics }; diagnostic_remote_cursor } -;; - -let network_allowed core = - let f=core.public_state.snapshot.startup in - f.network_phase <> Local_only && f.authenticated -;; -let startup_scope_is_current core (scope : startup_scope) = - let f=core.public_state.snapshot.startup in - scope.account_generation=f.account_generation && scope.presentation_generation=f.presentation_generation -;; -let timeline_scope_is_current core (scope : timeline_scope) = - startup_scope_is_current core scope.startup - && core.public_state.snapshot.selected_graph=Some scope.graph_id - && core.public_state.snapshot.startup.graph_generation=scope.graph_generation -;; -let network_phase core phase = - let snapshot=core.public_state.snapshot in - set_snapshot core {snapshot with startup={snapshot.startup with network_phase=phase}} -;; -let local_unavailable core reason message = - let snapshot=core.public_state.snapshot in - let startup={snapshot.startup with local_access=Local_unavailable reason; - restoring_local=false;bootstrapping=false;catalog_loading=false;awaiting_e2ee_password=false; - failure=Some (if reason=Local_catalog_unavailable then During_catalog else During_local_restore)} in - let next=set_snapshot core {snapshot with startup; unlock = Unlock_failed (Retry_preparation message); - last_error=Some message;last_error_attention=Some Requires_attention;sync_phase=Offline} in - {next;effects=[publish next]} -;; - -let fail core stage message = - let startup = - { core.public_state.snapshot.startup with - failure = Some stage - ; restoring_local = false - ; bootstrapping = false - ; catalog_loading = false - ; local_access = (match core.public_state.snapshot.startup.local_access with - | Local_available -> Local_available - | _ -> Local_unavailable (if stage=During_catalog then Local_catalog_unavailable else Local_restore_failed)) - ; awaiting_e2ee_password = false - } - in - let snapshot = - { core.public_state.snapshot with - sync_phase = Failed - ; startup - ; unlock = Unlock_failed (Retry_preparation message) - ; last_error_attention = Some Requires_attention - ; last_error = Some message - } - in - let core = - { core with - pending_effects = [] - ; pull_owner = None - ; pull_wanted = None; force_pull_pending = false - ; pull_retry_delay = 1. - ; pending_mirror_inspection = None - ; pending_selected_download = None - ; pending_attachment = None - ; unlock_preparation = No_unlock_preparation - ; bootstrap = No_bootstrap - ; pending_sync_inspection = None - ; pending_outbox_transition = None - ; active_authoritative_batch = None - ; queued_authoritative_batch = None - ; deferred_authoritative_owner = None - } - in - let next = set_snapshot core snapshot in - { next; effects = [ publish next ] } -;; - -let cleanup_failed core message = - let next = - set_snapshot core { core.public_state.snapshot with last_error = Some message; last_error_attention = Some Requires_attention } - in - { next; effects = [ publish next ] } -;; - -let account_scope core user_id = - { managed_sync_origin = core.config.managed_sync_origin - ; user_id - ; account_generation = core.public_state.snapshot.startup.account_generation - ; presentation_generation = core.public_state.snapshot.startup.presentation_generation - ; lifecycle_generation = core.lifecycle_generation - } -;; - -let graph_scope_is_current core scope = core.current_graph_scope = Some scope - -let connection_is_current core (scope : connection_scope) = - graph_scope_is_current core scope.graph - && scope.connection_generation = core.connection_generation -;; - -let request_kind : type a. a runner_request -> a request_kind = function - | Load_catalog _ -> Load_catalog_kind - | Save_catalog _ -> Save_catalog_kind - | Fetch_catalog _ -> Fetch_catalog_kind - | Fetch_snapshot_baseline _ -> Fetch_snapshot_baseline_kind - | Fetch_snapshot_metadata _ -> Fetch_snapshot_metadata_kind - | Download_snapshot _ -> Download_snapshot_kind - | Fetch_e2ee_graph_key _ -> Fetch_e2ee_graph_key_kind - | Fetch_e2ee_user_keys _ -> Fetch_e2ee_user_keys_kind - | Load_and_unlock_graph_key _ -> Load_and_unlock_graph_key_kind - | Fetch_and_unlock_graph_key _ -> Fetch_and_unlock_graph_key_kind - | Unlock_private_key _ -> Unlock_private_key_kind - | Delete_account_secrets _ -> Delete_account_secrets_kind - | Encrypt_protected_values _ -> Encrypt_protected_values_kind - | Decrypt_protected_values _ -> Decrypt_protected_values_kind -;; - -let issue_request : type a. t -> a runner_request -> t * runner_effect = - fun core request -> - let ticket = - { id = core.next_effect_id - ; scope = - (match request, (bootstrap_scope core.bootstrap) with - | (Fetch_e2ee_user_keys _ | Unlock_private_key _), Some graph -> - (* Shared account material is requested for this graph's challenge. *) - effect_scope_of_graph graph - | _ -> scope_of_request request) - ; kind = request_kind request - } - in - ( { core with - next_effect_id = core.next_effect_id + 1 - ; pending_effects = Pending_effect ticket :: core.pending_effects - } - , Request (ticket, request) ) -;; - -let begin_snapshot_bootstrap core scope = - if not (network_allowed core) then local_unavailable core Local_mirror_unavailable "This graph is not downloaded. Connect to download it." else - let unlock = match core.public_state.snapshot.unlock with - | Unlock_running submission when submission.stage = Attachment -> Unlock_bootstrapping submission - | phase -> phase in - let core = set_snapshot - { core with bootstrap = Awaiting_snapshot_metadata (scope, core.next_effect_id) } - { core.public_state.snapshot with unlock } in - let next, request = issue_request core (Fetch_snapshot_metadata scope) in - let effects = match unlock with - | Unlock_bootstrapping _ -> [ publish next; Run request ] - | _ -> [ Run request ] in - { next; effects } -;; - -let begin_e2ee_key_access core scope = - if not (network_allowed core) then local_unavailable core Local_key_unavailable "The local encryption key is unavailable. Connect to recover access." else - let core = set_snapshot - { core with bootstrap = Preparing_snapshot scope - ; unlock_preparation = Fetching_graph_key (scope, core.next_effect_id) } - { core.public_state.snapshot with unlock = Unlock_inactive } in - let next, request = issue_request core (Fetch_e2ee_graph_key scope) in - { next; effects = [ Run request ] } -;; - -let start_websocket core graph = - if not (network_allowed core) then unchanged core else - let core = - { core with - connection_generation = core.connection_generation + 1 - ; websocket_live = false - ; submission_owner = None - ; pull_owner = None - ; pull_wanted = - (match core.pull_wanted with - | Some (scope, _) as wanted when scope = graph -> wanted - | _ -> None) - ; pull_retry_delay = 1. - ; active_authoritative_batch = None - ; queued_authoritative_batch = None - ; deferred_authoritative_owner = None - } - in - let scope = { graph; connection_generation = core.connection_generation } in - let uri = - Uri.with_scheme core.config.managed_sync_origin (Some "wss") - |> fun uri -> - Uri.with_path - uri - ("/sync/" ^ Logseq_db_types.Graph_types.Uuid.to_string graph.graph_id) - in - let next = - set_snapshot core { core.public_state.snapshot with sync_phase = Connecting } - in - { next; effects = [ publish next; Run (Start_websocket { scope; uri }) ] } -;; - -let authoritative_input_error_message = function - | Overlay.Invalid_server_cursor -> "authoritative server cursor is invalid" - | Invalid_encoded_transaction -> "authoritative transaction is empty or its byte limit is invalid" - | Authoritative_transaction_too_large -> "authoritative transaction exceeds the byte limit" - | Authoritative_batch_empty -> "authoritative batch contains no transactions" - | Authoritative_batch_too_large -> "authoritative batch exceeds its transaction count or byte limit" - | Authoritative_cursor_not_strictly_ordered -> "authoritative transaction cursors are not strictly ordered" - | Authoritative_through_mismatch -> "authoritative batch final cursor does not match its through cursor" -;; - -let checksum value = Overlay.Checksum.of_string ("checksum:v1:" ^ value) - -let decode_snapshot_baseline source = - Sync_protocol.decode_snapshot_baseline source - |> Result.map_error Sync_protocol.error_to_string -;; - -type snapshot_uri_response = - { ok : bool - ; url : string - ; t : int64 - } -[@@deriving of_yojson { strict = false }] - -let decode_snapshot_uri source = - try - match - Logseq_db_types.Json_codec.decode - snapshot_uri_response_of_yojson - (Yojson.Safe.from_string source) - with - | Ok { ok = true; url; t } -> - let uri = Uri.of_string url in - (match Server_cursor.of_int64 t with - | Error `Negative_cursor -> Error "snapshot metadata cursor is invalid" - | Ok server_t -> - (match Uri.scheme uri, Uri.host uri, Uri.userinfo uri, Uri.fragment uri with - | Some "https", Some host, None, None when String.length host > 0 -> - Ok (uri, server_t) - | _ -> Error "snapshot download URL is invalid")) - | Ok _ -> Error "snapshot metadata: server reported ok=false" - | Error message -> - let fields = List.map (fun name -> "Core.snapshot_uri_response." ^ name, - "field " ^ name ^ " is missing or has an invalid type") ["ok"; "url"; "t"] in - Error ("snapshot metadata: " ^ Logseq_db_types.Json_codec.error_context ~fields message) - with - | Yojson.Json_error _ -> Error "snapshot metadata is not valid JSON" -;; - -let save_cache core = - match core.user_id with - | None -> core, [] - | Some user_id -> - let cache = - catalog_cache - ~user_id - ~graphs:core.public_state.snapshot.catalog - ~selected_graph:core.cached_selected_graph - in - let cache = if core.resident_enabled then { cache with cache_resident_graphs = Some core.resident_order } else cache in - let next, request = - issue_request core (Save_catalog { account = account_scope core user_id; cache }) - in - next, [ Run request ] -;; - -(* Only operation state shared by graph/account replacement paths. Scope, - pending-effect and teardown policies remain with each transition. *) -let reset_graph_operations core = - { core with - graph_key = None - ; pending_graph_open = None - ; pending_attachment = None - ; unlock_preparation = No_unlock_preparation - ; bootstrap = No_bootstrap - ; sync_view = None - ; pending_sync_inspection = None - ; pending_outbox_transition = None - ; submission_owner = None - ; pull_owner = None - ; pull_wanted = None; force_pull_pending = false - ; pull_retry_delay = 1. - ; active_authoritative_batch = None - ; queued_authoritative_batch = None - ; deferred_authoritative_owner = None - } -;; - -let authenticate_new_account core user_id = - let startup = - { initial_startup with - authenticated = true - ; network_phase = core.public_state.snapshot.startup.network_phase - ; catalog_loading = true - ; account_generation = core.public_state.snapshot.startup.account_generation + 1 - ; presentation_generation = - core.public_state.snapshot.startup.presentation_generation + 1 - } - in - let snapshot = - { core.public_state.snapshot with - local_deletion = None - ; catalog = [] - ; selected_graph = None - ; applied_server_t = None - ; startup - ; last_error = None - ; last_error_attention = None - } - in - let core = - set_snapshot - { (reset_graph_operations core) with - user_id = Some user_id - ; pending_effects = [] - ; catalog_cache_loading = false - ; deferred_catalog_reconciliation = false - ; deletion_scope = None - ; cached_selected_graph = None - ; current_graph_scope = None - ; pending_selected_download = None - ; pending_mirror_inspection = None - ; websocket_live = false - } - snapshot - in - let next, request = issue_request core (Fetch_catalog (account_scope core user_id)) in - { next; effects = [ publish next; Run request ] } -;; - -let clear_reconciliation_failure (snapshot : snapshot) = - match snapshot.startup.failure with - | Some (During_local_restore | During_e2ee | During_bootstrap) -> snapshot - | None | Some _ -> - { snapshot with - startup = { snapshot.startup with failure = None } - ; last_error = None - } -;; - -let request_catalog_reconciliation core = - if not (network_allowed core) then unchanged { core with deferred_catalog_reconciliation = false } else - let snapshot = clear_reconciliation_failure core.public_state.snapshot in - let startup = { snapshot.startup with authenticated = true; catalog_loading = true } in - let core = - set_snapshot - { core with deferred_catalog_reconciliation = false } - { snapshot with startup } - in - match core.user_id with - | None -> unchanged core - | Some user_id -> - let next, request = issue_request core (Fetch_catalog (account_scope core user_id)) in - { next; effects = [ publish next; Run request ] } -;; - -let authenticate core user_id = - match core.user_id with - | Some current_user_id when String.equal current_user_id user_id -> - let snapshot = clear_reconciliation_failure core.public_state.snapshot in - let startup = { snapshot.startup with authenticated = true } in - let core = set_snapshot core { snapshot with startup } in - if core.catalog_cache_loading - then ( - let next = { core with deferred_catalog_reconciliation = true } in - { next; effects = [ publish next ] }) - else request_catalog_reconciliation core - | None | Some _ -> authenticate_new_account core user_id -;; - -let restore_local core user_id = - let startup = - { initial_startup with - restoring_local = true - ; presentation_generation = core.public_state.snapshot.startup.presentation_generation + 1 - ; account_generation = core.public_state.snapshot.startup.account_generation + 1 - } - in - let core = - set_snapshot - { (reset_graph_operations core) with - user_id = Some user_id - ; pending_effects = [] - ; catalog_cache_loading = true - ; deferred_catalog_reconciliation = false - ; deletion_scope = None - ; cached_selected_graph = None - ; selected_graph_value = None - ; current_graph_scope = None - ; pending_selected_download = None - ; pending_mirror_inspection = None - ; websocket_live = false - } - { core.public_state.snapshot with local_deletion = None; startup } - in - let next, request = issue_request core (Load_catalog (account_scope core user_id)) in - { next; effects = [ publish next; Run request ] } -;; - -let sign_out core = - let old_account = Option.map (account_scope core) core.user_id in - let generation_increment = if Option.is_some old_account then 1 else 0 in - let current_startup = core.public_state.snapshot.startup in - let snapshot = - { core.public_state.snapshot with - local_deletion = None - ; sync_phase = Offline - ; catalog = [] - ; selected_graph = None - ; applied_server_t = None - ; startup = - { initial_startup with - account_generation = current_startup.account_generation + generation_increment - ; graph_generation = current_startup.graph_generation - ; presentation_generation = - current_startup.presentation_generation + generation_increment - } - } - in - let next = - set_snapshot - { (reset_graph_operations core) with - user_id = None - ; pending_effects = [] - ; catalog_cache_loading = false - ; deferred_catalog_reconciliation = false - ; deletion_scope = None - ; cached_selected_graph = None - ; current_graph_scope = None - ; pending_selected_download = None - ; pending_mirror_inspection = None - ; websocket_live = false - } - snapshot - in - match old_account with - | None -> { next; effects = [ publish next ] } - | Some account -> - let next, deletion = issue_request next (Delete_account_secrets account) in - { next - ; effects = - [ Run (Cancel_effects (effect_scope_of_account account)) - ; Delegate (Reset_managed_account account) - ; Run deletion - ; publish next - ] - } -;; - -let graph_in_catalog graphs graph_id = - List.find_opt - (fun (graph : graph) -> - Logseq_db_types.Graph_types.Uuid.equal graph.graph_id graph_id) - graphs -;; - -let is_graph_key_access (Pending_effect ticket) = - match ticket.kind with - | Load_and_unlock_graph_key_kind - | Fetch_e2ee_graph_key_kind - | Fetch_e2ee_user_keys_kind - | Unlock_private_key_kind - | Fetch_and_unlock_graph_key_kind -> true - | _ -> false -;; - -let host_fence_blocks core graph_id=match core.host_fence with - |None->false|Some fence->match fence.target with All_graphs->true|One_graph id->id=graph_id -;; -let select_graph_transition core ~persist graph_id = - if host_fence_blocks core graph_id then unchanged core else - match core.user_id, graph_in_catalog core.public_state.snapshot.catalog graph_id with - | Some user_id, Some graph -> - let graph_generation = max core.resident_next_generation core.public_state.snapshot.startup.graph_generation + 1 in - let startup = - { core.public_state.snapshot.startup with - awaiting_selection = false - ; local_access = Local_pending - ; presentation_generation = core.public_state.snapshot.startup.presentation_generation + 1 - ; network_phase = (match core.public_state.snapshot.startup.network_phase with - | Online_by_user -> Online_by_user | _ -> Local_only) - ; restoring_local = true - ; bootstrapping = false - ; awaiting_e2ee_password = false - ; failure = None - ; graph_generation - } - in - let snapshot = - { core.public_state.snapshot with - selected_graph = Some graph_id - ; applied_server_t = None - ; sync_phase = Offline - ; last_error = None - ; startup - } - in - let scope = { account = { (account_scope core user_id) with presentation_generation=startup.presentation_generation }; graph_id; graph_generation } in - let mirror_request = { graph; scope } in - let core = - set_snapshot - { (reset_graph_operations core) with - cached_selected_graph = Some graph_id - ; resident_next_generation = graph_generation - ; websocket_live = false - ; pending_effects = List.filter (fun pending -> not (is_graph_key_access pending)) core.pending_effects - ; selected_graph_value = Some graph - ; current_graph_scope = Some scope - ; pending_selected_download = (if persist then Some scope else None) - ; pending_mirror_inspection = Some mirror_request - } - snapshot - in - let next, cache = if persist then save_cache core else core, [] in - { next; effects = Delegate (Inspect_mirror mirror_request) :: publish next :: cache } - | _ -> unchanged core -;; - -let select_graph core graph_id = select_graph_transition core ~persist:true graph_id - -let begin_missing_mirror_bootstrap core scope = - let startup = - { core.public_state.snapshot.startup with restoring_local = false; bootstrapping = true } - in - let core = set_snapshot { core with pending_selected_download = None } - { core.public_state.snapshot with startup } in - match core.selected_graph_value with - | Some graph when graph.encrypted -> - let next, request = issue_request core (Load_and_unlock_graph_key scope) in - { next; effects = [ publish next; Run request ] } - | Some _ | None -> - let requested = begin_snapshot_bootstrap core scope in - { requested with effects = publish requested.next :: requested.effects } -;; - -let mirror_inspected core = function - | Mirror_available request - when graph_scope_is_current core request.scope - && core.pending_mirror_inspection = Some request -> - let core = { core with pending_mirror_inspection = None; pending_selected_download = None } in - (match core.selected_graph_value, core.graph_key with - | Some graph, Some key - when graph.encrypted && graph_key_handle_scope key = request.scope -> - { next = { core with pending_attachment = Some request } - ; effects = [ Delegate (Attach_graph request) ] - } - | Some graph, _ when graph.encrypted -> - let next, key_request = - issue_request core (Load_and_unlock_graph_key request.scope) - in - { next = { next with pending_graph_open = Some request } - ; effects = [ Run key_request ] - } - | Some _, _ | None, _ -> - { next = { core with pending_attachment = Some request } - ; effects = [ Delegate (Attach_graph request) ] - }) - | Mirror_absent scope - when graph_scope_is_current core scope - && Option.fold - ~none:false - ~some:(fun (request : mirror_request) -> request.scope = scope) - core.pending_mirror_inspection -> - let core = { core with pending_mirror_inspection = None } in - if not (network_allowed core) then - if core.pending_selected_download = Some scope then - (* Only the user's current selection may request online admission. Native - permission and verified identity still precede any remote effect. *) - let core = network_phase core Online_by_user in - let snapshot = core.public_state.snapshot in - let startup = { snapshot.startup with authenticated = false; - restoring_local = false; bootstrapping = true } in - let next = set_snapshot { core with bootstrap = Preparing_snapshot scope } - { snapshot with startup } in - { next; effects = [publish next] } - else - local_unavailable core Local_mirror_unavailable "This graph is not downloaded. Download it to open it on this device." - else begin_missing_mirror_bootstrap core scope - | _ -> unchanged core -;; - -let graph_attached core (attachment : graph_attachment) = - if - (not (graph_scope_is_current core attachment.scope)) - || not - (Option.fold - ~none:false - ~some:(fun (request : mirror_request) -> request.scope = attachment.scope) - core.pending_attachment) - then unchanged core - else ( - let core = { core with pending_attachment = None } in - let startup = - { core.public_state.snapshot.startup with - restoring_local = false - ; local_access = Local_available - ; bootstrapping = false - ; failure = None - } - in - let snapshot = - { core.public_state.snapshot with - sync_phase = (if network_allowed core then Current else Offline) - ; applied_server_t = Some (Overlay.sync_view_checkpoint attachment.sync) - ; timeline_presentation_pending = true - ; unlock = (match core.public_state.snapshot.unlock with - | (Unlock_running submission | Unlock_bootstrapping submission) - when submission.stage = Attachment - && submission.challenge.challenge_scope = attachment.scope -> - Unlock_presentable { submission; attached_scope = attachment.scope } - | phase -> phase) - ; startup - ; last_error = None - } - in - let core = set_snapshot { core with sync_view = Some attachment.sync } snapshot in - let online=start_websocket core attachment.scope in - if online.effects=[] then {next=core;effects=[publish core]} else online) -;; - -let operation_name = function - | Overlay.Save_block_operation -> "save-block" - | Append_journal_operation -> "append-journal" - | Insert_blocks_operation -> "insert-blocks" - | Delete_blocks_operation -> "delete-blocks" - | Create_journal_page_operation -> "create-journal-page" - | Set_task_status_operation -> "set-task-status" - | Clear_task_status_operation -> "clear-task-status" - | Publish_asset_operation -> "publish-asset" - | Set_asset_reference_operation -> "set-asset-reference" -;; - -let submission_message batch = - let txs = - Overlay.submission_batch_wires batch - |> List.map (fun wire -> - Sync_protocol.Client. - { tx = Overlay.submission_wire_protected_transaction wire - ; tx_id = Some (Overlay.submission_wire_mutation_id wire) - ; outliner_op = Some (operation_name (Overlay.submission_wire_operation wire)) - }) - in - Sync_protocol.Client.Tx_batch - { client_revision = - Some (Overlay.Submission_batch_id.to_string (Overlay.submission_batch_id batch)) - ; t_before = Overlay.submission_batch_t_before batch - ; txs - } -;; - -let pull_target_pending core graph cursor = - match core.pull_wanted with - | Some (scope, target) -> scope = graph && Server_cursor.compare target cursor > 0 - | None -> false -;; - -let pull_in_progress core = - match core.current_graph_scope, core.public_state.snapshot.applied_server_t with - | Some graph, Some cursor -> - core.force_pull_pending - || Option.is_some core.pull_owner - || Option.is_some core.active_authoritative_batch - || pull_target_pending core graph cursor - | _ -> false -;; - -let settled_sync_phase ?(baseline = false) core = - if pull_in_progress core || (baseline && core.websocket_live) then Pulling else Current -;; - -let phase_transition core sync_phase = - let sync_phase = if sync_phase = Current then settled_sync_phase core else sync_phase in - if core.public_state.snapshot.sync_phase = sync_phase - then unchanged core - else ( - let next = set_snapshot core { core.public_state.snapshot with sync_phase } in - { next; effects = [ publish next ] }) -;; - -let has_blocked_submissions sync = - List.exists - (fun (item : Overlay.submission_descriptor) -> item.state = Overlay.Blocked) - (Overlay.sync_view_submissions sync) -;; - -let blocked_changes_message = - "Some local changes could not be synced and are not included in the journal. They \ - remain stored on this device. Do not delete the local graph copy." -;; - -let rejection_message (rejection : Sync_protocol.rejection) = - let reason = - match rejection.reason with - | Empty_tx_data -> "empty transaction" - | Invalid_tx -> "invalid transaction" - | Invalid_t_before -> "invalid sync version" - | Db_transact_failed -> "the server could not apply the transaction" - | Snapshot_upload_in_progress -> "a graph upload is in progress" - | Stale -> "the graph changed before submission" - in - "The server rejected your changes: " - ^ reason - ^ ". " - ^ blocked_changes_message - ^ Option.fold - ~none:"" - ~some:(fun detail -> " Details: " ^ detail) - rejection.error_detail -;; - -let plan_submission core = - if not (network_allowed core) then unchanged core else - match core.sync_view, core.current_graph_scope, core.websocket_live with - | Some sync, Some scope, true -> - let descriptors = Overlay.sync_view_submissions sync in - let recovering = - List.find_map - (fun (item : Overlay.submission_descriptor) -> - match item.state with - | Overlay.Submitted batch_id - when Option.fold - ~none:true - ~some:(Overlay.Submission_batch_id.equal batch_id) - core.deferred_authoritative_owner -> Some batch_id - | Queued - | Submitted _ - | Accepted_pending_authoritative _ - | Stale_rejected_pending_authoritative _ - | Blocked -> None) - descriptors - in - let barrier_pending = - List.exists - (fun (item : Overlay.submission_descriptor) -> - match item.state with - | Overlay.Accepted_pending_authoritative _ - | Stale_rejected_pending_authoritative _ -> true - | Queued | Submitted _ | Blocked -> false) - descriptors - in - let request transition = - let request = - { expected = Overlay.sync_view_token sync - ; key = core.graph_key - ; scope - ; transition - } - in - let next = - set_snapshot - { core with pending_outbox_transition = Some request } - { core.public_state.snapshot with sync_phase = Submitting } - in - { next; effects = [ publish next; Delegate (Apply_outbox_transition request) ] } - in - (match core.pending_outbox_transition, core.submission_owner with - | Some _, _ -> unchanged core - | None, Some owner -> - phase_transition - core - (if Option.is_some owner.accepted_through then Pulling else Submitting) - | None, None -> - if - Option.is_some core.active_authoritative_batch - && Option.is_none core.deferred_authoritative_owner - then phase_transition core Pulling - else ( - match recovering with - | Some batch_id -> request (Overlay.Retry_group batch_id) - | None when barrier_pending || Option.is_some core.active_authoritative_batch -> - phase_transition core Pulling - | None -> - let rec take n acc = function - | [] -> List.rev acc - | _ when n = 0 -> List.rev acc - | (item : Overlay.submission_descriptor) :: rest -> - if item.operation = Overlay.Delete_blocks_operation - then if acc = [] then [ item.mutation_id ] else List.rev acc - else take (n - 1) (item.mutation_id :: acc) rest - in - let ids = - descriptors - |> List.filter (fun (item : Overlay.submission_descriptor) -> - item.state = Overlay.Queued && item.dependency_eligible) - |> take core.config.limits.submission_batch_size [] - in - if ids = [] - then - if has_blocked_submissions sync - then ( - let next = - set_snapshot - core - { core.public_state.snapshot with - sync_phase = Paused - ; last_error = - Some - (Option.value - core.public_state.snapshot.last_error - ~default:blocked_changes_message) - } - in - if next.public_state = core.public_state - then unchanged core - else { next; effects = [ publish next ] }) - else phase_transition core Current - else request (Overlay.Submit_group ids))) - | _ -> unchanged core -;; - -(* Retained observation only; synchronization still uses its existing owners and targets. *) -let note_remote_cursor core graph through = - let diagnostic_remote_cursor = - match core.diagnostic_remote_cursor, through with - | Some (scope, old), Some cursor when scope = graph -> - Some (graph, if Server_cursor.compare old cursor >= 0 then old else cursor) - | _, Some cursor -> Some (graph, cursor) - | previous, None -> previous - in - { core with diagnostic_remote_cursor } -;; - -let note_pull_target core graph through = - let core = note_remote_cursor core graph through in - let previous = - match core.pull_wanted with - | Some (scope, target) when scope = graph -> Some target - | _ -> None - in - let target = - match previous, through with - | Some old, Some target -> - Some (if Server_cursor.compare old target >= 0 then old else target) - | None, target | target, None -> target - in - { core with pull_wanted = Option.map (fun target -> graph, target) target } -;; - -(* The owner spans send, response, and authoritative COMMIT completion. Timers - are policy facts: the runner never waits or retries a pull on its own. *) -let request_pull ?through ?(force = false) core = - match core.current_graph_scope, core.public_state.snapshot.applied_server_t with - | Some graph, Some cursor when core.websocket_live -> - let core = note_pull_target core graph through in - let occupied = - Option.fold - ~none:false - ~some:(fun owner -> connection_is_current core owner.pull_connection) - core.pull_owner - in - if - occupied - || Option.is_some core.active_authoritative_batch - || (core.force_pull_pending && not force - && not (pull_target_pending core graph cursor) - && (Option.is_some core.pending_outbox_transition - || Option.is_some core.submission_owner)) - || not (force || core.force_pull_pending || pull_target_pending core graph cursor) - then unchanged core - else ( - let connection = { graph; connection_generation = core.connection_generation } in - let timer = core.next_effect_id in - let next = - set_snapshot - { core with - next_effect_id = timer + 1 - ; force_pull_pending = false - ; pull_owner = - Some - { pull_connection = connection - ; issued_since = cursor - ; pull_stage = Awaiting_pull_response timer - } - } - { core.public_state.snapshot with sync_phase = Pulling } - in - { next - ; effects = - (if next.public_state = core.public_state then [] else [ publish next ]) - @ [ Run - (Send_websocket - { scope = connection - ; message = Sync_protocol.Client.Pull { since = Some cursor } - }) - ; Run - (Schedule_timer - { id = timer - ; scope = effect_scope_of_connection connection - ; delay_seconds = 30. - }) - ] - }) - | _ -> unchanged core -;; - -let pull_no_progress core (connection : connection_scope) = - let cursor = - Option.value core.public_state.snapshot.applied_server_t ~default:Server_cursor.zero - in - let core = { core with pull_owner = None } in - if not (pull_target_pending core connection.graph cursor) - then unchanged { core with pull_retry_delay = 1. } - else ( - let timer = core.next_effect_id in - { next = - { core with - next_effect_id = timer + 1 - ; pull_owner = - Some - { pull_connection = connection - ; issued_since = cursor - ; pull_stage = Awaiting_pull_retry timer - } - ; pull_retry_delay = min 30. (core.pull_retry_delay *. 2.) - } - ; effects = - [ Run - (Schedule_timer - { id = timer - ; scope = effect_scope_of_connection connection - ; delay_seconds = core.pull_retry_delay - }) - ] - }) -;; - -let outbox_transition_applied core (result : outbox_transition_result) = - if - (not (graph_scope_is_current core result.scope)) - || not - (Option.fold - ~none:false - ~some:(fun (request : outbox_transition_request) -> - request.scope = result.scope - && request.transition = result.commit.transition) - core.pending_outbox_transition) - then unchanged core - else ( - let transition_batch_id = - match result.commit.transition with - | Overlay.Retry_group _ -> None - | Accept_group { batch_id; _ } | Reject_group { batch_id; _ } -> Some batch_id - | Submit_group _ -> None - in - let resumes_deferred = - match core.deferred_authoritative_owner, transition_batch_id with - | Some awaited, Some observed -> Overlay.Submission_batch_id.equal awaited observed - | None, _ | Some _, None -> false - in - let submission_owner = - match result.commit.transition, core.submission_owner with - | (Overlay.Reject_group { batch_id; _ } | Accept_group { batch_id; _ }), Some owner - when Overlay.Submission_batch_id.equal batch_id owner.batch_id -> None - | (Submit_group _ | Retry_group _ | Accept_group _ | Reject_group _), owner -> owner - in - let core = - { core with - pending_outbox_transition = None - ; sync_view = Some result.sync - ; submission_owner - ; deferred_authoritative_owner = - (if resumes_deferred then None else core.deferred_authoritative_owner) - } - in - let resume_effects = - match resumes_deferred, core.active_authoritative_batch with - | true, Some batch -> [ Delegate (Apply_authoritative_batch batch) ] - | false, _ | true, None -> [] - in - match result.commit.submission_batch, core.current_graph_scope with - | Some _, Some _ when not core.websocket_live -> unchanged core - | Some batch, Some graph -> - let connection = { graph; connection_generation = core.connection_generation } in - let mutation_ids = - Overlay.submission_batch_wires batch - |> List.map Overlay.submission_wire_mutation_id - in - let next = - { core with - next_effect_id = core.next_effect_id + 1 - ; submission_owner = - Some - { batch_id = Overlay.submission_batch_id batch - ; mutation_ids - ; connection - ; accepted_through = None - ; response_timer = Some core.next_effect_id - } - } - in - { next - ; effects = - resume_effects - @ [ Run - (Send_websocket { scope = connection; message = submission_message batch }) - ; Run - (Schedule_timer - { id = core.next_effect_id - ; scope = effect_scope_of_connection connection - ; delay_seconds = 30. - }) - ] - } - | _ -> - let next = - set_snapshot core { core.public_state.snapshot with sync_phase = Current } - in - if resume_effects <> [] - then ( - let next = - set_snapshot next { next.public_state.snapshot with sync_phase = Pulling } - in - { next; effects = resume_effects }) - else ( - match result.commit.transition with - | Overlay.Accept_group _ | Reject_group _ -> - let through = - match result.commit.transition with - | Overlay.Accept_group { barrier; _ } -> Some barrier.through - | Reject_group { resolution = Stale { through }; _ } -> Some through - | Reject_group { resolution = Definitive { partition; _ }; _ } -> - Option.map - (fun (barrier : Overlay.acceptance_barrier) -> barrier.through) - partition.acceptance_barrier - | Submit_group _ | Retry_group _ -> None - in - let demanded = - match next.current_graph_scope with - | Some graph -> note_pull_target next graph through - | None -> next - in - let next = - set_snapshot - demanded - { demanded.public_state.snapshot with - sync_phase = - settled_sync_phase ~baseline:(Option.is_none through) demanded - } - in - let pulled = request_pull ?through ~force:(Option.is_none through) next in - { pulled with effects = publish pulled.next :: pulled.effects } - | Submit_group _ | Retry_group _ -> - let planned = plan_submission next in - { next = planned.next; effects = publish next :: planned.effects })) -;; - -let websocket_opened core connection = - if (not (connection_is_current core connection)) || core.websocket_live - then unchanged core - else ( - let next = - set_snapshot - { core with websocket_live = true } - { core.public_state.snapshot with sync_phase = Pulling } - in - let pulled = request_pull ~force:true next in - { pulled with effects = publish next :: pulled.effects }) -;; - -let authoritative_input core t checksum_value txs = - let current = core.public_state.snapshot.applied_server_t in - let ahead cursor = - Option.fold - ~none:true - ~some:(fun current -> Server_cursor.compare cursor current > 0) - current - in - let txs = List.filter (fun tx -> ahead tx.Sync_protocol.Server.t) txs in - if - txs = [] - && Option.fold - ~none:true - ~some:(fun current -> Server_cursor.compare t current >= 0) - current - then Ok None - else ( - let rec encode acc = function - | [] -> Ok (List.rev acc) - | tx :: rest -> - (match - Overlay.encoded_transaction_of_string - ~maximum_bytes:core.config.limits.maximum_response_bytes - tx.Sync_protocol.Server.tx - with - | Error failure -> Error (authoritative_input_error_message failure) - | Ok transaction -> - encode - (Overlay.authoritative_transaction ~cursor:tx.t ~transaction :: acc) - rest) - in - Result.bind (encode [] txs) (fun transactions -> - let checksum = - match checksum_value with - | None -> Ok None - | Some value -> Result.map Option.some (checksum value) - in - Result.bind checksum (fun checksum -> - Overlay.authoritative_batch - ~maximum_count:4096 - ~maximum_bytes:core.config.limits.maximum_response_bytes - ~transactions - ~through:t - ~checksum - |> Result.map Option.some - |> Result.map_error authoritative_input_error_message))) -;; - -let enqueue_authoritative core batch = - match core.active_authoritative_batch with - | None -> - { next = { core with active_authoritative_batch = Some batch } - ; effects = [ Delegate (Apply_authoritative_batch batch) ] - } - | Some _ -> - { next = { core with queued_authoritative_batch = Some batch }; effects = [] } -;; - -let apply_accept core owner t checksum_value = - match core.sync_view, checksum_value with - | Some sync, Some checksum_value -> - let through = t in - (match checksum checksum_value with - | Error _ -> fail core During_catalog "invalid acceptance checksum" - | Ok checksum -> - let core = note_remote_cursor core owner.connection.graph (Some t) in - let request = - { expected = Overlay.sync_view_token sync - ; key = core.graph_key - ; scope = owner.connection.graph - ; transition = - Overlay.Accept_group - { batch_id = owner.batch_id; barrier = Overlay.{ through; checksum } } - } - in - { next = - { core with - pending_outbox_transition = Some request - ; submission_owner = - Some { owner with accepted_through = Some t; response_timer = None } - } - ; effects = [ Delegate (Apply_outbox_transition request) ] - }) - | _ -> fail core During_catalog "invalid acceptance barrier" -;; - -let apply_rejection core owner rejection = - match core.sync_view with - | None -> unchanged core - | Some sync -> - let uuid_equal = Logseq_db_types.Graph_types.Uuid.equal in - let rec consume_prefix members prefix = - match members, prefix with - | rest, [] -> Ok rest - | member :: members, item :: prefix when uuid_equal member item -> - consume_prefix members prefix - | [], _ :: _ | _ :: _, _ :: _ -> - Error "rejection accepted IDs are not a batch prefix" - in - let partition () = - Result.bind - (consume_prefix owner.mutation_ids rejection.Sync_protocol.success_tx_ids) - (fun remaining -> - match rejection.failed_tx_id, remaining with - | Some failed, expected :: suffix when uuid_equal failed expected -> - Ok (Some failed, suffix) - | None, expected :: suffix when rejection.success_tx_ids = [] -> - Ok (Some expected, suffix) - | Some _, _ -> Error "rejection failed ID is not the next submitted member" - | None, _ -> Error "rejection omitted the failed submitted member") - in - let acceptance_barrier () = - match rejection.success_tx_ids with - | [] -> Ok None - | _ :: _ -> - (match rejection.t, rejection.checksum with - | Some through, Some checksum_value -> - Result.map - (fun checksum -> Some Overlay.{ through; checksum }) - (checksum checksum_value) - | _ -> Error "partial rejection omitted its acceptance barrier") - in - let reason = - match rejection.Sync_protocol.reason with - | Sync_protocol.Empty_tx_data | Invalid_tx | Invalid_t_before -> - Overlay.Invalid_request - | Db_transact_failed when rejection.missing_block_uuids <> [] -> - Overlay.Missing_dependencies - | Db_transact_failed | Snapshot_upload_in_progress -> Overlay.Operational_failure - | Stale -> Overlay.Operational_failure - in - let resolution = - match rejection.Sync_protocol.reason, rejection.t with - | Sync_protocol.Stale, Some through -> Ok (Overlay.Stale { through }) - | Stale, None -> Error "stale rejection omitted cursor" - | _, _ -> - Result.bind (partition ()) (fun (failed_member, unexecuted_suffix) -> - Result.map - (fun acceptance_barrier -> - Overlay.Definitive - { reason - ; partition = - { accepted_prefix = rejection.success_tx_ids - ; failed_member - ; unexecuted_suffix - ; acceptance_barrier - ; missing_uuids = rejection.missing_block_uuids - ; diagnostics = Option.to_list rejection.error_detail - } - }) - (acceptance_barrier ())) - in - (match resolution with - | Error message -> fail core During_catalog message - | Ok resolution -> - let core = note_remote_cursor core owner.connection.graph rejection.t in - let request = - { expected = Overlay.sync_view_token sync - ; key = core.graph_key - ; scope = owner.connection.graph - ; transition = Overlay.Reject_group { batch_id = owner.batch_id; resolution } - } - in - let next = - { core with - pending_outbox_transition = Some request - ; submission_owner = Some { owner with response_timer = None } - } - in - let next, feedback = - match resolution with - | Overlay.Stale _ -> next, [] - | Definitive _ -> - let next = - set_snapshot - next - { next.public_state.snapshot with - last_error_attention = Some Requires_attention - ; last_error = Some (rejection_message rejection) - } - in - next, [ publish next ] - in - { next; effects = feedback @ [ Delegate (Apply_outbox_transition request) ] }) -;; - -let websocket_message core connection message = - if (not (connection_is_current core connection)) || not core.websocket_live - then unchanged core - else ( - match message with - | Sync_protocol.Server.Pull_ok { t; checksum; txs } -> - let accepting = - Option.fold - ~none:(Option.is_none core.active_authoritative_batch) - ~some:(fun owner -> - owner.pull_connection = connection - && Server_cursor.compare t owner.issued_since >= 0 - && - match owner.pull_stage with - | Awaiting_pull_response _ -> true - | _ -> false) - core.pull_owner - in - if not accepting - then unchanged core - else ( - let core = note_pull_target core connection.graph (Some t) in - match authoritative_input core t checksum txs with - | Error message -> fail core During_catalog message - | Ok None -> - let completed = pull_no_progress core connection in - let next = - set_snapshot - completed.next - { core.public_state.snapshot with - sync_phase = settled_sync_phase completed.next - } - in - let planned = plan_submission next in - { next = planned.next - ; effects = (publish next :: completed.effects) @ planned.effects - } - | Ok (Some input) -> - let core = - { core with - pull_owner = - Option.map - (fun owner -> { owner with pull_stage = Applying_pull }) - core.pull_owner - } - in - enqueue_authoritative - core - { input - ; key = core.graph_key - ; scope = connection - ; presentation_generation = - core.public_state.snapshot.startup.presentation_generation - ; lifecycle_generation = core.lifecycle_generation - }) - | Tx_batch_ok { t; checksum } -> - (match core.submission_owner with - | Some owner when owner.connection = connection -> - apply_accept core owner t checksum - | _ -> unchanged core) - | Tx_reject rejection -> - (match core.submission_owner with - | Some owner when owner.connection = connection -> - apply_rejection core owner rejection - | _ -> unchanged core) - | Hello { t; _ } | Changed { t } -> request_pull ~through:t core - | Error { message } -> fail core During_catalog ("sync server error: " ^ message) - | Online_users _ | Presence _ | Pong -> unchanged core) -;; - -let authoritative_applied core (result : authoritative_commit_result) = - if - (not (graph_scope_is_current core result.scope)) - || Option.is_none core.active_authoritative_batch - then unchanged core - else ( - let applied_server_t = result.commit.checkpoint in - let receipt_mutation_id (receipt : Overlay.mutation_receipt) = - match receipt with - | Applied_receipt { mutation_id; _ } - | No_change_receipt { mutation_id; _ } - | Discarded_receipt { mutation_id; _ } -> mutation_id - | Remote_won_receipt receipt -> receipt.mutation_id - in - let owner = - match core.submission_owner with - | Some owner -> - let retains = - List.exists - (fun (receipt : Overlay.terminal_receipt) -> - match receipt.transport_disposition with - | Overlay.Retain_terminal_owner_until_response batch_id -> - Overlay.Submission_batch_id.equal batch_id owner.batch_id - | No_transport_owner | Clear_transport_owner -> false) - result.commit.terminal_receipts - in - let clears = - List.exists - (fun (receipt : Overlay.terminal_receipt) -> - receipt.transport_disposition = Overlay.Clear_transport_owner - && List.exists - (Logseq_db_types.Graph_types.Uuid.equal - (receipt_mutation_id receipt.receipt)) - owner.mutation_ids) - result.commit.terminal_receipts - in - if retains - then Some { owner with accepted_through = None } - else if clears - then None - else ( - match owner.accepted_through with - | Some through when Server_cursor.compare applied_server_t through >= 0 -> None - | _ -> Some owner) - | None -> None - in - let queued = core.queued_authoritative_batch in - let core = - set_snapshot - { core with - sync_view = Some result.sync - ; submission_owner = owner - ; pull_owner = None - ; pull_retry_delay = 1. - ; active_authoritative_batch = None - ; queued_authoritative_batch = None - ; deferred_authoritative_owner = None - } - { core.public_state.snapshot with - sync_phase = Current - ; applied_server_t = Some applied_server_t - ; last_error = - (if has_blocked_submissions result.sync - then core.public_state.snapshot.last_error - else None) - } - in - let core = - set_snapshot - core - { core.public_state.snapshot with sync_phase = settled_sync_phase core } - in - let planned = plan_submission core in - let next, queued_effects = - match queued with - | Some batch -> - ( { planned.next with active_authoritative_batch = Some batch } - , [ Delegate (Apply_authoritative_batch batch) ] ) - | None -> planned.next, [] - in - let pulled = request_pull next in - { pulled with - effects = (publish core :: planned.effects) @ queued_effects @ pulled.effects - }) -;; - -let authoritative_deferred core (result : authoritative_deferred_result) = - if - (not (graph_scope_is_current core result.scope)) - || Option.is_none core.active_authoritative_batch - then unchanged core - else ( - let (Overlay.Await_submission_outcome batch_id) = result.defer in - match core.submission_owner with - | Some owner when Overlay.Submission_batch_id.equal owner.batch_id batch_id -> - unchanged { core with deferred_authoritative_owner = Some batch_id } - | None -> - let durable = - Option.fold - ~none:false - ~some:(fun sync -> - List.exists - (fun (item : Overlay.submission_descriptor) -> - item.state = Overlay.Submitted batch_id) - (Overlay.sync_view_submissions sync)) - core.sync_view - in - if durable || Option.is_some core.pending_outbox_transition - then plan_submission { core with deferred_authoritative_owner = Some batch_id } - else fail core During_catalog "authoritative defer owner mismatch" - | Some _ -> fail core During_catalog "authoritative defer owner mismatch") -;; - -let pending_is_network (Pending_effect ticket) = - match ticket.kind with - | Fetch_catalog_kind | Fetch_snapshot_baseline_kind | Fetch_snapshot_metadata_kind - | Download_snapshot_kind | Fetch_e2ee_graph_key_kind | Fetch_e2ee_user_keys_kind -> true - | Load_catalog_kind | Save_catalog_kind | Load_and_unlock_graph_key_kind - | Fetch_and_unlock_graph_key_kind | Unlock_private_key_kind | Delete_account_secrets_kind - | Encrypt_protected_values_kind | Decrypt_protected_values_kind -> false -;; - -let consume_ticket : type a. t -> a effect_ticket -> t option = - fun core ticket -> - let rec loop kept = function - | [] -> None - | Pending_effect candidate :: rest when candidate.id = ticket.id -> - Some { core with pending_effects = List.rev_append kept rest } - | item :: rest -> loop (item :: kept) rest - in - loop [] core.pending_effects -;; - -let snapshot_ticket_is_current : type a. t -> a effect_ticket -> bool = - fun core ticket -> - let current scope = graph_scope_is_current core scope - && ticket.scope = effect_scope_of_graph scope in - match ticket.kind with - | Fetch_snapshot_metadata_kind -> - (match core.bootstrap with - | Awaiting_snapshot_metadata (scope, id) -> current scope && ticket.id = id - | _ -> false) - | Download_snapshot_kind -> - (match core.bootstrap with - | Downloading_snapshot request -> current request.scope && ticket.id = request.download_id - | _ -> false) - | Fetch_snapshot_baseline_kind -> - Option.fold ~none:false ~some:current (bootstrap_scope core.bootstrap) - | Fetch_e2ee_graph_key_kind -> - (match core.unlock_preparation with - | Fetching_graph_key (scope, id) -> current scope && id = ticket.id - | _ -> false) - | Fetch_e2ee_user_keys_kind -> - (match core.unlock_preparation with - | Fetching_user_keys (scope, _, id) -> current scope && id = ticket.id - | _ -> false) - | Unlock_private_key_kind | Fetch_and_unlock_graph_key_kind -> - (match core.public_state.snapshot.unlock with - | Unlock_running submission | Unlock_cancelled submission -> - current submission.challenge.challenge_scope - && (match submission.stage with - | Private_key id | Graph_key id -> id = ticket.id - | Attachment -> false) - | _ -> false) - | _ -> true -;; - -let graph_key_loaded core key = - match core.current_graph_scope with - | Some scope when graph_key_handle_scope key = scope -> - (match core.pending_graph_open with - | Some request when request.scope = scope -> - { next = - { core with - graph_key = Some key - ; pending_graph_open = None - ; pending_attachment = Some request - } - ; effects = [ Delegate (Attach_graph request) ] - } - | None -> - let core = { core with graph_key = Some key } in - begin_snapshot_bootstrap core scope - | Some _ -> unchanged core) - | None | Some _ -> - fail core During_e2ee "graph key handle is outside the selected graph scope" -;; - -let clear_selected_graph_for_catalog core catalog = - let previous_scope = core.current_graph_scope in - let startup = - { core.public_state.snapshot.startup with - catalog_loading = false - ; awaiting_selection = true - ; restoring_local = false - ; bootstrapping = false - ; awaiting_e2ee_password = false - ; failure = None - ; graph_generation = core.public_state.snapshot.startup.graph_generation + 1 - } - in - let snapshot = - { local_deletion = None - ; sync_phase = Offline - ; catalog - ; resident_graphs = [] - ; selected_graph = None - ; applied_server_t = None - ; timeline_presentation_pending = false - ; unlock = Unlock_inactive - ; startup - ; last_error = None - ; last_error_attention = None - } - in - let next = - set_snapshot - { (reset_graph_operations core) with - pending_effects = [] - ; cached_selected_graph = None - ; selected_graph_value = None - ; current_graph_scope = None - ; pending_selected_download = None - ; pending_mirror_inspection = None - ; websocket_live = false - } - snapshot - in - let teardown = - Option.fold - ~none:[] - ~some:(fun scope -> - [ Run (Cancel_effects (effect_scope_of_graph scope)) - ; Delegate (Detach_graph scope) - ]) - previous_scope - in - { next; effects = teardown @ [ publish next ] } -;; - -let catalog_refreshed core catalog = - match core.public_state.snapshot.selected_graph with - | Some selected_graph -> - (match graph_in_catalog catalog selected_graph with - | None -> clear_selected_graph_for_catalog core catalog - | Some graph -> - let snapshot = clear_reconciliation_failure core.public_state.snapshot in - let startup = - { snapshot.startup with catalog_loading = false; awaiting_selection = false } - in - let next = - set_snapshot - { core with - cached_selected_graph = Some selected_graph - ; selected_graph_value = Some graph - } - { snapshot with catalog; startup } - in - { next; effects = [ publish next ] }) - | None -> - let cached_selected_graph = - Option.bind core.cached_selected_graph (fun graph_id -> - Option.map (fun _ -> graph_id) (graph_in_catalog catalog graph_id)) - in - let startup = - { core.public_state.snapshot.startup with - catalog_loading = false - ; awaiting_selection = true - ; restoring_local = false - ; failure = None - } - in - let next = - set_snapshot - { core with cached_selected_graph } - { core.public_state.snapshot with - catalog - ; selected_graph = None - ; startup - ; last_error = None - } - in - { next; effects = [ publish next ] } -;; - -let apply_fetched_catalog core catalog = - let refreshed = catalog_refreshed core catalog in - let next, save = save_cache refreshed.next in - { next; effects = refreshed.effects @ save } -;; - -let complete_catalog_load core cache = - let catalog, cached_selected_graph = - match core.user_id, cache with - | Some user_id, Some cache when String.equal user_id cache.cache_user_id -> - let catalog = cache.cache_graphs in - let cached_selected_graph = - Option.bind cache.cache_selected_graph (fun graph_id -> - Option.map (fun _ -> graph_id) (graph_in_catalog catalog graph_id)) - in - catalog, cached_selected_graph - | None, _ | Some _, None | Some _, Some _ -> [], None - in - let startup = - { core.public_state.snapshot.startup with - restoring_local = false - ; awaiting_selection = true - } - in - let loaded = - set_snapshot - { core with catalog_cache_loading = false; cached_selected_graph - ; restored_residents = Option.bind cache (fun cache -> cache.cache_resident_graphs) } - { core.public_state.snapshot with catalog; selected_graph = None; startup } - in - let restored = - match cached_selected_graph with - | Some graph_id -> select_graph_transition loaded ~persist:false graph_id - | None -> { next = loaded; effects = [ publish loaded ] } - in - if restored.next.deferred_catalog_reconciliation - then ( - let requested = request_catalog_reconciliation restored.next in - { next = requested.next; effects = restored.effects @ requested.effects }) - else if catalog=[] then - local_unavailable restored.next Local_catalog_unavailable "No local graph catalog is available. Refresh graphs to connect." - else restored -;; - -let deletion_failed ?source core stage = - let core = match source with - | None -> core - | Some source -> - let operation, reason = match stage with - | Closing_graph -> "closeDatabase", "closeFailed" - | Deleting_mirror -> "deleteMirror", "deleteFailed" - | Clearing_selection -> "saveCatalog", "persistenceFailed" in - let diagnostic = Logseq_db_types.Failure_diagnostic.of_source - ~operation ~reason source in - let groups = core.public_state.diagnostics.groups in - let entries = List.concat_map (fun group -> - if group.title = "Sync errors" then group.entries else []) groups in - let group = { title = "Sync errors"; entries = - (operation, Logseq_db_types.Failure_diagnostic.message diagnostic) - :: List.remove_assoc operation entries } in - { core with public_state = { core.public_state with diagnostics = - { groups = group :: List.filter (fun group -> group.title <> "Sync errors") groups } } } - in - let message = - match stage with - | Closing_graph -> "Local graph close failed." - | Deleting_mirror -> "Local graph deletion failed. Try again." - | Clearing_selection -> "Clearing the saved graph selection failed." - in - let next = - set_snapshot - { core with deletion_scope = None } - { core.public_state.snapshot with - local_deletion = None - ; last_error_attention = Some Requires_attention - ; last_error = Some message - } - in - { next; effects = [ publish next ] } -;; - -let deletion_selection_saved core = - match core.public_state.snapshot.local_deletion with - | Some (Deletion_in_progress Clearing_selection) -> - let snapshot = core.public_state.snapshot in - let next = - set_snapshot - { core with deletion_scope = None } - { snapshot with - local_deletion = None - ; startup = { snapshot.startup with awaiting_selection = true } - } - in - { next; effects = [ publish next ] } - | None | Some _ -> unchanged core -;; - -let complete_runner : type a. t -> a effect_ticket -> a -> transition = - fun core ticket value -> - match ticket.kind with - | Load_catalog_kind -> complete_catalog_load core value - | Save_catalog_kind -> deletion_selection_saved core - | Fetch_catalog_kind -> apply_fetched_catalog core value - | Fetch_snapshot_baseline_kind -> - (match decode_snapshot_baseline value, bootstrap_scope core.bootstrap with - | Ok _, Some scope when graph_scope_is_current core scope -> - begin_snapshot_bootstrap core scope - | Error message, _ -> fail core During_bootstrap message - | _ -> unchanged core) - | Fetch_snapshot_metadata_kind -> - (match decode_snapshot_uri value, core.bootstrap with - | Ok (uri, metadata_cursor), Awaiting_snapshot_metadata (scope, attempt_id) -> - let core = note_remote_cursor core scope (Some metadata_cursor) in - let request = - { scope; uri; metadata_cursor; attempt_id; download_id = core.next_effect_id - ; expected_bytes = None - ; maximum_bytes = core.config.limits.maximum_artifact_bytes } - in - let next, request_effect = issue_request core (Download_snapshot request) in - (* Metadata is a lower bound for the live stream, never its maximum t. *) - { next = { next with bootstrap = Downloading_snapshot request } - ; effects = [ Run request_effect ] } - | Error message, _ -> fail core During_bootstrap message - | _ -> unchanged core) - | Download_snapshot_kind -> - (match core.bootstrap, core.selected_graph_value with - | Downloading_snapshot request, Some graph - when graph_scope_is_current core request.scope - && value.download.attempt_id = request.attempt_id - && value.download = request -> - let key = if graph.encrypted then core.graph_key else None in - (match key with - | None when graph.encrypted -> - fail core During_bootstrap "encrypted snapshot requires a graph key handle" - | Some key when graph_key_handle_scope key <> request.scope -> - fail core During_bootstrap "graph key handle is outside the selected graph scope" - | _ -> - let activation = - { artifact = value; scope = request.scope - ; applied_server_t = request.metadata_cursor; key } - in - { next = { core with bootstrap = Activating_snapshot activation } - ; effects = [ Delegate (Activate_snapshot activation) ] }) - | Downloading_snapshot _, _ -> - fail core During_bootstrap "snapshot artifact is outside the active attempt" - | _ -> unchanged core) - | Load_and_unlock_graph_key_kind -> graph_key_loaded core value - | Fetch_e2ee_graph_key_kind -> - (match core.unlock_preparation with - | Fetching_graph_key (scope, _) -> - let core = { core with unlock_preparation = - Fetching_user_keys (scope, value, core.next_effect_id) } in - let next, request = issue_request core (Fetch_e2ee_user_keys scope.account) in - { next; effects = [ Run request ] } - | _ -> unchanged core) - | Fetch_e2ee_user_keys_kind -> - (match core.unlock_preparation with - | Fetching_user_keys (scope, encrypted_graph_key, _) -> - let challenge = { challenge_scope = scope; challenge_id = ticket.id - ; encrypted_graph_key; private_key_package = value } in - let next = set_snapshot { core with unlock_preparation = No_unlock_preparation } - { core.public_state.snapshot with unlock = Unlock_challenge challenge; last_error = None - ; startup = { core.public_state.snapshot.startup with failure = None } } in - { next; effects = [ publish next ] } - | _ -> unchanged core) - | Unlock_private_key_kind -> - (match core.public_state.snapshot.unlock with - | (Unlock_running submission | Unlock_cancelled submission) as phase -> - let scope = submission.challenge.challenge_scope in - let submission = { submission with stage = Graph_key core.next_effect_id } in - let unlock = match phase with Unlock_cancelled _ -> Unlock_cancelled submission - | _ -> Unlock_running submission in - let core = set_snapshot core { core.public_state.snapshot with unlock } in - let next, request = issue_request core (Fetch_and_unlock_graph_key - { scope; encrypted_graph_key = submission.challenge.encrypted_graph_key }) in - { next; effects = [ Run request ] } - | _ -> unchanged core) - | Fetch_and_unlock_graph_key_kind -> - let unlock = match core.public_state.snapshot.unlock with - | Unlock_running submission -> Unlock_running { submission with stage = Attachment } - | Unlock_cancelled submission -> Unlock_cancelled { submission with stage = Attachment } - | phase -> phase in - let core = set_snapshot core { core.public_state.snapshot with unlock } in - graph_key_loaded core value - | Delete_account_secrets_kind -> unchanged core - | Encrypt_protected_values_kind -> unchanged core - | Decrypt_protected_values_kind -> unchanged core -;; - -let consume_completion core = function - | Asset_completion _ | Protected_completion _ -> unchanged core - | Completion (ticket, result) -> - (match consume_ticket core ticket with - | None -> unchanged core - | Some core when not (snapshot_ticket_is_current core ticket) -> unchanged core - | Some core -> - (match result with - | Ok value -> complete_runner core ticket value - | Error (Effect_failed message | Crypto_failed (_, message)) -> - let source = message in - let message = match Logseq_db_types.Failure_diagnostic.decode source with - | Some diagnostic -> Logseq_db_types.Failure_diagnostic.message diagnostic - | None -> message in - (match ticket.kind with - | Save_catalog_kind -> - (* Keep the existing advisory/deletion outcome. Only the safe - diagnostics projection retains the producer's specific cause. *) - let diagnostic = Logseq_db_types.Failure_diagnostic.of_source - ~operation:"saveCatalog" ~reason:"persistenceFailed" source in - let group = - { title = "Sync errors" - ; entries = - [ "saveCatalog", - "saveCatalog · phase=catalog · code=catalog.save.failed · " ^ - "Local graph selection could not be saved. Retry by " ^ - "refreshing or selecting the graph again. " ^ - Logseq_db_types.Failure_diagnostic.message diagnostic - ] - } in - let next = { core with public_state = - { core.public_state with diagnostics = - { groups = group :: List.filter - (fun g -> g.title <> group.title) core.public_state.diagnostics.groups } } } in - (match core.public_state.snapshot.local_deletion with - | Some (Deletion_in_progress Clearing_selection) -> - deletion_failed next Clearing_selection - | None | Some _ -> { next; effects = [ publish next ] }) - | Load_catalog_kind -> - fail - { core with - catalog_cache_loading = false - ; deferred_catalog_reconciliation = false - } - During_catalog - message - | Load_and_unlock_graph_key_kind -> - (match core.current_graph_scope with - | Some scope when network_allowed core -> begin_e2ee_key_access core scope - | Some _ -> local_unavailable core Local_key_unavailable "The local encryption key is unavailable. Connect to recover access." - | None -> unchanged core) - | Unlock_private_key_kind when core.public_state.snapshot.startup.network_phase = Local_only -> - fail core During_e2ee message - | Unlock_private_key_kind -> - (match core.public_state.snapshot.unlock with - | Unlock_running submission -> - let failed = fail core During_e2ee message in - let next = set_snapshot { failed.next with bootstrap = core.bootstrap } - { failed.next.public_state.snapshot with - unlock = Unlock_failed (Retry_password (submission.challenge, message)) } in - { next; effects = [ publish next ] } - | _ -> fail core During_e2ee message) - | Fetch_e2ee_graph_key_kind - | Fetch_e2ee_user_keys_kind - | Fetch_and_unlock_graph_key_kind -> fail core During_e2ee message - | Delete_account_secrets_kind -> - cleanup_failed core "account secret cleanup failed" - | Fetch_snapshot_baseline_kind - | Fetch_snapshot_metadata_kind - | Download_snapshot_kind -> fail core During_bootstrap message - | _ -> fail core During_catalog message))) -;; - -let websocket_closed core connection termination = - let message, last_error_attention = - match termination with - | Recoverable_close { message; attention } -> message, Option.map (fun _ -> attention) message - | Local_close -> None, None - | Authentication_failure message | Protocol_failure message -> Some message, Some Requires_attention - in - if not (connection_is_current core connection) - then unchanged core - else ( - let next = - set_snapshot - { core with - websocket_live = false - ; pull_owner = None - ; submission_owner = None - ; pull_retry_delay = 1. - ; active_authoritative_batch = None - ; queued_authoritative_batch = None - ; deferred_authoritative_owner = None - } - { core.public_state.snapshot with sync_phase = Offline; last_error = message; last_error_attention } - in - { next; effects = [ publish next ] }) -;; - -let return_to_graph_picker core = - let detach = - Option.fold - ~none:[] - ~some:(fun scope -> [ Delegate (Detach_graph scope) ]) - core.current_graph_scope - in - let startup = - { core.public_state.snapshot.startup with - awaiting_selection = true - ; catalog_loading = false - ; restoring_local = false - ; bootstrapping = false - ; awaiting_e2ee_password = false - ; failure = None - ; graph_generation = core.public_state.snapshot.startup.graph_generation + 1 - ; presentation_generation = - core.public_state.snapshot.startup.presentation_generation + 1 - } - in - let next = - set_snapshot - { (reset_graph_operations core) with - selected_graph_value = None - ; current_graph_scope = None - ; pending_selected_download = None - ; pending_mirror_inspection = None - ; pending_effects = [] - ; websocket_live = false - } - { core.public_state.snapshot with - sync_phase = Offline - ; selected_graph = None - ; applied_server_t = None - ; timeline_presentation_pending = false - ; startup - ; last_error = None - } - in - let cancel = - Option.fold - ~none:[] - ~some:(fun scope -> [ Run (Cancel_effects (effect_scope_of_graph scope)) ]) - core.current_graph_scope - in - { next; effects = cancel @ detach @ [ publish next ] } -;; - -let scoped_error_is_current core (error : scoped_error) = - Option.fold - ~none:false - ~some:(fun (scope : graph_scope) -> effect_scope_of_graph scope = error.scope) - core.current_graph_scope -;; - -let inspect_sync core scope = - match core.pending_sync_inspection with - | None -> - { next = { core with pending_sync_inspection = Some scope } - ; effects = [ Delegate (Inspect_sync scope) ] - } - | Some _ -> unchanged core -;; - -let begin_local_deletion core graph_id = - let snapshot = core.public_state.snapshot in - match core.user_id, graph_in_catalog snapshot.catalog graph_id with - | Some user_id, Some _ - when snapshot.startup.authenticated - && snapshot.selected_graph <> Some graph_id - && snapshot.local_deletion = None - && not snapshot.startup.restoring_local - && not snapshot.startup.bootstrapping - && not snapshot.startup.awaiting_e2ee_password - && core.pending_attachment = None - && not (bootstrap_downloading core.bootstrap) -> - let account = account_scope core user_id in - (* A private flight identity leaves the live graph's generations untouched. *) - let account = { account with lifecycle_generation = Int64.of_int core.next_effect_id } in - let scope = { account; graph_id; graph_generation = snapshot.startup.graph_generation } in - let next = set_snapshot - { core with deletion_scope = Some scope - ; next_effect_id = core.next_effect_id + 1 - } - { snapshot with local_deletion = Some (Deletion_in_progress Deleting_mirror); last_error = None } in - { next; effects = [publish next; Delegate (Delete_mirror - { account; graph_id; scope = effect_scope_of_graph scope })] } - | _ -> unchanged core -;; - -let graph_detached core scope result = - match core.deletion_scope, core.public_state.snapshot.local_deletion with - | Some captured, Some (Deletion_in_progress Closing_graph) when captured = scope -> - (match result with - | Error source -> deletion_failed ~source core Closing_graph - | Ok () -> - let next = - set_snapshot - core - { core.public_state.snapshot with - local_deletion = Some (Deletion_in_progress Deleting_mirror) - } - in - { next - ; effects = - [ publish next - ; Delegate - (Delete_mirror - { account = scope.account - ; graph_id = scope.graph_id - ; scope = effect_scope_of_graph scope - }) - ] - }) - | _ -> unchanged core -;; - -let mirror_deleted core (request : mirror_deletion) result = - match core.deletion_scope, core.public_state.snapshot.local_deletion with - | Some scope, Some (Deletion_in_progress Deleting_mirror) - when request.account = scope.account && request.graph_id = scope.graph_id - && request.scope = effect_scope_of_graph scope -> - (match result with - | Error message -> - let failed = deletion_failed ~source:message core Deleting_mirror in - if List.mem message - [ "Switch to another graph before deleting." - ; "The local copy is still in use. Close attachment previews and try again." ] - then - let next = set_snapshot failed.next { failed.next.public_state.snapshot with last_error = Some message; last_error_attention = Some Requires_attention } in - { next; effects = [publish next] } - else failed - | Ok () -> - let next = set_snapshot { core with deletion_scope = None } - { core.public_state.snapshot with local_deletion = None; last_error = None } in - { next; effects = [publish next] }) - | _ -> unchanged core -;; - -let deletion_blocks_event core event = - match core.public_state.snapshot.local_deletion, event with - | Some (Deletion_in_progress _), - (Local_cache_deletion_requested _ | Graph_selected _ | Graph_picker_requested) -> true - | _ -> false -;; - -let step core event = - if core.closed || deletion_blocks_event core event - then unchanged core - else ( - match event with - | Host_fence_started _ | Host_fence_released _ - | IO_fence_requested _ | IO_fence_completed _ -> unchanged core - | Sign_in_requested scope when startup_scope_is_current core scope -> - let next=network_phase core Online_by_user in {next;effects=[publish next]} - | Online_cancelled scope when startup_scope_is_current core scope -> - let next = network_phase core Local_only in - let snapshot = next.public_state.snapshot in - let local_pending = List.filter (fun pending -> not (pending_is_network pending)) next.pending_effects in - let local_restore_pending = - next.catalog_cache_loading - || Option.is_some next.pending_mirror_inspection - || Option.is_some next.pending_attachment - || Option.is_some next.pending_sync_inspection - || List.exists (fun (Pending_effect ticket) -> match ticket.kind with - | Load_and_unlock_graph_key_kind | Fetch_and_unlock_graph_key_kind - | Unlock_private_key_kind -> true | _ -> false) local_pending - || bootstrap_activating next.bootstrap - in - let local_access = match snapshot.startup.local_access with - | Local_pending when not local_restore_pending -> - Local_unavailable (if Option.is_some next.pending_graph_open - then Local_key_unavailable else if Option.is_some (bootstrap_scope next.bootstrap) - then Local_mirror_unavailable else Local_restore_failed) - | access -> access - in - (* Presentation tickets retire independently of the retained DB/key scope. *) - let startup = { snapshot.startup with - authenticated = false; - presentation_generation = snapshot.startup.presentation_generation + 1; - restoring_local = snapshot.startup.restoring_local && local_restore_pending; - bootstrapping = false; catalog_loading = next.catalog_cache_loading; - awaiting_e2ee_password = false; local_access; - failure = (if Option.is_some next.pending_selected_download then Some During_local_restore else snapshot.startup.failure) } in - let next = set_snapshot { next with - pending_selected_download = None; - pending_effects = local_pending; websocket_live = false; deferred_catalog_reconciliation = false; - unlock_preparation = No_unlock_preparation; - bootstrap = (if bootstrap_downloading next.bootstrap then No_bootstrap else next.bootstrap); pull_wanted = None; force_pull_pending = false; - pull_owner = (match next.pull_owner with - | Some {pull_stage=Applying_pull; _} as owner -> owner | _ -> None) - } { snapshot with timeline_presentation_pending = false; startup; sync_phase = Offline - ; unlock = (match snapshot.unlock with - | Unlock_running submission | Unlock_bootstrapping submission -> Unlock_cancelled submission - | Unlock_cancelled _ as phase -> phase - | _ -> Unlock_inactive) } in - let scope = { account_generation = Some snapshot.startup.account_generation; - graph_generation = None; connection_generation = None; - presentation_generation = None; lifecycle_generation = None } in - { next; effects = [Run (Cancel_network_effects scope); publish next] } - | Local_account_unavailable (scope,reason) when startup_scope_is_current core scope -> - local_unavailable core reason "No local account is available. Sign in to connect." - | Sign_in_requested _ | Online_cancelled _ | Local_account_unavailable _ -> unchanged core - | Restore_local_account { user_id } -> restore_local core user_id - | Account_authenticated { user_id = Some user_id } -> - if core.public_state.snapshot.startup.network_phase=Local_only then unchanged core else - let authenticated=authenticate core user_id in - let next=authenticated.next in - let resumed = match next.current_graph_scope,next.sync_view,core.public_state.snapshot.startup.local_access with - | Some scope,None,_ when next.pending_selected_download = Some scope - && next.public_state.snapshot.startup.bootstrapping -> - begin_missing_mirror_bootstrap next scope - | Some scope,Some _,_ when not next.websocket_live && (match next.websocket_owner with - | None | Some {phase=Retiring;_} -> true | Some _ -> false) -> start_websocket next scope - | Some scope,None,Local_unavailable Local_key_unavailable - when core.public_state.snapshot.startup.failure=Some During_local_restore -> - begin_e2ee_key_access next scope - | Some scope,None,Local_unavailable _ - when next.public_state.snapshot.startup.network_phase=Online_by_user - && core.public_state.snapshot.startup.failure=Some During_local_restore -> - select_graph_transition next ~persist:false scope.graph_id - | _ -> unchanged next in - {next=resumed.next;effects=authenticated.effects @ resumed.effects} - | Account_authenticated { user_id = None } -> sign_out core - | Local_outbox_changed when not (network_allowed core) -> unchanged core - | Local_outbox_changed -> - (match core.current_graph_scope with - | Some scope -> inspect_sync core scope - | None -> unchanged core) - | Runner_completed completion -> consume_completion core completion - | Graph_selected graph_id -> select_graph core graph_id - | Graph_detached (scope, result) -> graph_detached core scope result - | Mirror_deleted (request, result) -> mirror_deleted core request result - | Mirror_inspected inspection -> mirror_inspected core inspection - | Graph_attached attachment -> graph_attached core attachment - | Sync_inspected inspection -> - if - graph_scope_is_current core inspection.scope - && core.pending_sync_inspection = Some inspection.scope - then - plan_submission - { core with pending_sync_inspection = None; sync_view = Some inspection.sync } - else unchanged core - | Outbox_transition_applied result -> outbox_transition_applied core result - | Websocket_opened connection -> websocket_opened core connection - | Websocket_message (connection, message) -> websocket_message core connection message - | Websocket_protocol_error (connection, error) - when connection_is_current core connection -> - fail core During_catalog (Sync_protocol.error_to_string error) - | Websocket_protocol_error _ -> unchanged core - | Websocket_closed (connection, message) -> websocket_closed core connection message - | Websocket_retired _ -> unchanged core - | Authoritative_batch_applied result -> authoritative_applied core result - | Authoritative_batch_deferred result -> authoritative_deferred core result - | Snapshot_activated activation -> - (match core.bootstrap, core.selected_graph_value with - | Activating_snapshot pending, Some graph - when pending = activation && graph_scope_is_current core activation.scope -> - let request = { graph; scope = activation.scope } in - { next = { core with pending_mirror_inspection = Some request; bootstrap = Preparing_snapshot activation.scope } - ; effects = [ Delegate (Inspect_mirror request) ] - } - | _ -> unchanged core) - | Snapshot_download_progress progress -> - (match core.bootstrap with - | Downloading_snapshot { scope; _ } - when graph_scope_is_current core scope - && Logseq_db_types.Graph_types.Uuid.equal scope.graph_id progress.graph_id - -> { next = core; effects = [ Publish (Bootstrap_progressed progress) ] } - | _ -> unchanged core) - | Graph_picker_requested -> return_to_graph_picker core - | Catalog_refresh_requested scope when not (startup_scope_is_current core scope) -> unchanged core - | Catalog_refresh_requested _ -> - let needs_permission = core.public_state.snapshot.startup.network_phase = Local_only in - let core=network_phase core Online_by_user in - let core = if needs_permission then - let snapshot=core.public_state.snapshot in - set_snapshot core {snapshot with startup={snapshot.startup with authenticated=false}} - else core in - (* Publishing opens the native gate; its verified identity round-trip owns - catalog admission when this request resumes a local-only session. *) - if needs_permission || not core.public_state.snapshot.startup.authenticated - then {next=core;effects=[publish core]} else - (match core.user_id with - | None -> unchanged core - | Some user_id -> - let next, request = - issue_request core (Fetch_catalog (account_scope core user_id)) - in - { next; effects = [ Run request ] }) - | Online_recovery_requested scope when not (timeline_scope_is_current core scope) -> unchanged core - | Online_recovery_requested _ -> - let core=network_phase core Online_by_user in - if not core.public_state.snapshot.startup.authenticated then {next=core;effects=[publish core]} else - (match core.current_graph_scope, core.public_state.snapshot.startup.failure with - | Some scope, Some During_bootstrap when core.sync_view = None -> - select_graph_transition core ~persist:false scope.graph_id - | Some _, _ - when core.public_state.snapshot.startup.awaiting_e2ee_password - || List.exists is_graph_key_access core.pending_effects -> unchanged core - | Some scope, Some During_local_restore - when core.public_state.snapshot.startup.local_access <> Local_unavailable Local_key_unavailable -> - select_graph_transition core ~persist:false scope.graph_id - | Some scope, Some (During_local_restore | During_e2ee) - when not core.public_state.snapshot.startup.awaiting_e2ee_password -> - let startup = - { core.public_state.snapshot.startup with - awaiting_e2ee_password = false - ; failure = None - } - in - let core = - set_snapshot - core - { core.public_state.snapshot with - sync_phase = Offline - ; startup - ; last_error = None - } - in - let requested = begin_e2ee_key_access core scope in - { requested with effects = publish requested.next :: requested.effects } - | Some scope, _ -> - (* The existing explicit-online entry owns manual freshness demand. - Busy work keeps a single intent until its response/commit retires. *) - let inspected = inspect_sync { core with force_pull_pending = true } scope in - let pulled = request_pull inspected.next in - { pulled with effects = inspected.effects @ pulled.effects } - | None, _ -> unchanged core) - | Local_cache_deletion_requested graph_id -> begin_local_deletion core graph_id - | Foreground_changed { foreground = false; lifecycle_generation } -> - let core = - { core with - lifecycle_generation - ; websocket_live = false - ; pull_owner = None - ; submission_owner = None - ; pull_retry_delay = 1. - ; active_authoritative_batch = None - ; queued_authoritative_batch = None - ; deferred_authoritative_owner = None - } - in - (match core.current_graph_scope with - | None -> unchanged core - | Some graph -> - { next = core - ; effects = - [ Run - (Close_websocket - { graph; connection_generation = core.connection_generation }) - ] - }) - | Foreground_changed { foreground = true; lifecycle_generation } -> - let core = { core with lifecycle_generation } in - (match core.current_graph_scope with - | None -> unchanged core - | Some graph -> start_websocket core graph) - | Timeline_presented scope -> - if not (timeline_scope_is_current core scope) - || core.public_state.snapshot.startup.local_access<>Local_available - || not core.public_state.snapshot.timeline_presentation_pending - then unchanged core else - let core=network_phase core Online_after_timeline in - let next=set_snapshot core {core.public_state.snapshot with timeline_presentation_pending=false} in - {next;effects=[publish next]} - | Snapshot_activation_failed error -> - (match core.bootstrap with - | Activating_snapshot pending - when pending = error.activation && graph_scope_is_current core pending.scope -> - fail core During_bootstrap error.message - | _ -> unchanged core) - | Graph_attachment_failed error - | Authoritative_batch_failed error - when scoped_error_is_current core error -> - fail core (if core.public_state.snapshot.startup.restoring_local then During_local_restore else During_bootstrap) error.message - | Graph_attachment_failed _ - | Authoritative_batch_failed _ -> unchanged core - | E2ee_password_submitted (challenge, password) -> - (match unlock_challenge core.public_state.snapshot.unlock with - | Some current when equal_unlock_challenge current challenge - && graph_scope_is_current core challenge.challenge_scope -> - let submission = { challenge; operation_id = core.next_effect_id - ; stage = Private_key core.next_effect_id } in - let core = set_snapshot core - { core.public_state.snapshot with unlock = Unlock_running submission - ; sync_phase = Offline; last_error = None - ; startup = { core.public_state.snapshot.startup with failure = None } } in - let next, request = issue_request core (Unlock_private_key - { scope = challenge.challenge_scope.account; password - ; private_key_package = challenge.private_key_package }) in - { next; effects = [ publish next; Run request ] } - | _ -> unchanged core) - | Outbox_transition_superseded {request;sync} -> - (match core.pending_outbox_transition with - | Some pending when graph_scope_is_current core request.scope - && pending.scope=request.scope && pending.transition=request.transition - && Overlay.sync_token_equal pending.expected request.expected - && not(Overlay.sync_token_equal request.expected(Overlay.sync_view_token sync))-> - (* An independently completed inspection may already have replaced the - preparation's view. Tokens are opaque: retain that observation - rather than guessing that the superseded result is newer. *) - let sync=match core.sync_view with - | Some observed when not(Overlay.sync_token_equal request.expected(Overlay.sync_view_token observed))->observed - | Some _ | None -> sync in - let core={core with sync_view=Some sync;pending_outbox_transition=None} in - (match request.transition with - | Overlay.Submit_group _ | Retry_group _ -> plan_submission core - | Accept_group _ | Reject_group _ -> - (* Keep the exact server outcome. Replanning submission here could - discard an accepted prefix or repeat already executed work. *) - let request={request with expected=Overlay.sync_view_token sync} in - {next={core with pending_outbox_transition=Some request}; - effects=[Delegate(Apply_outbox_transition request)]}) - | Some _ | None -> unchanged core) - | Outbox_transition_failed { request; message } -> - (match core.pending_outbox_transition with - | Some pending - when graph_scope_is_current core request.scope - && pending.scope = request.scope - && pending.transition = request.transition - && Overlay.sync_token_equal pending.expected request.expected -> - let close = - match core.current_graph_scope with - | Some graph when core.websocket_live -> - [ Run - (Close_websocket - { graph; connection_generation = core.connection_generation }) - ] - | Some _ | None -> [] - in - let failed = - fail - { core with - websocket_live = false - ; submission_owner = None - ; connection_generation = core.connection_generation + 1 - } - During_catalog - message - in - { failed with effects = close @ failed.effects } - | Some _ | None -> unchanged core) - | Timer_elapsed id -> - let restart (connection : connection_scope) = - let restarted = start_websocket core connection.graph in - { restarted with effects = Run (Close_websocket connection) :: restarted.effects } - in - let submission_timeout () = - match core.submission_owner with - | Some owner - when owner.response_timer = Some id - && connection_is_current core owner.connection - && core.websocket_live -> restart owner.connection - | _ -> unchanged core - in - (match core.pull_owner with - | Some owner - when core.websocket_live && connection_is_current core owner.pull_connection -> - (match owner.pull_stage with - | Awaiting_pull_response timer when timer = id -> restart owner.pull_connection - | Awaiting_pull_retry timer when timer = id -> - request_pull { core with pull_owner = None } - | _ -> submission_timeout ()) - | _ -> submission_timeout ()) - | Asset_requested _ | Protected_requested _ -> unchanged core - | Resident_graphs_changed _ | Visible_graph_changed _ | Local_graph_outbox_changed _ - | Local_feed_acknowledged -> unchanged core - | Shutdown -> - let scope = - { account_generation = None - ; graph_generation = None - ; connection_generation = None - ; presentation_generation = None - ; lifecycle_generation = None - } - in - { next = - { core with - closed = true - ; pending_effects = [] - ; pending_graph_open = None - ; pending_mirror_inspection = None - ; pending_attachment = None - ; bootstrap = No_bootstrap - ; pending_sync_inspection = None - ; pending_outbox_transition = None - ; submission_owner = None - ; pull_owner = None - ; pull_wanted = None; force_pull_pending = false - ; pull_retry_delay = 1. - ; active_authoritative_batch = None - ; queued_authoritative_batch = None - ; deferred_authoritative_owner = None - } - ; effects = [ Run (Cancel_effects scope) ] - }) -;; - -(* Asset execution is a separate identity/acceptance owner. Download demand and - durable import publication remain in their respective policy reducers. *) -let asset_public_request (request : asset_io_request) : asset_request = - { scope = request.context.scope - ; operation = request.operation - ; action = request.action - } -;; - -let asset_output request result = Publish (Asset_finished { request; result }) - -let issue_asset ?(notify = true) core (request : asset_io_request) = - let ticket = { asset_serial = core.next_effect_id; asset_request = request } in - { next = - { core with - next_effect_id = core.next_effect_id + 1 - ; assets_pending = - { asset_ticket = ticket; asset_cancelled = false; asset_notify = notify } - :: core.assets_pending - } - ; effects = [ Run (Asset_io (ticket, request)) ] - } -;; - -let asset_cleanup_action = function - | Release_asset_file _ - | Release_staged_file _ - | Close_asset_scope - | Await_asset_operations - | Prepare_graph_asset_deletion _ - | Commit_graph_asset_deletion _ - | Cancel_graph_asset_deletion _ - | Delete_account_assets - | Cancel_asset_operation _ -> true - | Check_asset_cache _ - | Fetch_asset _ - | Stage_asset_file _ - | Put_asset_file _ - | Publish_staged_asset _ - | Retain_asset_file _ - | Retain_staged_file _ - | Prune_asset_staging _ - | Asset_retry_after _ -> false -;; - -let asset_cleanup core (request : asset_io_request) result = - let action = - match result with - | Ok (Asset_cached (Some handle) | Asset_downloaded handle) -> - Some (Release_asset_file handle) - | Ok (Asset_retained (Some (lease, _))) -> Some (Release_asset_file lease) - | Ok (Asset_staged { file; _ }) -> Some (Release_staged_file file) - | Ok (Asset_deletion_prepared deletion) -> Some (Cancel_graph_asset_deletion deletion) - | Ok - ( Asset_cached None - | Asset_retained None - | Asset_unit | Asset_pruned _ | Asset_retry_elapsed ) - | Error _ -> None - in - match action with - | None -> unchanged core - | Some action -> - issue_asset - ~notify:false - core - { request with operation = "cleanup:" ^ string_of_int core.next_effect_id; action } -;; - -let asset_result_matches (request : asset_io_request) = function - | Error _ -> true - | Ok value -> - (match request.action, value with - | Check_asset_cache _, Asset_cached _ - | Fetch_asset _, Asset_downloaded _ - | Stage_asset_file _, Asset_staged _ - | Put_asset_file _, Asset_unit - | Publish_staged_asset _, Asset_unit - | Retain_asset_file _, Asset_retained _ - | Retain_staged_file _, Asset_retained _ - | Release_asset_file _, Asset_unit - | Release_staged_file _, Asset_unit - | Prune_asset_staging _, Asset_pruned _ - | Close_asset_scope, Asset_unit - | Await_asset_operations, Asset_unit - -> true - | Prepare_graph_asset_deletion retirement, Asset_deletion_prepared deletion -> - retirement = deletion.retirement && request.context.scope = deletion.deletion_scope - | Commit_graph_asset_deletion _, Asset_unit - | Cancel_graph_asset_deletion _, Asset_unit - | Delete_account_assets, Asset_unit - | Asset_retry_after _, Asset_retry_elapsed - | Cancel_asset_operation _, Asset_unit -> true - | _ -> false) -;; - -let complete_asset core ticket result = - match - List.find_opt (fun pending -> pending.asset_ticket = ticket) core.assets_pending - with - | None -> unchanged core - | Some pending -> - let core = - { core with - assets_pending = - List.filter (fun item -> item.asset_ticket <> ticket) core.assets_pending - } - in - let request = ticket.asset_request in - let core = match request.action, result with - | (Commit_graph_asset_deletion deletion | Cancel_graph_asset_deletion deletion), Ok Asset_unit -> - {core with asset_retirement_observed=Asset_retirement_set.add - (asset_retirement_key (asset_deletion_scope deletion) (asset_deletion_retirement deletion)) () - core.asset_retirement_observed} - | _ -> core in - if pending.asset_cancelled - then asset_cleanup core request result - else if not (asset_result_matches request result) - then ( - let cleaned = asset_cleanup core request result in - { cleaned with - effects = - (if pending.asset_notify - then - [ asset_output - (asset_public_request request) - (Error (Asset_invalid_content "Asset completion kind mismatch")) - ] - else []) - @ cleaned.effects - }) - else - { next = core - ; effects = - (if pending.asset_notify - then [ asset_output (asset_public_request request) result ] - else []) - } -;; - -let cancel_asset_pending core matches = - let assets_pending, outputs = - List.fold_right - (fun pending (retained, outputs) -> - if pending.asset_cancelled || not (matches pending.asset_ticket.asset_request) - then pending :: retained, outputs - else - ( { pending with asset_cancelled = true } :: retained - , if pending.asset_notify - then - asset_output - (asset_public_request pending.asset_ticket.asset_request) - (Error Asset_cancelled) - :: outputs - else outputs )) - core.assets_pending - ([], []) - in - { core with assets_pending }, outputs -;; - -let same_asset_account (left : account_scope) (right : account_scope) = - Uri.equal left.managed_sync_origin right.managed_sync_origin - && String.equal left.user_id right.user_id -;; - -let request_asset core (request : asset_request) = - if (match request.action with Fetch_asset _ | Put_asset_file _ -> true | _ -> false) - && not (network_allowed core) - then {next=core;effects=[asset_output request (Error Asset_network)]} else - let cleanup = asset_cleanup_action request.action in - let context = - match asset_context core with - | Some context when context.scope = request.scope -> Some context - | _ when cleanup -> - let known = - match request.action with - | Delete_account_assets -> - List.exists - (fun (scope : graph_scope) -> scope.account = request.scope.account) - core.asset_known_scopes - | Commit_graph_asset_deletion deletion | Cancel_graph_asset_deletion deletion -> - asset_deletion_scope deletion = request.scope - && (List.mem request.scope core.asset_known_scopes - || Option.fold ~none:false ~some:(fun user -> request.scope.account = account_scope core user) core.user_id) - | Prepare_graph_asset_deletion _ -> - let same_target (scope : graph_scope) = scope.account = request.scope.account - && scope.graph_id = request.scope.graph_id in - Option.fold ~none:false ~some:same_target core.deletion_scope - || Option.fold ~none:false ~some:same_target core.current_graph_scope - | Close_asset_scope|Await_asset_operations->List.mem request.scope core.asset_known_scopes - || Host_cleanup_scopes.mem request.scope core.host_cleanup_scopes - | _ -> List.mem request.scope core.asset_known_scopes - in - if known - then Some { scope = request.scope; encrypted = false; key = None } - else None - | _ -> None - in - let duplicate = - List.exists - (fun pending -> - let previous = pending.asset_ticket.asset_request in - (not pending.asset_cancelled) - && previous.context.scope = request.scope - && previous.operation = request.operation) - core.assets_pending - in - let valid = - request.operation <> "" - && - match request.action with - | Fetch_asset { maximum_plaintext_bytes; _ } - | Put_asset_file { maximum_plaintext_bytes; _ } -> - maximum_plaintext_bytes >= 0 && maximum_plaintext_bytes <= 100 * 1024 * 1024 - | Asset_retry_after seconds -> Float.is_finite seconds && seconds >= 0. - | Commit_graph_asset_deletion deletion | Cancel_graph_asset_deletion deletion -> - asset_deletion_scope deletion = request.scope - | _ -> true - in - if duplicate - then unchanged core - else if (core.closed && not cleanup) - || (match request.action with Prepare_graph_asset_deletion _ -> core.closed || core.user_id=None | _ -> false) - || context = None || not valid - then { next = core; effects = [ asset_output request (Error Asset_cancelled) ] } - else ( - let core, cancelled = - match request.action with - | Cancel_asset_operation operation -> - cancel_asset_pending core (fun previous -> - previous.context.scope = request.scope && previous.operation = operation) - | Close_asset_scope -> - cancel_asset_pending core (fun previous -> previous.context.scope = request.scope - && not(asset_cleanup_action previous.action)) - | Prepare_graph_asset_deletion retirement - when not (Asset_retirement_set.mem (asset_retirement_key request.scope retirement) core.asset_retirement_observed) -> - cancel_asset_pending core (fun previous -> - same_asset_account previous.context.scope.account request.scope.account - && previous.context.scope.graph_id = request.scope.graph_id) - | Delete_account_assets -> - cancel_asset_pending core (fun previous -> - same_asset_account previous.context.scope.account request.scope.account) - | _ -> core, [] - in - let core = match request.action with - | Prepare_graph_asset_deletion retirement -> - {core with - asset_retirement_observed=Asset_retirement_set.add (asset_retirement_key request.scope retirement) () - core.asset_retirement_observed - ; asset_known_scopes=if List.mem request.scope core.asset_known_scopes then core.asset_known_scopes - else request.scope :: core.asset_known_scopes} - | _ -> core in - let issued = - issue_asset - core - { context = Option.get context - ; operation = request.operation - ; action = request.action - } - in - { issued with effects = cancelled @ issued.effects }) -;; - -let graph_step = step - -let step core event = - let transition = - match event with - | Asset_requested request -> request_asset core request - | Runner_completed (Asset_completion (ticket, result)) -> - complete_asset core ticket result - | _ -> graph_step core event - in - let current = if transition.next.closed then None else asset_context transition.next in - let transition = - match current with - | Some context when not (List.mem context.scope transition.next.asset_known_scopes) -> - { transition with - next = - { transition.next with - asset_known_scopes = context.scope :: transition.next.asset_known_scopes - } - } - | Some _ | None -> transition - in - let retired = - transition.next.assets_pending - |> List.filter_map (fun pending -> - let context = pending.asset_ticket.asset_request.context in - if - pending.asset_cancelled - || asset_cleanup_action pending.asset_ticket.asset_request.action - || - match current with - | Some active -> active.scope = context.scope - | None -> false - then None - else Some context) - |> List.sort_uniq compare - in - List.fold_left - (fun transition (context : asset_context) -> - let next, cancelled = - cancel_asset_pending transition.next (fun request -> - request.context.scope = context.scope - && not (asset_cleanup_action request.action)) - in - let closed = - issue_asset - ~notify:false - next - { context - ; operation = "close:" ^ string_of_int next.next_effect_id - ; action = Close_asset_scope - } - in - { closed with effects = transition.effects @ cancelled @ closed.effects }) - transition - retired -;; - -let asset_step = step - -let protected_output request result = - Publish (Protected_finished { protected_request = request; protected_result = result }) -;; - -let protected_current core request = - (not core.closed) - && core.current_graph_scope = Some request.protected_scope - && - match request.protected_action with - | Encrypt_values (key, _) | Decrypt_value (key, _) -> - graph_key_handle_scope key = request.protected_scope -;; - -let step core event = - let transition = - match event with - | Protected_requested request -> - if - List.exists - (fun ticket -> - ticket.protected_input.protected_operation = request.protected_operation - && ticket.protected_input.protected_scope = request.protected_scope) - core.protected_pending - then unchanged core - else if not (protected_current core request) - then - { next = core - ; effects = - [ protected_output - request - (Error (Effect_failed "Protected-value request expired")) - ] - } - else ( - let ticket = - { protected_serial = core.next_effect_id; protected_input = request } - in - { next = - { core with - next_effect_id = core.next_effect_id + 1 - ; protected_pending = ticket :: core.protected_pending - } - ; effects = [ Run (Protected_io (ticket, request)) ] - }) - | Runner_completed (Protected_completion (ticket, result)) -> - if not (List.mem ticket core.protected_pending) - then unchanged core - else ( - let next = - { core with - protected_pending = - List.filter (fun pending -> pending <> ticket) core.protected_pending - } - in - let valid = - match ticket.protected_input.protected_action, result with - | _, Error _ - | Encrypt_values _, Ok (Encrypted_values _) - | Decrypt_value _, Ok (Decrypted_value _) -> true - | _ -> false - in - let result = - if not valid - then Error (Effect_failed "Protected completion kind mismatch") - else result - in - { next; effects = [ protected_output ticket.protected_input result ] }) - | _ -> asset_step core event - in - let active, retired = - List.partition - (fun ticket -> protected_current transition.next ticket.protected_input) - transition.next.protected_pending - in - { next = { transition.next with protected_pending = active } - ; effects = - transition.effects - @ List.map - (fun ticket -> - protected_output - ticket.protected_input - (Error (Effect_failed "Protected-value request expired"))) - retired - } -;; - -(* Physical ownership is distinct from the current graph. A scope change may - invalidate business events while the old socket still requires retirement. *) -let step_business = step - -let cancel_reconnect core = - match core.reconnect_timer with - | None -> core, [] - | Some (_, scope) -> - ( { core with reconnect_timer = None } - , [ Run (Cancel_effects (effect_scope_of_connection scope)) ] ) -;; - -let enforce_websocket_owner before (transition : transition) = - let next = transition.next in - let next = - match next.websocket_owner with - | Some owner - when owner.phase = Open - && next.websocket_live - && connection_is_current next owner.physical_scope - && next.public_state.snapshot.sync_phase = Current -> - { next with reconnect_delay = 1. } - | _ -> next - in - let target_changed = before.current_graph_scope <> next.current_graph_scope in - let next, cancelled = - if target_changed || (not next.foreground) || next.closed || not (network_allowed next) - then cancel_reconnect next - else next, [] - in - let wanted = - List.fold_left - (fun wanted -> function - | Run (Start_websocket request) -> Some request - | _ -> wanted) - (if target_changed then None else next.wanted_websocket) - transition.effects - in - let wanted = - match wanted with - | Some request - when next.foreground - && network_allowed next - && (not next.closed) - && next.current_graph_scope = Some request.scope.graph -> wanted - | Some _ | None -> None - in - let close_requested = - List.exists - (function - | Run (Close_websocket _) -> true - | _ -> false) - transition.effects - in - let effects = - List.filter - (function - | Run (Start_websocket _ | Close_websocket _) -> false - | _ -> true) - transition.effects - in - let next = - { next with - wanted_websocket = wanted - ; reconnect_pending = - next.reconnect_pending - && (not target_changed) - && next.foreground - && network_allowed next - && not next.closed - } - in - match next.websocket_owner with - | Some owner -> - let replace = - Option.fold - ~none:false - ~some:(fun (request : websocket_request) -> request.scope <> owner.physical_scope) - wanted - in - if - owner.phase <> Retiring - && (close_requested - || replace - || target_changed - || (not next.foreground) - || next.closed - || not (network_allowed next)) - then - { next = { next with websocket_owner = Some { owner with phase = Retiring } } - ; effects = cancelled @ effects @ [ Run (Close_websocket owner.physical_scope) ] - } - else { next; effects = cancelled @ effects } - | None -> - (match wanted with - | Some request -> - { next = - { next with - websocket_owner = Some { physical_scope = request.scope; phase = Opening } - ; wanted_websocket = None - } - ; effects = cancelled @ effects @ [ Run (Start_websocket request) ] - } - | None -> { next; effects = cancelled @ effects }) -;; - -let websocket_retired core scope = - match core.websocket_owner with - | Some owner when owner.physical_scope = scope && owner.phase = Retiring -> - let next = { core with websocket_owner = None } in - let cancelled = Run (Cancel_effects (effect_scope_of_connection scope)) in - if - next.reconnect_pending - && next.foreground - && (not next.closed) - && next.current_graph_scope = Some scope.graph - && next.wanted_websocket = None - then ( - let timer = next.next_effect_id in - let timer_scope = - { scope with connection_generation = next.connection_generation } - in - { next = - { next with - next_effect_id = timer + 1 - ; reconnect_timer = Some (timer, timer_scope) - ; reconnect_pending = false - ; reconnect_delay = min 30. (next.reconnect_delay *. 2.) - } - ; effects = - [ cancelled - ; Run - (Schedule_timer - { id = timer - ; scope = effect_scope_of_connection timer_scope - ; delay_seconds = next.reconnect_delay - }) - ] - }) - else - enforce_websocket_owner - core - { next = { next with reconnect_pending = false }; effects = [ cancelled ] } - | _ -> unchanged core -;; - -let step core event = - match event with - | Websocket_retired scope -> websocket_retired core scope - | Websocket_closed (scope, termination) -> - (match core.websocket_owner with - | Some owner - when owner.physical_scope = scope && owner.phase <> Retiring && not core.closed -> - let transition = step_business core event in - let retry = - match termination with - | Recoverable_close _ -> true - | Local_close | Authentication_failure _ | Protocol_failure _ -> false - in - let next = - { transition.next with - connection_generation = core.connection_generation + 1 - ; reconnect_pending = - retry && core.foreground && core.current_graph_scope = Some scope.graph - ; wanted_websocket = None - ; websocket_owner = Some { owner with phase = Retiring } - } - in - { next; effects = transition.effects @ [ Run (Close_websocket scope) ] } - | _ -> unchanged core) - | Websocket_opened scope - | Websocket_message (scope, _) - | Websocket_protocol_error (scope, _) -> - (match core.websocket_owner with - | Some owner - when owner.physical_scope = scope - && owner.phase <> Retiring - && connection_is_current core scope - && not core.closed -> - let core = - match event with - | Websocket_opened _ -> - { core with websocket_owner = Some { owner with phase = Open } } - | _ -> core - in - let transition = step_business core event in - let transition = - match event with - | Websocket_protocol_error _ -> - { transition with - effects = transition.effects @ [ Run (Close_websocket scope) ] - } - | _ -> transition - in - enforce_websocket_owner core transition - | _ -> unchanged core) - | Foreground_changed { foreground; lifecycle_generation } -> - if - Option.fold - ~none:false - ~some:(fun previous -> lifecycle_generation <= previous) - core.foreground_generation - then unchanged core - else ( - let same = foreground = core.foreground in - let core = - { core with - foreground - ; foreground_generation = Some lifecycle_generation - ; lifecycle_generation - } - in - if same - then unchanged core - else enforce_websocket_owner core (step_business core event)) - | Timer_elapsed id - when Option.fold ~none:false ~some:(fun (timer, _) -> timer = id) core.reconnect_timer - -> - let _, scope = Option.get core.reconnect_timer in - let next = { core with reconnect_timer = None } in - if - core.foreground - && (not core.closed) - && core.current_graph_scope = Some scope.graph - && core.websocket_owner = None - then enforce_websocket_owner core (start_websocket next scope.graph) - else unchanged next - | _ -> enforce_websocket_owner core (step_business core event) -;; - -let with_sync_cursor_diagnostics core (state : state) = - let title = "Sync transactions" in - let other_groups = List.filter (fun group -> group.title <> title) state.diagnostics.groups in - let groups = - match state.snapshot.selected_graph with - | None -> other_groups - | Some graph_id -> - let remote = - match core.diagnostic_remote_cursor with - | Some (scope, cursor) - when scope.graph_id = graph_id - && scope.account.account_generation = state.snapshot.startup.account_generation - && scope.graph_generation = state.snapshot.startup.graph_generation -> Some cursor - | _ -> None - in - let format = Option.fold ~none:"Not available" - ~some:(fun cursor -> Int64.to_string (Server_cursor.to_int64 cursor)) in - { title - ; entries = - [ "Local tx", format state.snapshot.applied_server_t - ; "Remote tx", format remote - ] - } :: other_groups - in - { state with diagnostics = { groups } } -;; - -let step_sync = step - -let step core event = - let transition = step_sync core event in - let transition = - if transition.next.force_pull_pending && transition.next.foreground - && not transition.next.closed && network_allowed transition.next - then - let pulled = request_pull transition.next in - { pulled with effects = transition.effects @ pulled.effects } - else transition - in - let public_state = with_sync_cursor_diagnostics transition.next transition.next.public_state in - let next = { transition.next with public_state } in - let effects = List.map (function - | Publish (State_changed state) -> Publish (State_changed (with_sync_cursor_diagnostics next state)) - | instruction -> instruction) transition.effects in - let effects = - if public_state.diagnostics <> core.public_state.diagnostics - && not (List.exists (function Publish (State_changed _) -> true | _ -> false) effects) - then effects @ [ publish next ] - else effects - in - { next; effects } -;; - - -(* Resident sessions use the existing graph policy as a lens. Account/catalog - fields and the global effect serial are never stored in graph_session. *) -let graph_pending (Pending_effect ticket) = ticket.scope.graph_generation <> None -let freeze_session identity core = - { saved_identity = identity - ; saved_snapshot = core.public_state.snapshot - ; saved_diagnostics = core.public_state.diagnostics - ; saved_pending_effects = List.filter graph_pending core.pending_effects - ; saved_asset_retirement_observed = core.asset_retirement_observed - ; saved_asset_known_scopes = core.asset_known_scopes - ; saved_protected_pending = core.protected_pending - ; saved_assets_pending = core.assets_pending - ; saved_selected_graph_value = core.selected_graph_value - ; saved_current_graph_scope = core.current_graph_scope - ; saved_connection_generation = core.connection_generation - ; saved_graph_key = core.graph_key - ; saved_pending_graph_open = core.pending_graph_open - ; saved_pending_mirror_inspection = core.pending_mirror_inspection - ; saved_pending_selected_download = core.pending_selected_download - ; saved_pending_attachment = core.pending_attachment - ; saved_unlock_preparation = core.unlock_preparation - ; saved_bootstrap = core.bootstrap - ; saved_sync_view = core.sync_view - ; saved_pending_sync_inspection = core.pending_sync_inspection - ; saved_pending_outbox_transition = core.pending_outbox_transition - ; saved_submission_owner = core.submission_owner - ; saved_pull_owner = core.pull_owner - ; saved_pull_wanted = core.pull_wanted - ; saved_force_pull_pending = core.force_pull_pending - ; saved_diagnostic_remote_cursor = core.diagnostic_remote_cursor - ; saved_pull_retry_delay = core.pull_retry_delay - ; saved_active_authoritative_batch = core.active_authoritative_batch - ; saved_queued_authoritative_batch = core.queued_authoritative_batch - ; saved_deferred_authoritative_owner = core.deferred_authoritative_owner - ; saved_websocket_live = core.websocket_live - ; saved_foreground = core.foreground - ; saved_foreground_generation = core.foreground_generation - ; saved_websocket_owner = core.websocket_owner - ; saved_wanted_websocket = core.wanted_websocket - ; saved_reconnect_timer = core.reconnect_timer - ; saved_reconnect_pending = core.reconnect_pending - ; saved_reconnect_delay = core.reconnect_delay - ; saved_closed = core.closed - } -let restore_session core session = - let shared = core.public_state.snapshot in - let snapshot = session.saved_snapshot in - let snapshot = {snapshot with catalog=shared.catalog;resident_graphs=core.resident_order; - startup={snapshot.startup with authenticated=shared.startup.authenticated; - account_generation=shared.startup.account_generation;catalog_loading=shared.startup.catalog_loading; - presentation_generation=max shared.startup.presentation_generation snapshot.startup.presentation_generation}} in - {core with public_state={snapshot;diagnostics=session.saved_diagnostics}; - pending_effects = List.filter (fun p -> not (graph_pending p)) core.pending_effects @ session.saved_pending_effects - ; asset_retirement_observed = session.saved_asset_retirement_observed - ; asset_known_scopes = session.saved_asset_known_scopes - ; protected_pending = session.saved_protected_pending - ; assets_pending = session.saved_assets_pending - ; selected_graph_value = session.saved_selected_graph_value - ; current_graph_scope = session.saved_current_graph_scope - ; connection_generation = session.saved_connection_generation - ; graph_key = session.saved_graph_key - ; pending_graph_open = session.saved_pending_graph_open - ; pending_mirror_inspection = session.saved_pending_mirror_inspection - ; pending_selected_download = session.saved_pending_selected_download - ; pending_attachment = session.saved_pending_attachment - ; unlock_preparation = session.saved_unlock_preparation - ; bootstrap = session.saved_bootstrap - ; sync_view = session.saved_sync_view - ; pending_sync_inspection = session.saved_pending_sync_inspection - ; pending_outbox_transition = session.saved_pending_outbox_transition - ; submission_owner = session.saved_submission_owner - ; pull_owner = session.saved_pull_owner - ; pull_wanted = session.saved_pull_wanted - ; force_pull_pending = session.saved_force_pull_pending - ; diagnostic_remote_cursor = session.saved_diagnostic_remote_cursor - ; pull_retry_delay = session.saved_pull_retry_delay - ; active_authoritative_batch = session.saved_active_authoritative_batch - ; queued_authoritative_batch = session.saved_queued_authoritative_batch - ; deferred_authoritative_owner = session.saved_deferred_authoritative_owner - ; websocket_live = session.saved_websocket_live - ; foreground = session.saved_foreground - ; foreground_generation = session.saved_foreground_generation - ; websocket_owner = session.saved_websocket_owner - ; wanted_websocket = session.saved_wanted_websocket - ; reconnect_timer = session.saved_reconnect_timer - ; reconnect_pending = session.saved_reconnect_pending - ; reconnect_delay = session.saved_reconnect_delay - ; closed = session.saved_closed - } -let remember_session identity core = - let saved = freeze_session identity core in - {core with resident_sessions=saved :: List.filter (fun s -> s.saved_identity <> identity) core.resident_sessions} -let remember_current core = - match core.current_graph_scope with None -> core | Some identity -> remember_session identity core -let resident_graphs core = - if core.resident_enabled then core.resident_order else Option.to_list core.public_state.snapshot.selected_graph -let scope_for_graph core ~graph_id = - if not core.resident_enabled then Option.bind core.current_graph_scope - (fun scope -> if scope.graph_id=graph_id then Some scope else None) - else if not (List.mem graph_id core.resident_order) then None - else List.find_map (fun session -> - if session.saved_identity.graph_id=graph_id && session.saved_current_graph_scope=Some session.saved_identity - then Some session.saved_identity else None) (remember_current core).resident_sessions -let state_for_scope core (scope : graph_scope) = - if scope_for_graph core ~graph_id:scope.graph_id <> Some scope then None - else if core.current_graph_scope=Some scope then Some core.public_state - else Option.map (fun session -> (restore_session core session).public_state) - (List.find_opt (fun s -> s.saved_identity=scope) core.resident_sessions) -type coordination_state = - { account : account_scope option - ; desired_residents : graph_id list - ; residents : (graph_scope * state) list - ; selected : timeline_scope option - ; deletion : (graph_scope * local_deletion) option - } -let coordination_state core = - let desired_residents=resident_graphs core in - let saved=(remember_current core).resident_sessions in - let residents=List.filter_map (fun id -> - Option.map (fun session -> session.saved_identity, - (restore_session core session).public_state) - (List.find_opt (fun session -> session.saved_identity.graph_id=id - && session.saved_identity.account.account_generation= - core.public_state.snapshot.startup.account_generation) saved)) desired_residents in - let snapshot=core.public_state.snapshot in - let selected=Option.bind snapshot.selected_graph (fun id -> - List.find_map (fun ((scope:graph_scope),_) -> - if scope.graph_id<>id then None else Some - {startup={account_generation=scope.account.account_generation; - presentation_generation=snapshot.startup.presentation_generation}; - graph_id=id;graph_generation=scope.graph_generation}) residents) in - {account=Option.map (account_scope core) core.user_id;desired_residents;residents;selected; - deletion=(match core.deletion_scope,snapshot.local_deletion with - | Some scope,Some phase -> Some(scope,phase) | _ -> None)} -let asset_context_for_scope core (scope : graph_scope) = - if scope_for_graph core ~graph_id:scope.graph_id <> Some scope then None - else if core.current_graph_scope=Some scope then asset_context core - else Option.bind (List.find_opt (fun s -> s.saved_identity=scope) core.resident_sessions) - (fun session -> asset_context (restore_session core session)) -let session_for_effect core (scope : effect_scope) = - List.find_opt (fun session -> - let s=session.saved_identity in - (scope.graph_generation=Some s.graph_generation && scope.account_generation=Some s.account.account_generation) - || List.exists (fun (Pending_effect ticket)->ticket.scope=scope) session.saved_pending_effects) - (remember_current core).resident_sessions -let private_deletion_asset_scope core (scope:graph_scope)= - Option.exists(fun (owner:graph_scope)->owner.graph_id=scope.graph_id && owner.account=scope.account)core.deletion_scope -;; -let event_graph_scope core = function - |Asset_requested{scope;action=(Close_asset_scope|Await_asset_operations);_} - when Host_cleanup_scopes.mem scope core.host_cleanup_scopes->None - |Runner_completed(Asset_completion(ticket,_)) - when (match ticket.asset_request.action with Close_asset_scope|Await_asset_operations->true|_->false) - && Host_cleanup_scopes.mem ticket.asset_request.context.scope core.host_cleanup_scopes->None - |Asset_requested {scope;action=(Prepare_graph_asset_deletion _|Commit_graph_asset_deletion _|Cancel_graph_asset_deletion _);_} - when private_deletion_asset_scope core scope->None - |Runner_completed(Asset_completion(ticket,_)) - when private_deletion_asset_scope core ticket.asset_request.context.scope->None - | Asset_requested ({scope;action;_} as request) when asset_cleanup_action action -> - (match List.find_opt (fun session->List.mem scope session.saved_asset_known_scopes) - (remember_current core).resident_sessions with - | Some session->Some session.saved_identity | None->Some request.scope) - | Local_graph_outbox_changed scope | Asset_requested {scope;_} - | Protected_requested {protected_scope=scope;_} | Graph_detached (scope,_) - | Mirror_inspected (Mirror_absent scope) -> Some scope - | Mirror_inspected (Mirror_available {scope;_}) | Graph_attached {scope;_} - | Sync_inspected {scope;_} | Outbox_transition_applied {scope;_} - | Authoritative_batch_applied {scope;_} | Authoritative_batch_deferred {scope;_} - | Outbox_transition_superseded {request={scope;_};_} - | Outbox_transition_failed {request={scope;_};_} -> Some scope - | Snapshot_activated activation -> Some activation.scope - | Snapshot_activation_failed error -> Some error.activation.scope - | Websocket_opened {graph;_} | Websocket_message ({graph;_},_) - | Websocket_protocol_error ({graph;_},_) | Websocket_closed ({graph;_},_) -> Some graph - | Websocket_retired connection -> - (match List.find_opt (fun session->Option.fold ~none:false - ~some:(fun owner->owner.physical_scope=connection) session.saved_websocket_owner) - (remember_current core).resident_sessions with - | Some session->Some session.saved_identity | None->Some connection.graph) - | E2ee_password_submitted (challenge,_) -> Some challenge.challenge_scope - | Graph_attachment_failed {scope;_} | Authoritative_batch_failed {scope;_} -> - Option.map (fun s -> s.saved_identity) (session_for_effect core scope) - | Runner_completed (Completion (ticket,_)) -> - List.find_map (fun session->if List.exists (fun (Pending_effect pending)-> - pending.id=ticket.id && pending.scope=ticket.scope) session.saved_pending_effects - then Some session.saved_identity else None) (remember_current core).resident_sessions - | Runner_completed (Asset_completion (ticket,_)) -> - List.find_map (fun session->if List.exists (fun pending->pending.asset_ticket=ticket) session.saved_assets_pending - then Some session.saved_identity else None) (remember_current core).resident_sessions - | Runner_completed (Protected_completion (ticket,_)) -> - List.find_map (fun session->if List.mem ticket session.saved_protected_pending - then Some session.saved_identity else None) (remember_current core).resident_sessions - | Timeline_presented scope | Online_recovery_requested scope -> scope_for_graph core ~graph_id:scope.graph_id - | Timer_elapsed id -> - List.find_map (fun s -> - let timer = Option.fold ~none:false ~some:(fun (timer,_) -> timer=id) s.saved_reconnect_timer - || Option.fold ~none:false ~some:(fun owner -> owner.response_timer=Some id) s.saved_submission_owner - || Option.fold ~none:false ~some:(fun owner -> match owner.pull_stage with - | Awaiting_pull_response timer | Awaiting_pull_retry timer -> timer=id | Applying_pull -> false) s.saved_pull_owner in - if timer then Some s.saved_identity else None) (remember_current core).resident_sessions - | _ -> None -let step_graph_policy = step -let scoped_publications (scope : graph_scope) effects = List.map (function - | Publish (State_changed state) -> Publish (Session_state_changed (scope,state)) - | instruction -> instruction) effects -let visible_identity core = core.public_state.snapshot.selected_graph -let restore_visible core id = - match id with - | Some id -> (match List.find_opt (fun s -> s.saved_identity.graph_id=id - && List.mem id core.resident_order && s.saved_current_graph_scope=Some s.saved_identity) core.resident_sessions with - | Some session -> restore_session core session | None -> core) - | None -> core -let run_session core (identity : graph_scope) event = - let core=remember_current core in - match List.find_opt (fun s -> s.saved_identity=identity) core.resident_sessions with - | None -> - (match event with - | Asset_requested request -> - {next=core;effects=[asset_output request (Error Asset_cancelled)]} - | Protected_requested request -> - {next=core;effects=[protected_output request (Error(Effect_failed "Protected-value request expired"))]} - | _->unchanged core) - | Some session -> - let visible=visible_identity core in - let shared=core.public_state.snapshot in - let transition=step_graph_policy (restore_session core session) event in - let snapshot=transition.next.public_state.snapshot in - let snapshot={snapshot with catalog=shared.catalog; - startup={snapshot.startup with authenticated=shared.startup.authenticated; - catalog_loading=shared.startup.catalog_loading;account_generation=shared.startup.account_generation}} in - let _,account_pending=List.fold_left (fun (ids,pending) (Pending_effect ticket as obligation)-> - if graph_pending obligation || Pending_id_set.mem ticket.id ids then ids,pending - else Pending_id_set.add ticket.id ids,obligation::pending) - (Pending_id_set.empty,[]) (core.pending_effects @ transition.next.pending_effects) in - let updated={transition.next with public_state={transition.next.public_state with snapshot}; - pending_effects=List.rev account_pending @ List.filter graph_pending transition.next.pending_effects} in - let fresh=Option.value updated.current_graph_scope ~default:identity in - let updated=if fresh=identity then updated else {updated with - resident_sessions=List.filter(fun session->session.saved_identity<>identity) updated.resident_sessions} in - let next=remember_session fresh updated |> fun core -> restore_visible core visible in - (* A cleanup-only lens must not replace the account's picker or startup - authority when no graph is visible. Its tickets still advance above. *) - let next=if visible=None then {next with public_state=core.public_state} else next in - {next;effects=if visible=Some fresh.graph_id then transition.effects - else scoped_publications fresh transition.effects} -let empty_session core = - let blank=initial core.config |> Result.get_ok in - let identity=Option.get core.current_graph_scope in - let empty=freeze_session identity blank in - let next=restore_session core empty in - {next with public_state={next.public_state with snapshot={next.public_state.snapshot with - startup={next.public_state.snapshot.startup with - graph_generation=core.resident_next_generation;network_phase=core.public_state.snapshot.startup.network_phase}}}} -let valid_residents core ids = - List.length ids<=3 && List.length (List.sort_uniq compare ids)=List.length ids - && List.for_all (fun id -> Option.is_some (graph_in_catalog core.public_state.snapshot.catalog id)) ids -let configure_residents core ids = - let ids=List.filter(fun id->not(host_fence_blocks core id))ids in - if not (valid_residents core ids) then unchanged core else - let old_visible=visible_identity core in - let core=remember_current {core with resident_enabled=true} in - let removed=List.filter (fun s -> List.mem s.saved_identity.graph_id core.resident_order - && not (List.mem s.saved_identity.graph_id ids)) core.resident_sessions in - let core={core with resident_order=ids} in - let core,effects=List.fold_left (fun (core,effects) session -> - let closed=step_graph_policy (restore_session core session) Graph_picker_requested in - let next=remember_session session.saved_identity closed.next in - next,effects @ scoped_publications session.saved_identity closed.effects) (core,[]) removed in - let core,effects=List.fold_left (fun (core,effects) id -> - if Option.is_some (scope_for_graph core ~graph_id:id) then core,effects else - let base = match core.current_graph_scope with - | Some _ -> empty_session core - | None -> let blank=initial core.config |> Result.get_ok in - let base={blank with host_fence=core.host_fence;host_cleanup_scopes=core.host_cleanup_scopes;user_id=core.user_id;next_effect_id=core.next_effect_id; - resident_order=core.resident_order;resident_enabled=true;resident_sessions=core.resident_sessions; - resident_next_generation=core.resident_next_generation;foreground=core.foreground; - lifecycle_generation=core.lifecycle_generation;public_state=core.public_state} in - {base with public_state={base.public_state with snapshot={base.public_state.snapshot with - startup={base.public_state.snapshot.startup with graph_generation=core.resident_next_generation}}}} in - let opened=select_graph_transition base ~persist:false id in - let snapshot=opened.next.public_state.snapshot in - let opened={opened with next={opened.next with public_state={opened.next.public_state with - snapshot={snapshot with startup={snapshot.startup with network_phase=base.public_state.snapshot.startup.network_phase}}}}} in - let scope=Option.get opened.next.current_graph_scope in - let next=remember_current {opened.next with resident_next_generation=scope.graph_generation} in - next,effects @ scoped_publications scope opened.effects) (core,effects) ids in - let visible=match old_visible with Some id when List.mem id ids -> Some id | _ -> List.nth_opt ids 0 in - let next=restore_visible core visible in - let snapshot={next.public_state.snapshot with selected_graph=visible;resident_graphs=ids} in - let next={next with cached_selected_graph=visible;public_state={next.public_state with snapshot}} in - let next,saved=save_cache next in - {next;effects=effects @ [publish next] @ saved} -let resume_resident_network core = - let shared=core.public_state.snapshot in - if shared.startup.network_phase=Local_only then unchanged core else - let original=visible_identity core in - let sessions=(remember_current core).resident_sessions in - let next,effects=List.fold_left (fun (core,effects) session -> - if not (List.mem session.saved_identity.graph_id core.resident_order) then core,effects else - let loaded=restore_session core session |> fun c -> network_phase c shared.startup.network_phase in - let resumed = if not shared.startup.authenticated then unchanged loaded else - match loaded.current_graph_scope,loaded.sync_view with - | Some scope,Some _ when not loaded.websocket_live && (match loaded.websocket_owner with - | None | Some {phase=Retiring;_} -> true | Some _ -> false) -> enforce_websocket_owner loaded (start_websocket loaded scope) - | Some scope,None when loaded.pending_selected_download=Some scope && loaded.public_state.snapshot.startup.bootstrapping -> - begin_missing_mirror_bootstrap loaded scope - | Some scope,None when loaded.public_state.snapshot.startup.local_access=Local_unavailable Local_key_unavailable - && loaded.public_state.snapshot.startup.failure=Some During_local_restore -> begin_e2ee_key_access loaded scope - | _ -> unchanged loaded in - let next=remember_session session.saved_identity resumed.next in - next,effects @ (if original=Some session.saved_identity.graph_id then resumed.effects - else scoped_publications session.saved_identity resumed.effects)) (remember_current core,[]) sessions in - {next=restore_visible next original;effects} -let cleanup_session core session = - let empty=freeze_session session.saved_identity (initial core.config |> Result.get_ok) in - {empty with saved_asset_known_scopes=session.saved_asset_known_scopes; - saved_asset_retirement_observed=session.saved_asset_retirement_observed; - saved_assets_pending=session.saved_assets_pending; - saved_protected_pending=session.saved_protected_pending} -let prune_sessions core = - {core with resident_sessions=List.filter_map (fun session -> - if List.mem session.saved_identity.graph_id core.resident_order - || session.saved_current_graph_scope<>None || session.saved_websocket_owner<>None - || session.saved_pending_effects<>[] || session.saved_assets_pending<>[] - || session.saved_protected_pending<>[] - then Some session - else if session.saved_asset_known_scopes=[] - then None - else ( - (* Original native leases retain cleanup authority across graph/account - retirement. This tombstone cannot acquire/read: it has no live snapshot, - key, or execution owner. Keep it for idempotent release even after a - checked cache close, including a lost per-handle ACK. *) - Some(cleanup_session core session))) core.resident_sessions} -let step core event = - match event with - | Resident_graphs_changed ids -> configure_residents core ids - | Graph_selected id when core.resident_enabled && List.mem id core.resident_order -> - let focused=restore_visible (remember_current core) (Some id) in - (match scope_for_graph focused ~graph_id:id with - | Some identity->run_session focused identity event | None->unchanged core) - | Visible_graph_changed id when List.mem id (resident_graphs core) -> - let core=remember_current core in - let next=restore_visible core (Some id) in - let next={next with cached_selected_graph=Some id} in - let next,effects=save_cache next in - {next;effects=publish next :: effects} - | Visible_graph_changed _ -> unchanged core - | Local_cache_deletion_requested id when List.mem id (resident_graphs core) -> unchanged core - | _ when not core.resident_enabled -> - let event = match event with - | Local_graph_outbox_changed scope when graph_scope_is_current core scope -> - Local_outbox_changed - | event -> event in - let transition=step_graph_policy core event in - let next={transition.next with resident_next_generation=max core.resident_next_generation transition.next.public_state.snapshot.startup.graph_generation} in - (match next.restored_residents with - | Some ids when ids<>[] -> - let restored=configure_residents {next with restored_residents=None} ids in - {restored with effects=transition.effects @ restored.effects} - | _ -> {transition with next}) - | Local_graph_outbox_changed scope -> run_session core scope Local_outbox_changed - | Account_authenticated {user_id} when user_id=None || user_id<>core.user_id -> - let retired=(remember_current core).resident_sessions in - let closed,closes=List.fold_left(fun(sessions,effects) session -> - if not(List.mem session.saved_identity.graph_id core.resident_order) then session::sessions,effects - else let part=step_graph_policy (restore_session core session) Graph_picker_requested in - freeze_session session.saved_identity part.next::sessions, - effects @ scoped_publications session.saved_identity part.effects) ([],[]) retired in - let signed_out=step_graph_policy core event in - (* Account cleanup owns the old physical files. Already admitted requests - must still receive their completion while their native owners retire. *) - let cleanup=List.filter_map(fun session -> - if session.saved_assets_pending=[] && session.saved_protected_pending=[] - && session.saved_asset_known_scopes=[] then None - else Some(cleanup_session signed_out.next session)) closed in - {next={signed_out.next with resident_order=[];resident_sessions=cleanup; - public_state={signed_out.next.public_state with snapshot={signed_out.next.public_state.snapshot with resident_graphs=[]}}}; - effects=closes @ signed_out.effects} - | Online_cancelled scope when startup_scope_is_current core scope -> - let visible=visible_identity core in - let sessions=(remember_current core).resident_sessions in - let next,effects=List.fold_left(fun(core,effects) session-> - let loaded=restore_session core session in - let startup=loaded.public_state.snapshot.startup in - let part=step_graph_policy loaded (Online_cancelled - {account_generation=startup.account_generation;presentation_generation=startup.presentation_generation}) in - remember_session session.saved_identity part.next, - effects @ (if visible=Some session.saved_identity.graph_id then part.effects - else scoped_publications session.saved_identity part.effects)) (remember_current core,[]) sessions in - let next=restore_visible next visible in - let effects=List.filter (function Publish(State_changed _)->false | _->true) effects in - {next;effects=effects @ [publish next]} - | Foreground_changed _ | Shutdown -> - let sessions=(remember_current core).resident_sessions in - let transition=List.fold_left (fun transition s -> - let part=run_session transition.next s.saved_identity event in - {next=part.next;effects=transition.effects @ part.effects}) (unchanged core) sessions in - if event=Shutdown then {transition with next={transition.next with resident_order=[]; - public_state={transition.next.public_state with snapshot={transition.next.public_state.snapshot with resident_graphs=[];selected_graph=None}}}} - else transition - | _ -> - match event_graph_scope core event with - | Some identity -> run_session core identity event - | None -> - let transition=step_graph_policy core event in - let next=match core.current_graph_scope,transition.next.current_graph_scope with - | Some old,Some fresh when old<>fresh && old.graph_id=fresh.graph_id -> - remember_current {transition.next with resident_sessions=List.filter - (fun session->session.saved_identity<>old) transition.next.resident_sessions} - | _->remember_current transition.next in - let resumed=resume_resident_network next in - let transition={next=resumed.next;effects=transition.effects @ resumed.effects} in - let next=transition.next in - (* Catalog/authentication have one authority. Revoke removed sessions. *) - let ids=List.filter (fun id -> Option.is_some (graph_in_catalog next.public_state.snapshot.catalog id)) next.resident_order in - if ids<>next.resident_order then - let reconciled=configure_residents next ids in - {reconciled with effects=transition.effects @ reconciled.effects} - else {transition with next} -;; - -let step core event = let transition=step core event in - {transition with next=prune_sessions transition.next} - -let step core event = - let transition=step core event in - let scopes c=List.filter_map (fun id->scope_for_graph c ~graph_id:id) (resident_graphs c) in - let changed=transition.next.resident_enabled && scopes core<>scopes transition.next in - if changed then {transition with effects=transition.effects @ [Publish(Resident_scopes_changed(scopes transition.next))]} - else transition -;; - -let step core = function - |IO_fence_requested(id,scope)->{next=core;effects=[Run(Await_io(id,scope))]} - |IO_fence_completed _->unchanged core - |event->step core event -;; - -let step core = function - |Host_fence_started fence-> - let current=Option.map(account_scope core)core.user_id in - let owner_matches=Option.exists(fun (owner:account_scope)-> - Uri.equal owner.managed_sync_origin fence.owner.managed_sync_origin - && owner.user_id=fence.owner.user_id && owner.account_generation=fence.owner.account_generation - && owner.lifecycle_generation=fence.owner.lifecycle_generation)current in - let replace_matches=match core.host_fence,fence.previous with - |None,None->true - |Some old,Some previous->old.id=previous && old.target=fence.target - |Some old,None->old.id=fence.id && old.target=fence.target - |_->false in - if not owner_matches || not replace_matches || Rrbvec.length fence.captured_scopes>8192 then unchanged core else - let residents=resident_graphs core in - let core=remember_current{core with host_fence=Some fence; - host_cleanup_scopes=Rrbvec.fold_left(fun scopes scope->Host_cleanup_scopes.add scope scopes)Host_cleanup_scopes.empty fence.captured_scopes; - resident_enabled=true;resident_order=residents} in - let closed=configure_residents core(List.filter(fun id->not(host_fence_blocks core id))residents) in - (match fence.target with - |One_graph _->closed - |All_graphs-> - let scope={ (effect_scope_of_account fence.owner) with - presentation_generation=None;lifecycle_generation=None } in - {closed with effects=Run(Cancel_effects scope)::closed.effects}) - |Host_fence_released id-> - (match core.host_fence with - |Some fence when fence.id=id->let next={core with host_fence=None;host_cleanup_scopes=Host_cleanup_scopes.empty} in {next;effects=[publish next]} - |_->unchanged core) - |Account_authenticated{user_id=Some user} when core.user_id=Some user - && Option.exists(fun fence->fence.target=All_graphs)core.host_fence->unchanged core - |(Catalog_refresh_requested _|Sign_in_requested _|Online_recovery_requested _) - when Option.exists(fun fence->fence.target=All_graphs)core.host_fence->unchanged core - |event->step core event -;; -let host_fence_active core=core.host_fence diff --git a/logseq_sync/lib/pure_reducer/dune b/logseq_sync/lib/pure_reducer/dune deleted file mode 100644 index 8e05a960..00000000 --- a/logseq_sync/lib/pure_reducer/dune +++ /dev/null @@ -1,8 +0,0 @@ -(library - (name logseq_sync_pure_reducer_impl) - (public_name logseq_sync.pure_reducer.impl) - (implements logseq_sync_pure_reducer) - (modules core sync_protocol asset_transfer) - (libraries logseq_db_types logseq_overlay_db ppx_deriving_yojson.runtime rrbvec uri yojson) - (preprocess - (pps ppx_deriving_yojson))) diff --git a/logseq_sync/lib/service_lifecycle/dune b/logseq_sync/lib/service_lifecycle/dune new file mode 100644 index 00000000..e0a13245 --- /dev/null +++ b/logseq_sync/lib/service_lifecycle/dune @@ -0,0 +1,4 @@ +(library + (name logseq_sync_service_lifecycle) + (public_name logseq_sync.internal.service_lifecycle) + (modules logseq_sync_service_lifecycle)) diff --git a/logseq_sync/lib/service_lifecycle/logseq_sync_service_lifecycle.ml b/logseq_sync/lib/service_lifecycle/logseq_sync_service_lifecycle.ml new file mode 100644 index 00000000..5d695b5f --- /dev/null +++ b/logseq_sync/lib/service_lifecycle/logseq_sync_service_lifecycle.ml @@ -0,0 +1,34 @@ +type lifecycle = + | Running + | Closing + | Closed + +type t = + { lifecycle : lifecycle + ; close_called : bool + } + +type close_error = + | Reentrant + | Already_called + +let initial = { lifecycle = Running; close_called = false } +let lifecycle state = state.lifecycle + +let request_close ~inside_callback state = + if inside_callback + then Error Reentrant + else if state.close_called + then Error Already_called + else + Ok + { lifecycle = + (match state.lifecycle with + | Running | Closing -> Closing + | Closed -> Closed) + ; close_called = true + } +;; + +let begin_cleanup state = { state with lifecycle = Closing } +let complete state = { state with lifecycle = Closed } diff --git a/logseq_sync/lib/service_lifecycle/logseq_sync_service_lifecycle.mli b/logseq_sync/lib/service_lifecycle/logseq_sync_service_lifecycle.mli new file mode 100644 index 00000000..00bf2517 --- /dev/null +++ b/logseq_sync/lib/service_lifecycle/logseq_sync_service_lifecycle.mli @@ -0,0 +1,16 @@ +type lifecycle = + | Running + | Closing + | Closed + +type t + +type close_error = + | Reentrant + | Already_called + +val initial : t +val lifecycle : t -> lifecycle +val request_close : inside_callback:bool -> t -> (t, close_error) result +val begin_cleanup : t -> t +val complete : t -> t diff --git a/logseq_sync/lib/sync_dsl/core.ml b/logseq_sync/lib/sync_dsl/core.ml new file mode 100644 index 00000000..b0c9a2ca --- /dev/null +++ b/logseq_sync/lib/sync_dsl/core.ml @@ -0,0 +1,2126 @@ +type cold +type ready +type draining +type retired + +type _ phase = + | Cold : cold phase + | Ready : ready phase + | Draining : draining phase + | Retired : retired phase + +type _ drainable = + | Cold_context : cold drainable + | Ready_context : ready drainable + | Draining_context : draining drainable + +type identity = + { origin : string + ; account : string + ; graph : string + ; account_generation : int64 + ; graph_generation : int64 + } + +module Overlay = Logseq_overlay_db + +type error = + | Invalid_input of string + | Identity_mismatch of string + | Stale_context + | Network_closed + | Network_failure of string + | Authentication_failure of string + | Locked + | Crypto_failure of string + | Protocol_failure of string + | Storage_failure of string + | Cancelled + | Outcome_unknown of string + | Cleanup_failure of + { primary : error option + ; failures : error Rrbvec.t + } + | Capacity_exceeded of string + | Sync_read_failure of Overlay.Types.sync_read_error + | Outbox_failure of Overlay.Types.outbox_transition_error + | Authoritative_failure of Overlay.Types.authoritative_transition_error + | Activation_failure of Overlay.Types.snapshot_activation_error + | Asset_failure of Logseq_db_types.Failure_diagnostic.t + | Unexpected_failure of string + +type limits = + { maximum_items : int + ; maximum_bytes : int + ; maximum_pending : int + ; maximum_resources : int + ; steps_per_turn : int + ; maximum_tasks : int + ; maximum_messages : int + ; maximum_message_bytes : int + ; maximum_demands : int + ; maximum_active_assets : int + ; foreground_reserve : int + ; foreground_demand_reserve : int + ; foreground_descriptor_reserve_bytes : int + ; asset_cache_budget_bytes : int64 + ; asset_maximum_file_bytes : int + ; asset_pending_budget_bytes : int64 + ; snapshot_maximum_artifact_bytes : int64 + ; snapshot_staging_budget_bytes : int64 + ; snapshot_scratch_budget_bytes : int64 + } + +type retry_policy = + { initial_seconds : float + ; maximum_seconds : float + ; multiplier : float + ; maximum_attempts : int + } + +type network_gate = + | Closed + | Open of { authorization_epoch : int64 } + +type key_status = + | Unencrypted + | Key_locked + | Key_available + +type observation = + { identity : identity + ; keys : key_status + ; network : network_gate + ; checkpoint : Overlay.Types.server_cursor option + ; pending_submissions : int + ; pending_assets : int + ; cleanup_pending : int + ; last_failure : error option + } + +type restore_report = + { keys : key_status + ; checkpoint : Overlay.Types.server_cursor option + ; recovery_pending : bool + } + +type activation = + | Activated of Overlay.Types.generation + | Activation_deferred of string + +type authoritative_result = + | Applied of Overlay.Types.authoritative_commit + | Deferred of Overlay.Types.authoritative_defer + +type resource_release = + { mutable release_owner : identity ref option + ; mutable release_metadata : string option + ; mutable release_acknowledged : bool + } + +let resource_release () = + { release_owner = None; release_metadata = None; release_acknowledged = false } +;; + +type resource = + { owner : identity + ; token : string + ; release : resource_release + } + +type snapshot = + { resource : resource + ; through : Overlay.Types.server_cursor + ; checksum : string + ; encrypted : bool + } + +type _ database = + | Inspect : Overlay.Types.sync_view database + | Verify_checksum : + Overlay.Types.server_cursor * Overlay.Types.checksum + -> unit database + | Prepare_submission : + Overlay.Types.sync_token * Overlay.Types.outbox_transition + -> Overlay.Types.outbox_commit database + | Apply_authoritative : + Overlay.Types.sync_token * Overlay.Types.authoritative_batch + -> authoritative_result database + | Activate_snapshot : snapshot -> activation database + | Release_snapshot : snapshot -> unit database + +type remote_head = + { through : Overlay.Types.server_cursor + ; snapshot_available : bool + ; expected_checksum : Overlay.Types.checksum option + } + +type pull_result = + | Pulled of Overlay.Types.authoritative_batch + | No_progress of + { since : Overlay.Types.server_cursor + ; through : Overlay.Types.server_cursor + ; expected_checksum : Overlay.Types.checksum option + } + +type submission_reply = + | Accepted of Overlay.Types.acceptance_barrier + | Rejected of Overlay.Types.rejection_resolution + +type connection = Connection of identity * string * int64 * resource_release + +type connection_end = + | Remote_closed + | Network_lost of string + | Access_revoked of string + | Invalid_protocol of string + | Locally_closed + +type remote_message = + | Remote_error of string + | Remote_changed of remote_head + | Connection_ended of connection_end + +type received = + { connection : connection + ; message : remote_message + } + +type _ transport = + | Connect : connection transport + | Receive : connection -> received transport + | Retire_connection : connection -> unit transport + | On_connection : connection * 'a transport -> 'a transport + | Head : remote_head transport + | Fetch_snapshot : Overlay.Types.server_cursor -> snapshot transport + | Pull : Overlay.Types.server_cursor -> pull_result transport + | Push : Overlay.Types.submission_batch -> submission_reply transport + +type unlock_input = string + +type _ crypto = + | Unlock : unlock_input -> key_status crypto + | Encrypt_values : string Rrbvec.t -> (string * string) Rrbvec.t crypto + | Decrypt_value : string -> string crypto + | Protect : + Overlay.Database.protection_request + -> (Overlay.Types.crypto_item_id * string) list crypto + | Unprotect : + Overlay.Database.unprotection_request + -> (Overlay.Types.crypto_item_id * string) list crypto + +type asset = Logseq_db_types.Asset_descriptor.t + +type asset_file = + { file : resource + ; asset : asset + } + +type asset_lease = + { lease : resource + ; source : asset_file + } + +type resource_reference = + | Snapshot_reference of snapshot + | Connection_reference of connection + | File_reference of asset_file + | Lease_reference of asset_lease + +type asset_result = + | Available of asset_file + | Waiting_remote + | Missing_local + +type cache_report = + { resident_bytes : int64 + ; pending_bytes : int64 + ; retained_bytes : int64 + ; cleanup_pending : int + ; over_budget : bool + } + +type _ assets = + | Prune_cache : cache_report assets + | Reconcile_staging : int assets + | Lookup : asset -> asset_file option assets + | Download : asset -> asset_result assets + | Upload : asset -> unit assets + | Retain : asset_file -> asset_lease assets + | Release_file : asset_file -> unit assets + | Release_lease : asset_lease -> unit assets + +type scope = + { scope_owner : identity ref + ; mutable scope_live : bool + ; mutable registrations : (unit ref * (unit -> unit)) list + } + +type 'a child_state = + | Pending_child of (unit -> ('a, error) result option) + | Completed_child of ('a, error) result + | Consumed_child + +type handle = { handle_scope : scope } +type demand_id = string + +type priority = + | Foreground + | Background + +type demand_change = + | Demand of + { id : demand_id + ; asset : asset + ; priority : priority + } + | Withdraw of demand_id + +type demand_receipt = Demand_receipt of handle +type unlock_receipt = Unlock_receipt of handle + +type service_input = + | Unlock_requested of unlock_receipt + | Demand_changed of demand_receipt * demand_change + | Network_changed of network_gate + | Stop_requested + +type asset_failure_reason = + | Asset_network of string + | Asset_network_closed + | Asset_not_found + | Asset_checksum_mismatch + | Asset_authentication of string + | Asset_locked + | Asset_storage_full + | Asset_missing_source + | Asset_size_rejected + | Asset_revoked_access + | Asset_cancelled + | Asset_invalid_content of string + | Asset_other of error + +type demand_transfer = + { result : (asset_result, asset_failure_reason) result + ; attempts : int + } + +type demand_availability = + | Demand_queued + | Demand_downloading + | Demand_waiting_network + | Demand_waiting_unlock + | Demand_failed of + { reason : asset_failure_reason + ; attempts : int + ; retry_scheduled : bool + } + | Demand_available of asset_file * asset_lease + | Demand_waiting_remote + | Demand_missing_local + | Demand_revoked + +type demand_delivery = Delivery of identity ref * bool ref + +type demand_output = + { delivery : demand_delivery + ; id : demand_id + ; demand : demand_receipt + ; availability : demand_availability + } + +type outbox_check = Outbox_check of handle + +type capacity_revision = + { capacity_scope : scope + ; sequence : int64 + } + +type notifications = Notifications_handle of handle + +type notification = + | Policy_failure of error + | Connected of connection + | Remote_received of received + | Connection_retired of connection + +type service_status = + | Service_idle + | Service_connecting + | Service_pulling + | Service_submitting + | Service_online + | Service_waiting_network + | Service_waiting_unlock + | Service_backoff of + { attempt : int + ; seconds : float + ; cause : error + } + | Service_blocked of error + +type 'a task = Task of 'a task_node + +and 'a child = + { child_scope : scope + ; registration : unit ref + ; mutable child_task : 'a task option + ; mutable child_state : 'a child_state + } + +and 'a wake = + | Input of service_input + | Child_ready of 'a child + +and _ task_operation = + | Notifications : scope -> notifications task_operation + | Notify : notifications * notification -> unit task_operation + | Next_notification : notifications -> notification task_operation + | Await_outbox_changed : outbox_check task_operation + | Acknowledge_outbox_check : outbox_check -> unit task_operation + | Abandon_outbox_check : outbox_check -> unit task_operation + | Resample_outbox : unit task_operation + | Capacity_revision : capacity_revision task_operation + | Await_capacity_changed : capacity_revision -> unit task_operation + | Task_observe : observation task_operation + | Publish_status : service_status -> unit task_operation + | Task_database : 'a database -> 'a task_operation + | Task_transport : 'a transport -> 'a task_operation + | Task_crypto : 'a crypto -> 'a task_operation + | Task_assets : 'a assets -> 'a task_operation + | Consume_unlock : unlock_receipt -> key_status task_operation + | Publish_demand : demand_receipt * demand_availability -> unit task_operation + | Download_demand : demand_receipt -> demand_transfer task_operation + | Finish_demand : demand_receipt -> unit task_operation + | Wait_network_open : unit task_operation + | Next_input : service_input task_operation + | Await_input_or_child : 'a child Rrbvec.t -> 'a wake task_operation + | Sleep : float -> unit task_operation + | Yield : unit task_operation + +and _ task_node = + | Task_return : 'a -> 'a task_node + | Task_fail : error -> 'a task_node + | Task_perform : + 'a task_operation * ('a -> 'b task) * (error -> 'b task) + -> 'b task_node + | Task_scope : (scope -> 'a task) -> 'a task_node + | Task_spawn : scope * 'a task -> ('a child, error) result task_node + | Task_spawn_asset : + scope * demand_receipt * 'a task + -> ('a child, error) result task_node + | Task_join : 'a child -> ('a, error) result task_node + | Task_cancel : 'a child -> unit task_node + | Task_bind : 'a task * ('a -> 'b task) -> 'b task_node + +type (_, _, _, _) operation = + | Run_task : 'a task -> (ready, ready, ready, 'a) operation + | Restore : (cold, ready, cold, restore_report) operation + | Install_initial_network : network_gate -> (ready, ready, ready, unit) operation + | Set_network : network_gate -> (ready, ready, ready, unit) operation + | Database : 'a database -> (ready, ready, ready, 'a) operation + | Transport : 'a transport -> (ready, ready, ready, 'a) operation + | Crypto : 'a crypto -> (ready, ready, ready, 'a) operation + | Assets : 'a assets -> (ready, ready, ready, 'a) operation + | Observe : ('s, 's, 's, observation) operation + | Begin_drain : 's drainable -> ('s, draining, draining, unit) operation + | Finish_retirement : (draining, retired, draining, unit) operation + | Release_file : asset_file -> ('s, 's, 's, unit) operation + | Release_lease : asset_lease -> ('s, 's, 's, unit) operation + | Release_snapshot : snapshot -> ('s, 's, 's, unit) operation + +module Program = Logseq_sync_indexed_program.Make (struct + type nonrec error = error + type nonrec ('a, 'b, 'c, 'd) operation = ('a, 'b, 'c, 'd) operation + end) + +type ('before, 'after, 'value) program = ('before, 'after, 'value) Program.t + +type (_, _, _) node = + | Return : 'a -> ('s, 's, 'a) node + | Abort : error -> ('s, 'final, 'a) node + | Perform : + ('before, 'after, 'failed, 'a) operation + * ('a -> ('after, 'final, 'b) program) + * (error -> ('failed, 'final, 'b) program) + -> ('before, 'final, 'b) node + +type policy_config = + { sync_retry : retry_policy + ; asset_retry : retry_policy + ; no_progress_seconds : float + } + +let return = Program.return +let abort = Program.abort +let bind = Program.bind +let perform = Program.perform + +let view + : type before after value. (before, after, value) program -> (before, after, value) node + = + fun program -> + match Program.view program with + | Program.Return value -> Return value + | Program.Abort error -> Abort error + | Program.Perform (operation, success, failure) -> Perform (operation, success, failure) +;; + +let task_return value = Task (Task_return value) +let task_bind task continuation = Task (Task_bind (task, continuation)) +let task_fail error = Task (Task_fail error) + +let task_perform operation ~success ~failure = + Task (Task_perform (operation, success, failure)) +;; + +let with_scope body = Task (Task_scope body) +let spawn scope task = Task (Task_spawn (scope, task)) +let spawn_asset scope receipt task = Task (Task_spawn_asset (scope, receipt, task)) +let equal_child first second = first == second +let join child = Task (Task_join child) +let cancel_child child = Task (Task_cancel child) +let task_view (Task node) = node +let run_task task = Run_task task +let restore = Restore +let install_initial_network gate = Install_initial_network gate +let set_network gate = Set_network gate +let database operation = Database operation +let transport operation = Transport operation +let crypto operation = Crypto operation +let assets operation = Assets operation +let observe = Observe +let begin_drain phase = Begin_drain phase +let finish_retirement = Finish_retirement +let release_file file = Release_file file +let release_lease lease = Release_lease lease +let release_snapshot snapshot = Release_snapshot snapshot +let invalid field = Error (Invalid_input field) +let finite value = Float.is_finite value +let nonempty value = String.length value > 0 && String.trim value = value + +let identity ~origin ~account ~graph ~account_generation ~graph_generation = + if + (not (nonempty account && nonempty graph)) + || account_generation < 0L + || graph_generation < 0L + then invalid "identity" + else ( + try + let uri = Uri.of_string origin in + match Uri.scheme uri, Uri.host uri with + | Some scheme, Some host + when String.lowercase_ascii scheme = "https" + && nonempty host + && Uri.userinfo uri = None + && Uri.query uri = [] + && Uri.fragment uri = None + && (Uri.path uri = "" || Uri.path uri = "/") + && + match Uri.port uri with + | None -> true + | Some port -> port > 0 && port <= 65535 -> + let port = + match Uri.port uri with + | Some 443 -> None + | port -> port + in + let origin = + Uri.to_string + (Uri.make ~scheme:"https" ~host:(String.lowercase_ascii host) ?port ()) + in + Ok { origin; account; graph; account_generation; graph_generation } + | _ -> invalid "HTTPS origin" + with + | _ -> invalid "HTTPS origin") +;; + +let limits + ~maximum_items + ~maximum_bytes + ~maximum_pending + ~maximum_resources + ~steps_per_turn + ~maximum_tasks + ~maximum_messages + ~maximum_message_bytes + ~maximum_demands + ~maximum_active_assets + ~foreground_reserve + ~foreground_demand_reserve + ~foreground_descriptor_reserve_bytes + ~asset_cache_budget_bytes + ~asset_maximum_file_bytes + ~asset_pending_budget_bytes + ~snapshot_maximum_artifact_bytes + ~snapshot_staging_budget_bytes + ~snapshot_scratch_budget_bytes + = + let bounded value maximum = value > 0 && value <= maximum in + if + not + (bounded maximum_items 4096 + && bounded maximum_pending 4096 + && bounded maximum_messages 4096 + && bounded maximum_demands 4096 + && bounded maximum_bytes 16777216 + && bounded maximum_message_bytes 16777216 + && bounded maximum_resources 1024 + && bounded steps_per_turn 65536 + && bounded maximum_tasks 256 + && bounded maximum_active_assets maximum_pending + && maximum_tasks >= maximum_active_assets + 4 + && maximum_pending >= maximum_active_assets + 4 + && bounded foreground_reserve maximum_active_assets + && bounded foreground_demand_reserve maximum_demands + && bounded foreground_descriptor_reserve_bytes maximum_message_bytes + && maximum_messages >= foreground_demand_reserve + 3 + && asset_cache_budget_bytes >= 0L + && asset_pending_budget_bytes >= 0L + && asset_maximum_file_bytes >= 0 + && asset_maximum_file_bytes <= 104857600 + && snapshot_maximum_artifact_bytes > 0L + && snapshot_staging_budget_bytes > 0L + && snapshot_scratch_budget_bytes > 0L) + then invalid "limits" + else + Ok + { maximum_items + ; maximum_bytes + ; maximum_pending + ; maximum_resources + ; steps_per_turn + ; maximum_tasks + ; maximum_messages + ; maximum_message_bytes + ; maximum_demands + ; maximum_active_assets + ; foreground_reserve + ; foreground_demand_reserve + ; foreground_descriptor_reserve_bytes + ; asset_cache_budget_bytes + ; asset_maximum_file_bytes + ; asset_pending_budget_bytes + ; snapshot_maximum_artifact_bytes + ; snapshot_staging_budget_bytes + ; snapshot_scratch_budget_bytes + } +;; + +let retry_policy ~initial_seconds ~maximum_seconds ~multiplier ~maximum_attempts = + if + not + (finite initial_seconds + && finite maximum_seconds + && finite multiplier + && initial_seconds > 0. + && maximum_seconds >= initial_seconds + && multiplier >= 1. + && maximum_attempts >= 0 + && maximum_attempts <= 10) + then invalid "retry policy" + else Ok { initial_seconds; maximum_seconds; multiplier; maximum_attempts } +;; + +let policy_config ~sync_retry ~asset_retry ~no_progress_seconds = + if not (finite no_progress_seconds && no_progress_seconds > 0.) + then invalid "no-progress delay" + else Ok { sync_retry; asset_retry; no_progress_seconds } +;; + +let resource ~owner ~token = + if nonempty token + then Ok { owner; token; release = resource_release () } + else invalid "resource token" +;; + +let snapshot ~resource ~through ~checksum ~encrypted = + if nonempty checksum + then Ok { resource; through; checksum; encrypted } + else invalid "snapshot checksum" +;; + +let connection ~owner ~token ~generation = + if nonempty token && generation >= 0L + then Ok (Connection (owner, token, generation, resource_release ())) + else invalid "connection" +;; + +let unlock_input ~wrapped_graph_key = + if String.length wrapped_graph_key > 0 + then Ok wrapped_graph_key + else invalid "wrapped graph key" +;; + +let wrapped_graph_key input = input +let asset_file file asset = Ok { file; asset } + +let asset_lease lease source = + if lease.owner <> source.file.owner + then Error (Identity_mismatch "asset lease") + else if lease.token = source.file.token + then invalid "duplicate resource reference" + else Ok { lease; source } +;; + +let demand_id value = + if nonempty value && String.length value <= 256 then Ok value else invalid "demand ID" +;; + +let compare_demand_id = String.compare + +module Producer = struct + type owner = { stamp : identity ref } + + let create identity = { stamp = ref identity } + let scope owner = { scope_owner = owner.stamp; scope_live = true; registrations = [] } + let original_scope owner scope = scope.scope_owner == owner.stamp + + let check_scope owner scope = + if original_scope owner scope && scope.scope_live then Ok () else Error Stale_context + ;; + + let close_scope owner scope = + if original_scope owner scope + then ( + scope.scope_live <- false; + List.iter (fun (_, release) -> release ()) scope.registrations; + scope.registrations <- []; + Ok ()) + else Error Stale_context + ;; + + let child owner scope task ~poll = + match check_scope owner scope with + | Error error -> Error error + | Ok () -> + let child = + { child_scope = scope + ; registration = ref () + ; child_task = Some task + ; child_state = Pending_child poll + } + in + scope.registrations + <- ( child.registration + , fun () -> + child.child_state <- Consumed_child; + child.child_task <- None ) + :: scope.registrations; + Ok child + ;; + + let check_child owner scope child = + match check_scope owner scope with + | Error error -> Error error + | Ok () -> + if child.child_scope == scope && Option.is_some child.child_task + then Ok () + else Error Stale_context + ;; + + let child_result owner scope child = + match check_child owner scope child with + | Error error -> Error error + | Ok () -> + (match child.child_state with + | Consumed_child -> Error Stale_context + | Completed_child result -> Ok (Some result) + | Pending_child poll -> + let result = poll () in + (match result with + | None -> () + | Some value -> child.child_state <- Completed_child value); + Ok result) + ;; + + let forget_child owner scope child = + if original_scope owner scope && child.child_scope == scope + then ( + child.child_state <- Consumed_child; + child.child_task <- None; + scope.registrations + <- List.filter + (fun (registration, _) -> registration != child.registration) + scope.registrations; + Ok ()) + else Error Stale_context + ;; + + let make owner scope build = + match check_scope owner scope with + | Error error -> Error error + | Ok () -> Ok (build { handle_scope = scope }) + ;; + + let demand_receipt owner scope = make owner scope (fun handle -> Demand_receipt handle) + let unlock_receipt owner scope = make owner scope (fun handle -> Unlock_receipt handle) + let outbox_check owner scope = make owner scope (fun handle -> Outbox_check handle) + + let notifications owner scope = + make owner scope (fun handle -> Notifications_handle handle) + ;; + + let capacity_revision owner scope ~sequence = + match check_scope owner scope with + | Error error -> Error error + | Ok () -> Ok { capacity_scope = scope; sequence } + ;; + + let capacity_sequence owner scope revision = + match check_scope owner scope with + | Error error -> Error error + | Ok () -> + if revision.capacity_scope == scope + then Ok revision.sequence + else Error Stale_context + ;; + + let check_handle owner scope handle = + match check_scope owner scope with + | Error error -> Error error + | Ok () -> if handle.handle_scope == scope then Ok () else Error Stale_context + ;; + + let check_demand_receipt owner scope (Demand_receipt handle) = + check_handle owner scope handle + ;; + + let check_unlock_receipt owner scope (Unlock_receipt handle) = + check_handle owner scope handle + ;; + + let check_outbox_check owner scope (Outbox_check handle) = + check_handle owner scope handle + ;; + + let check_notifications owner scope (Notifications_handle handle) = + check_handle owner scope handle + ;; + + let demand_output owner scope demand ~id availability = + match check_demand_receipt owner scope demand with + | Error error -> Error error + | Ok () -> + Ok { delivery = Delivery (owner.stamp, ref false); id; demand; availability } + ;; + + let check_delivery owner (Delivery (stamp, _)) = + if owner.stamp == stamp then Ok () else Error Stale_context + ;; + + let delivery_released owner (Delivery (stamp, released)) = + if owner.stamp == stamp then Ok !released else Error Stale_context + ;; + + let acknowledge_delivery owner (Delivery (stamp, released)) = + if owner.stamp == stamp + then ( + released := true; + Ok ()) + else Error Stale_context + ;; + + let reference_registration = function + | Snapshot_reference snapshot -> + ( snapshot.resource.release + , Marshal.to_string (0, snapshot.through, snapshot.checksum, snapshot.encrypted) [] + ) + | File_reference file -> file.file.release, Marshal.to_string (1, file.asset) [] + | Lease_reference lease -> + ( lease.lease.release + , Marshal.to_string + (2, lease.source.file.owner, lease.source.file.token, lease.source.asset) + [] ) + | Connection_reference (Connection (identity, token, generation, release)) -> + release, Marshal.to_string (3, identity, token, generation) [] + ;; + + let same_reference_registration left right = + let left, _ = reference_registration left + and right, _ = reference_registration right in + left == right + ;; + + let reference_state owner reference = + let release, metadata = reference_registration reference in + match release.release_owner with + | None -> Ok `Unregistered + | Some stamp when stamp == owner.stamp && release.release_metadata = Some metadata -> + Ok (if release.release_acknowledged then `Released else `Retained) + | Some _ -> Error Stale_context + ;; + + let register_reference owner reference = + match reference_state owner reference with + | Error error -> Error error + | Ok `Released -> Error Stale_context + | Ok `Retained -> Ok () + | Ok `Unregistered -> + let release, metadata = reference_registration reference in + release.release_owner <- Some owner.stamp; + release.release_metadata <- Some metadata; + Ok () + ;; + + let acknowledge_reference owner reference = + match reference_state owner reference with + | Error error -> Error error + | Ok `Unregistered -> Error Stale_context + | Ok (`Retained | `Released) -> + let release, _ = reference_registration reference in + release.release_acknowledged <- true; + Ok () + ;; +end + +let error_message = function + | Invalid_input message -> "Invalid input: " ^ message + | Identity_mismatch message -> "Identity mismatch: " ^ message + | Stale_context -> "Stale context" + | Network_closed -> "Network closed" + | Network_failure message -> "Network failure: " ^ message + | Authentication_failure message -> "Authentication failure: " ^ message + | Locked -> "Locked" + | Crypto_failure message -> "Crypto failure: " ^ message + | Protocol_failure message -> "Protocol failure: " ^ message + | Storage_failure message -> "Storage failure: " ^ message + | Cancelled -> "Cancelled" + | Outcome_unknown message -> "Outcome unknown: " ^ message + | Cleanup_failure _ -> "Cleanup failure" + | Capacity_exceeded message -> "Capacity exceeded: " ^ message + | Sync_read_failure _ -> "Sync read failure" + | Outbox_failure _ -> "Outbox transition failure" + | Authoritative_failure error -> + let module T = Overlay.Types in + let module Diagnostic = Logseq_db_types.Authoritative_diagnostic in + let classified reason = "Authoritative transition failed: " ^ reason in + let diagnostic fallback source = + match Diagnostic.decode source with + | Some detail -> Diagnostic.message detail + | None -> classified fallback + in + (match error with + | T.Authoritative_database_closed -> classified "database closed during preparation" + | Authoritative_sync_token_conflict -> classified "preparation sync token conflict" + | Authoritative_cursor_discontinuous -> classified "server cursor discontinuous" + | Authoritative_checksum_mismatch -> classified "checksum mismatch" + | Authoritative_crypto_required -> classified "decryption result required" + | Authoritative_crypto_unexpected -> classified "unexpected decryption result" + | Authoritative_crypto_error error -> + classified + (match error with + | Crypto_result_missing_item _ -> "decryption result missing item" + | Crypto_result_extra_item _ -> "decryption result extra item" + | Crypto_result_duplicate_item _ -> "decryption result duplicate item" + | Crypto_result_reordered -> "decryption result reordered" + | Crypto_result_stale -> "decryption result stale" + | Crypto_result_limit_exceeded -> "decryption result limit exceeded") + | Authoritative_decode_failed source -> + diagnostic "transaction decode rejected" source + | Authoritative_origin_unresolved -> classified "origin unresolved" + | Authoritative_preparation_consumed -> classified "preparation already consumed" + | Authoritative_prepare_busy -> classified "preparation busy" + | Authoritative_integrity_failure source -> + diagnostic "integrity validation rejected" source + | Authoritative_commit_database_closed -> classified "database closed during commit" + | Authoritative_commit_token_conflict -> classified "commit sync token conflict" + | Authoritative_commit_generation_invalidated -> + classified "commit generation invalidated" + | Authoritative_commit_busy -> classified "commit busy" + | Authoritative_commit_persistence_failed _ -> classified "commit persistence failed" + | Authoritative_commit_fatal_state _ -> classified "commit fatal state") + | Activation_failure _ -> "Snapshot activation failure" + | Asset_failure diagnostic -> Logseq_db_types.Failure_diagnostic.message diagnostic + | Unexpected_failure message -> "Unexpected failure: " ^ message +;; + +let ( let* ) = task_bind +let task_effect operation = task_perform operation ~success:task_return ~failure:task_fail + +let attempt operation = + task_perform + operation + ~success:(fun value -> task_return (Ok value)) + ~failure:(fun error -> task_return (Error error)) +;; + +let rec capacity_task_effect : type value. value task_operation -> value task = + fun operation -> + let* revision = task_effect Capacity_revision in + let* result = attempt operation in + match result with + | Ok value -> task_return value + | Error (Capacity_exceeded _) -> + let* () = task_effect (Await_capacity_changed revision) in + capacity_task_effect operation + | Error error -> task_fail error +;; + +let rec capture_task : type value. value task -> (value, error) result task = + fun task -> + match task_view task with + | Task_return value -> task_return (Ok value) + | Task_fail error -> task_return (Error error) + | Task_perform (operation, success, failure) -> + task_perform + operation + ~success:(fun value -> capture_task (success value)) + ~failure:(fun error -> capture_task (failure error)) + | Task_bind (first, continuation) -> + let* result = capture_task first in + (match result with + | Ok value -> capture_task (continuation value) + | Error error -> task_return (Error error)) + | Task_scope body -> with_scope (fun scope -> capture_task (body scope)) + | Task_spawn (scope, body) -> + let* value = spawn scope body in + task_return (Ok value) + | Task_spawn_asset (scope, receipt, body) -> + let* value = spawn_asset scope receipt body in + task_return (Ok value) + | Task_join child -> + let* value = join child in + task_return (Ok value) + | Task_cancel child -> + let* () = cancel_child child in + task_return (Ok ()) +;; + +let finalize primary cleanup = + let* cleaned = capture_task cleanup in + match primary, cleaned with + | Ok value, Ok () -> task_return value + | Error error, Ok () -> task_fail error + | Ok _, Error error -> + task_fail (Cleanup_failure { primary = None; failures = Rrbvec.singleton error }) + | Error error, Error cleanup -> + task_fail + (Cleanup_failure { primary = Some error; failures = Rrbvec.singleton cleanup }) +;; + +let retry_delay retry attempt = + let rec grow remaining seconds = + if remaining = 0 || seconds >= retry.maximum_seconds + then min seconds retry.maximum_seconds + else grow (remaining - 1) (min retry.maximum_seconds (seconds *. retry.multiplier)) + in + grow attempt retry.initial_seconds +;; + +module Types = Overlay.Types +module Cursor = Types.Server_cursor + +let request connection operation = + capacity_task_effect (Task_transport (On_connection (connection, operation))) +;; + +let inspect () = capacity_task_effect (Task_database Inspect) +let checkpoint view = Types.sync_view_checkpoint view +let token view = Types.sync_view_token view +let greatest first second = if Cursor.compare first second >= 0 then first else second + +let rec settle_reply batch reply = + let* view = inspect () in + let batch_id = Types.submission_batch_id batch in + let transition = + match reply with + | Accepted barrier -> Types.Accept_group { batch_id; barrier } + | Rejected resolution -> Types.Reject_group { batch_id; resolution } + in + let* result = + capture_task + (capacity_task_effect (Task_database (Prepare_submission (token view, transition)))) + in + match result with + | Ok _ -> task_return () + | Error + (Outbox_failure + (Types.Outbox_sync_token_conflict | Types.Outbox_commit_token_conflict)) -> + let* () = task_effect Yield in + settle_reply batch reply + | Error error -> task_fail error +;; + +let rec frozen_batch view transition = + let* outcome = + capture_task + (capacity_task_effect (Task_database (Prepare_submission (token view, transition)))) + in + match outcome with + | Error + (Outbox_failure + (Types.Outbox_sync_token_conflict | Types.Outbox_commit_token_conflict)) -> + let* view = inspect () in + frozen_batch view transition + | Error error -> task_fail error + | Ok commit -> + (match commit.Types.submission_batch with + | Some batch -> task_return batch + | None -> + task_fail (Protocol_failure "submission preparation returned no durable wires")) +;; + +let submitted view = + List.find_map + (fun (descriptor : Types.submission_descriptor) -> + match descriptor.state with + | Types.Submitted batch -> Some batch + | _ -> None) + (Types.sync_view_submissions view) +;; + +let queued view = + List.find_opt + (fun (descriptor : Types.submission_descriptor) -> + descriptor.state = Types.Queued && descriptor.dependency_eligible) + (Types.sync_view_submissions view) +;; + +let recover_submission connection view batch_id = + let* () = task_effect (Publish_status Service_submitting) in + let* batch = frozen_batch view (Types.Retry_group batch_id) in + let* reply = request connection (Push batch) in + settle_reply batch reply +;; + +let rec apply_batch config connection batch = + let* view = inspect () in + let* applied = + capture_task + (capacity_task_effect (Task_database (Apply_authoritative (token view, batch)))) + in + match applied with + | Error + (Authoritative_failure + ( Types.Authoritative_sync_token_conflict + | Types.Authoritative_commit_token_conflict )) -> + let* () = task_effect Yield in + apply_batch config connection batch + | Error error -> task_fail error + | Ok (Applied _) -> task_return () + | Ok (Deferred (Types.Await_submission_outcome batch_id)) -> + let* fresh = inspect () in + (match submitted fresh with + | Some current when Types.Submission_batch_id.equal current batch_id -> + let* () = recover_submission connection fresh batch_id in + apply_batch config connection batch + | _ -> + let* () = task_effect (Sleep config.no_progress_seconds) in + apply_batch config connection batch) +;; + +let pull_once ?head config connection view = + let* () = task_effect (Publish_status Service_pulling) in + let since = checkpoint view in + let* pulled = request connection (Pull since) in + match pulled with + | Pulled batch -> + if Cursor.compare (Types.authoritative_batch_through batch) since <= 0 + then task_fail (Protocol_failure "non-advancing authoritative batch") + else + let* batch = + match head with + | Some head + when Cursor.equal head.through (Types.authoritative_batch_through batch) -> + (match head.expected_checksum, Types.authoritative_batch_checksum batch with + | Some expected, Some actual when not (Types.Checksum.equal expected actual) -> + task_fail (Protocol_failure "Conflicting remote checkpoint checksums") + | Some expected, None -> + let transactions = Types.authoritative_batch_transactions batch in + let bytes = + List.fold_left + (fun total transaction -> + total + + Types.encoded_transaction_byte_length + (Types.authoritative_transaction_payload transaction)) + 0 + transactions + in + (match + Types.authoritative_batch + ~maximum_count:(List.length transactions) + ~maximum_bytes:bytes + ~transactions + ~through:head.through + ~checksum:(Some expected) + with + | Ok batch -> task_return batch + | Error _ -> task_fail (Protocol_failure "Invalid authoritative batch")) + | None, _ | Some _, Some _ -> task_return batch) + | None | Some _ -> task_return batch + in + let* () = apply_batch config connection batch in + task_return (Types.authoritative_batch_through batch) + | No_progress report -> + if (not (Cursor.equal report.since since)) || Cursor.compare report.through since < 0 + then task_fail (Protocol_failure "invalid no-progress baseline") + else + let* () = + match report.expected_checksum with + | Some expected when Cursor.equal report.through since -> + task_effect (Task_database (Verify_checksum (report.through, expected))) + | None | Some _ -> task_return () + in + let* () = task_effect (Sleep config.no_progress_seconds) in + task_return report.through +;; + +let send_batch ?head config connection batch = + let* () = task_effect (Publish_status Service_submitting) in + let* outcome = capture_task (request connection (Push batch)) in + match outcome with + | Ok reply -> + let* () = settle_reply batch reply in + let target = + match reply with + | Accepted barrier -> barrier.Types.through + | Rejected (Types.Stale { through }) -> through + | Rejected (Types.Definitive { partition; _ }) -> + (match partition.acceptance_barrier with + | Some barrier -> barrier.through + | None -> Types.submission_batch_t_before batch) + in + task_return target + | Error (Outcome_unknown _) -> + let* view = inspect () in + pull_once ?head config connection view + | Error error -> task_fail error +;; + +let verify_head_checksum head ~through ~checksum = + match head.expected_checksum with + | Some expected + when Cursor.equal head.through through + && not (String.equal (Types.Checksum.to_string expected) checksum) -> + task_fail (Protocol_failure "Remote checkpoint checksum mismatch") + | None | Some _ -> task_return () +;; + +let rec round config connection head target check = + let* inspected = capture_task (inspect ()) in + let* view = + match inspected with + | Ok view -> task_return view + | Error error -> + let cleanup = + match check with + | None -> task_return () + | Some check -> task_effect (Abandon_outbox_check check) + in + finalize (Error error) cleanup + in + let recovery = submitted view in + let fresh = queued view in + let* () = + match check with + | None -> task_return () + | Some check -> task_effect (Acknowledge_outbox_check check) + in + let* () = + match head.expected_checksum with + | Some expected when Cursor.equal head.through (checkpoint view) -> + task_effect (Task_database (Verify_checksum (head.through, expected))) + | None | Some _ -> task_return () + in + match recovery with + | Some batch_id -> + let* batch = frozen_batch view (Types.Retry_group batch_id) in + let* through = send_batch ~head config connection batch in + let* () = task_effect Yield in + round config connection head (greatest target through) None + | None when Cursor.compare (checkpoint view) target < 0 -> + let* through = pull_once ~head config connection view in + let* () = task_effect Yield in + round config connection head (greatest target through) None + | None -> + (match fresh with + | Some descriptor -> + let* batch = frozen_batch view (Types.Submit_group [ descriptor.mutation_id ]) in + let* through = send_batch ~head config connection batch in + let* head = request connection Head in + round config connection head (greatest through (greatest target head.through)) None + | None -> + let unsettled = + List.exists + (fun (descriptor : Types.submission_descriptor) -> + match descriptor.state with + | Types.Accepted_pending_authoritative _ + | Types.Stale_rejected_pending_authoritative _ -> true + | Types.Submitted _ | Types.Queued | Types.Blocked -> false) + (Types.sync_view_submissions view) + in + if unsettled + then + let* through = pull_once ~head config connection view in + round config connection head (greatest target through) None + else task_return (checkpoint view)) +;; + +let rec activate_initial config connection head = + let* observation = task_effect Task_observe in + match observation.checkpoint with + | Some _ -> task_return () + | None when not head.snapshot_available -> + task_fail (Protocol_failure "remote snapshot unavailable") + | None -> + let* check = task_effect Await_outbox_changed in + let* fetched = capture_task (request connection (Fetch_snapshot head.through)) in + let* staged = + match fetched with + | Ok staged -> task_return staged + | Error error -> finalize (Error error) (task_effect (Abandon_outbox_check check)) + in + let* activation = + capture_task + (let* () = task_effect (Acknowledge_outbox_check check) in + let* () = + verify_head_checksum head ~through:staged.through ~checksum:staged.checksum + in + task_effect (Task_database (Activate_snapshot staged))) + in + let* activation = + finalize activation (task_effect (Task_database (Release_snapshot staged))) + in + (match activation with + | Activated _ -> task_return () + | Activation_deferred _ -> activate_initial config connection head) +;; + +type demand_wait = + | Demand_complete + | Wait_network + | Wait_unlock + | Wait_remote + | Wait_repair + | Wait_network_repair + | Retry_asset of asset_failure_reason + +type demand_progress = + { asset_wait : demand_wait + ; asset_attempts : int + ; asset_retries : int + } + +let asset_reason = function + | Network_failure message -> Asset_network message + | Network_closed -> Asset_network_closed + | Authentication_failure message -> Asset_authentication message + | Locked -> Asset_locked + | Cancelled -> Asset_cancelled + | error -> Asset_other error +;; + +let publish receipt state = capacity_task_effect (Publish_demand (receipt, state)) + +let release_asset file lease = + let* lease_result = + match lease with + | None -> task_return (Ok ()) + | Some lease -> attempt (Task_assets (Release_lease lease)) + in + let* file_result = attempt (Task_assets (Release_file file)) in + match lease_result, file_result with + | Ok (), Ok () -> task_return () + | Error error, Ok () | Ok (), Error error -> task_fail error + | Error first, Error second -> + task_fail + (Cleanup_failure { primary = Some first; failures = Rrbvec.singleton second }) +;; + +let deliver_file receipt file = + let* retained = attempt (Task_assets (Retain file)) in + match retained with + | Error error -> finalize (Error error) (release_asset file None) + | Ok lease -> + let* delivered = capture_task (publish receipt (Demand_available (file, lease))) in + (match delivered with + | Ok () -> task_return Demand_complete + | Error error -> finalize (Error error) (release_asset file (Some lease))) +;; + +let demand_task config receipt ~attempts ~retries ~retry = + let* () = + match retry with + | None -> publish receipt Demand_queued + | Some reason -> + let* () = + publish receipt (Demand_failed { reason; attempts; retry_scheduled = true }) + in + task_effect (Sleep (retry_delay config.asset_retry retries)) + in + let* report = capacity_task_effect (Download_demand receipt) in + let attempts = attempts + report.attempts in + let retries = retries + if retry <> None && report.attempts > 0 then 1 else 0 in + let* wait = + match report.result with + | Ok (Available file) -> deliver_file receipt file + | Ok Waiting_remote -> + let* () = publish receipt Demand_waiting_remote in + task_return Wait_remote + | Ok Missing_local -> + let* () = publish receipt Demand_missing_local in + task_return Wait_repair + | Error Asset_network_closed -> + let* () = publish receipt Demand_waiting_network in + task_return Wait_network + | Error Asset_locked -> + let* () = publish receipt Demand_waiting_unlock in + task_return Wait_unlock + | Error reason -> + let* () = + publish receipt (Demand_failed { reason; attempts; retry_scheduled = false }) + in + (match reason with + | Asset_network _ + when report.attempts > 0 && retries < config.asset_retry.maximum_attempts -> + task_return (Retry_asset reason) + | Asset_network _ -> task_return Wait_network_repair + | _ -> task_return Wait_repair) + in + let* () = + match wait with + | Retry_asset _ -> task_return () + | _ -> task_effect (Finish_demand receipt) + in + task_return + { asset_wait = wait + ; asset_attempts = attempts + ; asset_retries = (if wait = Demand_complete then 0 else retries) + } +;; + +type policy_event = + | Policy_connected of connection + | Policy_head of remote_head + | Policy_round of Types.server_cursor + | Policy_received of received + | Policy_outbox of outbox_check + | Policy_timer + | Policy_capacity + | Policy_demand of demand_progress + | Policy_unlocked of key_status + +type job_kind = + | Connecting_job + | Head_job + | Round_job + | Receive_job + | Outbox_job + | Retry_job + | Capacity_job + | Unlock_job + | Demand_job of demand_id * demand_receipt + +type policy_job = + { kind : job_kind + ; child : policy_event child + } + +type policy_demand = + { demand_id : demand_id + ; receipt : demand_receipt + ; change : demand_change + ; wait : demand_wait option + ; attempts : int + ; retries : int + } + +type policy_state = + { jobs : policy_job list + ; connection : connection option + ; head : remote_head option + ; target : Types.server_cursor option + ; checked : outbox_check option + ; initialized : bool + ; dirty : bool + ; failure : error option + ; retries : int + ; retry_pending : bool + ; backoff : (float * error) option + ; unlock_pending : unlock_receipt option + ; gate : network_gate + ; keys : key_status + ; demands : policy_demand list + } + +let same_kind first second = + match first, second with + | Connecting_job, Connecting_job + | Head_job, Head_job + | Round_job, Round_job + | Receive_job, Receive_job + | Outbox_job, Outbox_job + | Retry_job, Retry_job + | Capacity_job, Capacity_job + | Unlock_job, Unlock_job -> true + | Demand_job (first_id, first_receipt), Demand_job (second_id, second_receipt) -> + compare_demand_id first_id second_id = 0 && first_receipt == second_receipt + | _ -> false +;; + +let has_job state kind = List.exists (fun job -> same_kind job.kind kind) state.jobs + +let network_open = function + | Open _ -> true + | Closed -> false +;; + +let epoch = function + | Closed -> None + | Open { authorization_epoch } -> Some authorization_epoch +;; + +let sync_retryable = function + | Network_failure _ -> true + | _ -> false +;; + +let add_job scope state kind task = + let* _revision = task_effect Capacity_revision in + let* spawned = + match kind with + | Demand_job (_, receipt) -> spawn_asset scope receipt task + | _ -> spawn scope task + in + match spawned with + | Ok child -> task_return { state with jobs = state.jobs @ [ { kind; child } ] } + | Error (Capacity_exceeded _) -> task_return state + | Error error -> task_fail error +;; + +let mapped task constructor = + let* value = task in + task_return (constructor value) +;; + +let schedule config scope state = + let* state = + if has_job state Capacity_job + then task_return state + else + let* revision = task_effect Capacity_revision in + let* next = + add_job + scope + state + Capacity_job + (let* () = task_effect (Await_capacity_changed revision) in + task_return Policy_capacity) + in + if has_job next Capacity_job + then task_return next + else task_fail (Capacity_exceeded "reserved policy control task") + in + let* state = + match state.unlock_pending with + | Some receipt when not (has_job state Unlock_job) -> + let* next = + add_job + scope + state + Unlock_job + (mapped (task_effect (Consume_unlock receipt)) (fun keys -> + Policy_unlocked keys)) + in + if has_job next Unlock_job + then task_return { next with unlock_pending = None } + else task_return next + | _ -> task_return state + in + let* state = + match state.backoff with + | Some (seconds, _) when network_open state.gate && not (has_job state Retry_job) -> + let* next = + add_job + scope + state + Retry_job + (let* () = task_effect (Sleep seconds) in + task_return Policy_timer) + in + if has_job next Retry_job + then task_return { next with backoff = None } + else task_return next + | _ -> task_return state + in + let* state = + if + (not (network_open state.gate)) + || state.keys = Key_locked + || state.failure <> None + || state.backoff <> None + || has_job state Retry_job + then task_return state + else ( + match state.connection with + | None -> + if has_job state Connecting_job + then task_return state + else + let* () = task_effect (Publish_status Service_connecting) in + add_job + scope + state + Connecting_job + (mapped (capacity_task_effect (Task_transport Connect)) (fun connection -> + Policy_connected connection)) + | Some connection -> + let* state = + if has_job state Receive_job + then task_return state + else + add_job + scope + state + Receive_job + (mapped (task_effect (Task_transport (Receive connection))) (fun received -> + Policy_received received)) + in + (match state.head with + | None -> + if has_job state Head_job + then task_return state + else + add_job + scope + state + Head_job + (mapped (request connection Head) (fun head -> Policy_head head)) + | Some head -> + let* state = + if + state.initialized + && (not (has_job state Outbox_job)) + && state.checked = None + then + add_job + scope + state + Outbox_job + (mapped (task_effect Await_outbox_changed) (fun check -> + Policy_outbox check)) + else task_return state + in + if (state.dirty || state.checked <> None) && not (has_job state Round_job) + then ( + let target = + match state.target with + | Some target -> target + | None -> head.through + in + let task = + let* () = task_effect (Publish_status Service_pulling) in + let* () = activate_initial config connection head in + mapped (round config connection head target state.checked) (fun cursor -> + Policy_round cursor) + in + let* next = add_job scope state Round_job task in + if has_job next Round_job + then task_return { next with dirty = false; checked = None } + else task_return next) + else task_return state)) + in + let rec demands state = function + | [] -> task_return state + | demand :: rest -> + let kind = Demand_job (demand.demand_id, demand.receipt) in + let retry = + match demand.wait with + | Some (Retry_asset reason) -> Some reason + | _ -> None + in + if (demand.wait <> None && retry = None) || has_job state kind + then demands state rest + else ( + let task = + match demand.change with + | Withdraw _ -> + let* () = publish demand.receipt Demand_revoked in + let* () = task_effect (Finish_demand demand.receipt) in + task_return + (Policy_demand + { asset_wait = Demand_complete; asset_attempts = 0; asset_retries = 0 }) + | Demand _ -> + mapped + (demand_task + config + demand.receipt + ~attempts:demand.attempts + ~retries:demand.retries + ~retry) + (fun progress -> Policy_demand progress) + in + let* next = add_job scope state kind task in + demands next rest) + in + demands state state.demands +;; + +let cancel_jobs jobs = + let rec cancel = function + | [] -> task_return () + | job :: rest -> + let* () = cancel_child job.child in + cancel rest + in + let rec collect failures = function + | [] -> + (match List.rev failures with + | [] -> task_return () + | failures -> + task_fail + (Cleanup_failure { primary = None; failures = Rrbvec.of_list failures })) + | job :: rest -> + let* result = join job.child in + let* cleanup = + match result with + | Ok (Policy_connected connection) -> + attempt (Task_transport (Retire_connection connection)) + | Ok (Policy_outbox check) -> attempt (Abandon_outbox_check check) + | Ok _ | Error _ -> task_return (Ok ()) + in + let failures = + match cleanup with + | Ok () -> failures + | Error error -> error :: failures + in + let failures = + match result with + | Ok _ | Error Cancelled | Error Network_closed | Error Stale_context -> failures + | Error error -> error :: failures + in + collect failures rest + in + let* () = cancel jobs in + collect [] jobs +;; + +let abandon = function + | None -> task_return () + | Some check -> task_effect (Abandon_outbox_check check) +;; + +let disconnect state = + let connection_jobs, other_jobs = + List.partition + (fun job -> + match job.kind with + | Connecting_job | Head_job | Round_job | Receive_job | Outbox_job | Retry_job -> + true + | Capacity_job | Unlock_job | Demand_job _ -> false) + state.jobs + in + let* cancelled = capture_task (cancel_jobs connection_jobs) in + let* () = abandon state.checked in + let cleanup = + match state.connection with + | None -> task_return () + | Some connection -> task_effect (Task_transport (Retire_connection connection)) + in + let* () = finalize cancelled cleanup in + task_return + { state with + jobs = other_jobs + ; connection = None + ; head = None + ; target = None + ; checked = None + ; initialized = false + ; dirty = true + ; backoff = None + ; retry_pending = false + } +;; + +let block_sync config _scope state error = + let* state = disconnect state in + match error with + | Cleanup_failure _ -> task_fail error + | Network_closed -> + let* () = task_effect (Publish_status Service_waiting_network) in + task_return { state with failure = Some error } + | Locked -> + let* () = task_effect (Publish_status Service_waiting_unlock) in + task_return { state with failure = Some error; keys = Key_locked } + | error + when sync_retryable error + && network_open state.gate + && state.retries < config.sync_retry.maximum_attempts -> + let seconds = retry_delay config.sync_retry state.retries in + let attempt = state.retries + 1 in + let* () = + task_effect (Publish_status (Service_backoff { attempt; seconds; cause = error })) + in + task_return { state with backoff = Some (seconds, error) } + | error -> + let* () = task_effect (Publish_status (Service_blocked error)) in + task_return { state with failure = Some error } +;; + +let replace_demand state receipt change = + let id = + match change with + | Demand demand -> demand.id + | Withdraw id -> id + in + let removed, jobs = + List.partition + (fun job -> + match job.kind with + | Demand_job (current, _) -> compare_demand_id current id = 0 + | _ -> false) + state.jobs + in + let* () = cancel_jobs removed in + let demands = + List.filter (fun demand -> compare_demand_id demand.demand_id id <> 0) state.demands + in + task_return + { state with + jobs + ; demands = + demands + @ [ { demand_id = id; receipt; change; wait = None; attempts = 0; retries = 0 } ] + } +;; + +let resume_demands predicate state = + { state with + demands = + List.map + (fun demand -> + if predicate demand.wait then { demand with wait = None } else demand) + state.demands + } +;; + +let suspend_demands state = + let affected, remaining = + List.partition + (fun job -> + match job.kind with + | Demand_job _ -> true + | _ -> false) + state.jobs + in + let rec cancel = function + | [] -> task_return () + | job :: rest -> + let* () = cancel_child job.child in + cancel rest + in + let rec collect demands = function + | [] -> task_return { state with jobs = remaining; demands } + | job :: rest -> + let* result = join job.child in + let* id, receipt = + match job.kind with + | Demand_job (id, receipt) -> task_return (id, receipt) + | _ -> task_fail (Protocol_failure "mismatched asset cancellation") + in + let update demand = + if compare_demand_id id demand.demand_id <> 0 || receipt != demand.receipt + then demand + else ( + match result with + | Ok (Policy_demand progress) -> + let wait = + match progress.asset_wait with + | Retry_asset _ -> None + | wait -> Some wait + in + { demand with + wait + ; attempts = progress.asset_attempts + ; retries = progress.asset_retries + } + | Error Cancelled | Error Network_closed -> { demand with wait = None } + | _ -> { demand with wait = Some Wait_repair }) + in + (match result with + | Error (Cleanup_failure _ as error) -> task_fail error + | _ -> collect (List.map update demands) rest) + in + let* () = cancel affected in + collect state.demands affected +;; + +let policy_task config = + with_scope (fun scope -> + let* observation = task_effect Task_observe in + let initial = + { jobs = [] + ; connection = None + ; head = None + ; target = None + ; checked = None + ; initialized = false + ; dirty = true + ; failure = None + ; retries = 0 + ; retry_pending = false + ; backoff = None + ; unlock_pending = None + ; gate = observation.network + ; keys = observation.keys + ; demands = [] + } + in + let rec loop state = + let* state = schedule config scope state in + let children = Rrbvec.of_list (List.map (fun job -> job.child) state.jobs) in + let* wake = task_effect (Await_input_or_child children) in + match wake with + | Input Stop_requested -> + let* cancelled = capture_task (cancel_jobs state.jobs) in + let* () = abandon state.checked in + let cleanup = + match state.connection with + | None -> task_return () + | Some connection -> task_effect (Task_transport (Retire_connection connection)) + in + let* () = finalize cancelled cleanup in + task_effect (Publish_status Service_idle) + | Input (Demand_changed (receipt, change)) -> + let* state = replace_demand state receipt change in + loop state + | Input (Unlock_requested receipt) -> + loop { state with unlock_pending = Some receipt } + | Input (Network_changed gate) -> + let new_epoch = network_open gate && epoch gate <> epoch state.gate in + let* state = + if network_open gate + then task_return state + else + let* state = disconnect state in + suspend_demands state + in + let failure = + match state.failure with + | Some Network_closed when network_open gate -> None + | Some (Network_failure _) when new_epoch -> None + | failure -> failure + in + let state = + { state with + gate + ; failure + ; retries = (if new_epoch then 0 else state.retries) + ; dirty = true + } + in + let state = + if network_open gate + then + resume_demands + (function + | Some Wait_network -> true + | _ -> false) + state + else state + in + let state = + if new_epoch + then + { state with + demands = + List.map + (fun demand -> + match demand.wait with + | Some Wait_network_repair -> + { demand with wait = None; retries = 0 } + | _ -> demand) + state.demands + } + else state + in + let* () = + if not (network_open gate) + then task_effect (Publish_status Service_waiting_network) + else ( + match state.failure with + | Some Locked -> task_effect (Publish_status Service_waiting_unlock) + | Some error -> task_effect (Publish_status (Service_blocked error)) + | None -> task_effect Resample_outbox) + in + loop state + | Child_ready child -> + (match List.find_opt (fun job -> equal_child child job.child) state.jobs with + | None -> task_fail (Protocol_failure "foreign policy completion") + | Some job -> + let* result = join child in + let state = + { state with + jobs = + List.filter (fun job -> not (equal_child child job.child)) state.jobs + } + in + (match result with + | Error (Cleanup_failure _ as error) -> task_fail error + | Error error -> + (match job.kind with + | Demand_job (id, receipt) -> + let* delivered = + attempt + (Publish_demand + ( receipt + , Demand_failed + { reason = asset_reason error + ; attempts = 0 + ; retry_scheduled = false + } )) + in + (match delivered with + | Error (Cleanup_failure _ as error) -> task_fail error + | _ -> + let demands = + List.map + (fun demand -> + if compare_demand_id demand.demand_id id = 0 + then { demand with wait = Some Wait_repair } + else demand) + state.demands + in + loop { state with demands }) + | Unlock_job -> + let* () = task_effect (Publish_status (Service_blocked error)) in + loop state + | Capacity_job -> task_fail error + | _ -> + let retries = + if + same_kind job.kind Connecting_job + && state.retry_pending + && error <> Network_closed + && error <> Locked + then state.retries + 1 + else state.retries + in + let* state = block_sync config scope { state with retries } error in + loop state) + | Ok event -> + (match event with + | Policy_capacity -> loop state + | Policy_timer -> loop { state with retry_pending = true } + | Policy_connected connection -> + let* () = task_effect Resample_outbox in + loop + { state with + connection = Some connection + ; retries = (state.retries + if state.retry_pending then 1 else 0) + ; retry_pending = false + } + | Policy_head head -> + loop + { state with + head = Some head + ; target = Some head.through + ; dirty = true + } + | Policy_round cursor -> + let dirty = + state.dirty + || + match state.target with + | Some target -> Cursor.compare cursor target < 0 + | None -> false + in + let* () = + task_effect + (Publish_status (if dirty then Service_pulling else Service_online)) + in + loop { state with initialized = true; retries = 0; dirty } + | Policy_outbox check -> + loop { state with checked = Some check; dirty = true } + | Policy_unlocked keys -> + let failure = + match state.failure with + | Some Locked -> None + | failure -> failure + in + let state = + resume_demands + (function + | Some Wait_unlock -> true + | _ -> false) + state + in + let* () = task_effect Resample_outbox in + loop { state with keys; failure; dirty = true } + | Policy_demand progress -> + (match job.kind with + | Demand_job (id, _) -> + let demands = + List.map + (fun demand -> + if compare_demand_id demand.demand_id id = 0 + then + { demand with + wait = Some progress.asset_wait + ; attempts = progress.asset_attempts + ; retries = progress.asset_retries + } + else demand) + state.demands + in + let demands = + List.filter + (fun demand -> + match demand.change, demand.wait with + | Withdraw _, Some Demand_complete -> false + | _ -> true) + demands + in + loop { state with demands } + | _ -> task_fail (Protocol_failure "mismatched demand completion")) + | Policy_received received -> + (match received.message with + | Remote_changed head -> + let target = + match state.target with + | None -> head.through + | Some target -> greatest target head.through + in + let state = + resume_demands + (function + | Some Wait_remote -> true + | _ -> false) + state + in + loop { state with target = Some target; dirty = true } + | Remote_error message -> + let* state = + block_sync config scope state (Protocol_failure message) + in + loop state + | Connection_ended reason -> + let error = + match reason with + | Remote_closed -> Network_failure "remote closed" + | Network_lost message -> Network_failure message + | Access_revoked message -> Authentication_failure message + | Invalid_protocol message -> Protocol_failure message + | Locally_closed -> Network_closed + in + let* state = block_sync config scope state error in + loop state)))) + in + let* () = + task_effect + (Publish_status + (if not (network_open initial.gate) + then Service_waiting_network + else if initial.keys = Key_locked + then Service_waiting_unlock + else Service_idle)) + in + loop initial) +;; + +let sync_policy config ~initial_network = + perform Restore ~failure:abort ~success:(fun _ -> + perform (Install_initial_network initial_network) ~failure:abort ~success:(fun () -> + perform (Run_task (policy_task config)) ~success:return ~failure:abort)) +;; diff --git a/logseq_sync/lib/sync_dsl/dune b/logseq_sync/lib/sync_dsl/dune new file mode 100644 index 00000000..c4fc2bbf --- /dev/null +++ b/logseq_sync/lib/sync_dsl/dune @@ -0,0 +1,6 @@ +(library + (name logseq_sync_sync_dsl_impl) + (public_name logseq_sync.sync_dsl.impl) + (implements sync_dsl) + (modules core) + (libraries logseq_overlay_db logseq_db_types rrbvec uri logseq_sync_indexed_program)) diff --git a/logseq_sync/lib/sync_service/core.ml b/logseq_sync/lib/sync_service/core.ml new file mode 100644 index 00000000..81b791e0 --- /dev/null +++ b/logseq_sync/lib/sync_service/core.ml @@ -0,0 +1,334 @@ +module Dsl = Sync_dsl.Core +module Execute = Execute.Core + +type config = + { identity : Dsl.identity + ; limits : Dsl.limits + ; network : Dsl.network_gate + ; policy : Dsl.policy_config + } + +type ingress = + { outbox_committed : + owner:Execute.database_owner -> identity:Dsl.identity -> (unit, Dsl.error) result + ; submit_unlock : + identity:Dsl.identity + -> valid_for:float + -> Dsl.unlock_input + -> (Execute.ingress_ack, Dsl.error) result + ; update_demand : Dsl.demand_change -> (Execute.ingress_ack, Dsl.error) result + ; receive_demand : unit -> (Dsl.demand_output, Dsl.error) result + ; demand_status : Dsl.demand_delivery -> (Execute.delivery_status, Dsl.error) result + ; release_demand : Dsl.demand_delivery -> (unit, Dsl.error) result + } + +type 'clock dependencies = + { clock : 'clock Eio.Time.Mono.t + ; database_owner : Execute.database_owner + ; capabilities : Execute.capabilities + ; bind_ingress : ingress -> unit + } + +type error = Dsl.error + +exception Reentrant_close +exception Close_already_called + +exception + Close_failed of + { error : error option + ; cause : (exn * Printexc.raw_backtrace) option + } + +module State = Logseq_sync_service_lifecycle + +type lifecycle = State.lifecycle = + | Running + | Closing + | Closed + +type status = + { lifecycle : lifecycle + ; observation : Execute.runtime_observation + } + +exception Stop_service + +type close_result = (unit, error option * (exn * Printexc.raw_backtrace) option) result + +type t = + { runtime : Execute.t + ; sw : Eio.Switch.t + ; mutable network_join : Execute.network_retirement option + ; mutable control_failure : error option + ; mutable control_cause : (exn * Printexc.raw_backtrace) option + ; callback_key : unit Eio.Fiber.key + ; resolve_stop : unit Eio.Promise.u + ; finished : close_result Eio.Promise.t + ; mutable state : State.t + } + +let set_network t gate = + if State.lifecycle t.state <> Running + then Error Dsl.Stale_context + else ( + match gate with + | Dsl.Closed -> + (match Execute.close_network t.runtime with + | Error error -> Error error + | Ok retirement -> + (match t.network_join with + | Some _ -> () + | None -> + t.network_join <- Some retirement; + Eio.Fiber.fork_daemon ~sw:t.sw (fun () -> + (match Execute.await_network_retirement retirement with + | Ok () -> () + | Error error -> + t.control_failure <- Some error; + ignore (Eio.Promise.try_resolve t.resolve_stop () : bool)); + `Stop_daemon)); + Ok ()) + | Dsl.Open { authorization_epoch } -> + let joined = + match t.network_join with + | None -> Ok () + | Some retirement -> Execute.await_network_retirement retirement + in + (match joined with + | Error error -> Error error + | Ok () when State.lifecycle t.state <> Running -> Error Dsl.Stale_context + | Ok () -> + (match Execute.open_network t.runtime ~authorization_epoch with + | Error error -> Error error + | Ok () -> + t.network_join <- None; + Ok ()))) +;; + +let close t = + match + State.request_close + ~inside_callback:(Option.is_some (Eio.Fiber.get t.callback_key)) + t.state + with + | Error State.Reentrant -> raise Reentrant_close + | Error State.Already_called -> raise Close_already_called + | Ok state -> + t.state <- state; + Eio.Cancel.protect (fun () -> + (try + match Execute.close_network t.runtime with + | Ok _ -> () + | Error error -> t.control_failure <- Some error + with + | exn -> t.control_cause <- Some (exn, Printexc.get_raw_backtrace ())); + ignore (Eio.Promise.try_resolve t.resolve_stop () : bool)); + (match Eio.Promise.await t.finished with + | Ok () -> () + | Error (error, cause) -> raise (Close_failed { error; cause })) +;; + +let retirement : type state. state Dsl.phase -> (state, Dsl.retired, unit) Dsl.program = + fun phase -> + let finish () = + Dsl.perform Dsl.finish_retirement ~success:Dsl.return ~failure:Dsl.abort + in + match phase with + | Dsl.Retired -> Dsl.return () + | Dsl.Cold -> + Dsl.perform (Dsl.begin_drain Dsl.Cold_context) ~success:finish ~failure:Dsl.abort + | Dsl.Ready -> + Dsl.perform (Dsl.begin_drain Dsl.Ready_context) ~success:finish ~failure:Dsl.abort + | Dsl.Draining -> + Dsl.perform (Dsl.begin_drain Dsl.Draining_context) ~success:finish ~failure:Dsl.abort +;; + +let start ~sw ~config ~dependencies ~on_change = + match + Execute.create + ~sw + ~clock:dependencies.clock + ~database_owner:dependencies.database_owner + ~identity:config.identity + ~limits:config.limits + dependencies.capabilities + with + | Error error -> Error error + | Ok (runtime, cold) -> + let stop, resolve_stop = Eio.Promise.create () in + let finished, resolve_finished = Eio.Promise.create () in + let t = + { runtime + ; sw + ; network_join = None + ; control_failure = None + ; control_cause = None + ; callback_key = Eio.Fiber.create_key () + ; resolve_stop + ; finished + ; state = State.initial + } + in + let context = ref (Execute.Context (Dsl.Cold, cold)) in + let cancel_running = ref None in + let failure = ref None in + let record_error error = + failure + := Some + (match !failure with + | None -> error + | Some primary -> + Dsl.Cleanup_failure + { primary = Some primary; failures = Rrbvec.of_list [ error ] }) + in + let callback_failed = ref false in + let invoke_callback callback = + try Eio.Fiber.with_binding t.callback_key () callback with + | exn -> + let backtrace = Printexc.get_raw_backtrace () in + callback_failed := true; + Printexc.raise_with_backtrace exn backtrace + in + let callback observation = + invoke_callback (fun () -> + on_change { lifecycle = State.lifecycle t.state; observation }) + in + let install_outcome + : type state value. state Dsl.phase -> (state, value) Execute.outcome -> unit + = + fun phase -> function + | Execute.Completed (next, _) -> context := Execute.Context (phase, next) + | Execute.Failed (next, error) -> + context := next; + (match State.lifecycle t.state, error with + | Closing, Dsl.Cancelled -> () + | (Running | Closing | Closed), _ -> record_error error) + in + let observed, resolve_observed = Eio.Promise.create () in + let observe () = + match Execute.observe_runtime runtime with + | Error error -> + record_error error; + raise Stop_service + | Ok initial -> + callback initial; + Eio.Promise.resolve resolve_observed (); + let rec loop previous = + match Execute.await_observation runtime ~after:previous.Execute.revision with + | Error error -> + record_error error; + raise Stop_service + | Ok observation -> + callback observation; + loop observation + in + loop initial + in + let policy () = + Eio.Promise.await observed; + if State.lifecycle t.state = Running + then ( + let program = Dsl.sync_policy config.policy ~initial_network:config.network in + match Execute.start runtime cold program with + | Error error -> + record_error error; + raise Stop_service + | Ok running -> + cancel_running + := Some + (fun () -> + (match Execute.cancel runtime running with + | Ok () | Error Dsl.Stale_context -> () + | Error error -> record_error error); + install_outcome Dsl.Ready (Execute.await running)); + install_outcome Dsl.Ready (Execute.await running); + cancel_running := None; + (match !failure with + | None -> () + | Some _ -> raise Stop_service)); + Eio.Promise.await stop + in + let cleanup () = + t.state <- State.begin_cleanup t.state; + (match !cancel_running with + | None -> () + | Some cancel -> cancel ()); + (match t.control_failure with + | None -> () + | Some error -> record_error error); + let (Execute.Context (phase, current)) = !context in + match Execute.start runtime current (retirement phase) with + | Error error -> record_error error + | Ok running -> install_outcome Dsl.Retired (Execute.await running) + in + Eio.Fiber.fork_daemon ~sw (fun () -> + let cause = + try + Eio.Fiber.yield (); + invoke_callback (fun () -> + dependencies.bind_ingress + { outbox_committed = + (fun ~owner ~identity -> + Execute.notify_outbox_changed runtime ~owner ~identity) + ; submit_unlock = + (fun ~identity ~valid_for input -> + Execute.submit_unlock runtime ~identity ~valid_for input) + ; update_demand = Execute.update_demand runtime + ; receive_demand = (fun () -> Execute.receive_demand runtime) + ; demand_status = Execute.demand_status runtime + ; release_demand = Execute.release_demand runtime + }); + Eio.Fiber.first + (fun () -> Eio.Fiber.both observe policy) + (fun () -> Eio.Promise.await stop); + None + with + | Stop_service -> None + | exn -> Some (exn, Printexc.get_raw_backtrace ()) + in + Eio.Cancel.protect (fun () -> + let cause = + try + cleanup (); + cause + with + | exn -> + let cleanup_cause = exn, Printexc.get_raw_backtrace () in + (match cause with + | None -> Some cleanup_cause + | Some original -> Some (Eio.Exn.combine original cleanup_cause)) + in + let cause = + match cause, t.control_cause with + | None, cause | cause, None -> cause + | Some primary, Some control -> Some (Eio.Exn.combine primary control) + in + let cause = + match !failure, cause with + | None, None -> + (try + t.state <- State.complete t.state; + (match Execute.observe_runtime runtime with + | Ok observation -> callback observation + | Error error -> + t.state <- State.begin_cleanup t.state; + record_error error); + None + with + | exn -> + t.state <- State.begin_cleanup t.state; + Some (exn, Printexc.get_raw_backtrace ())) + | _, cause -> cause + in + match !failure, cause with + | None, None -> Eio.Promise.resolve resolve_finished (Ok ()) + | error, cause -> + Eio.Promise.resolve resolve_finished (Error (error, cause)); + (match cause with + | Some (exn, backtrace) when !callback_failed -> + Printexc.raise_with_backtrace exn backtrace + | Some _ | None -> ())); + `Stop_daemon); + Ok t +;; diff --git a/logseq_sync/lib/sync_service/dune b/logseq_sync/lib/sync_service/dune new file mode 100644 index 00000000..d9957964 --- /dev/null +++ b/logseq_sync/lib/sync_service/dune @@ -0,0 +1,6 @@ +(library + (name logseq_sync_sync_service_impl) + (public_name logseq_sync.sync_service.impl) + (implements logseq_sync_sync_service) + (modules core) + (libraries eio sync_dsl execute rrbvec logseq_sync_service_lifecycle)) diff --git a/logseq_sync/lib/transport/INTEGRATION.md b/logseq_sync/lib/transport/INTEGRATION.md new file mode 100644 index 00000000..422194a0 --- /dev/null +++ b/logseq_sync/lib/transport/INTEGRATION.md @@ -0,0 +1,57 @@ +# Native transport integration (not activated in App) + +Keep the legacy effect-runner HTTP/snapshot entry points until App is switched. Do not install both transport dispatchers for one graph. This lane does not change App or the asset owner. + +## Graph owner wiring + +- Create `Logseq_sync_transport.Core.create_native` inside the graph's Eio switch. Pass the existing identity, bounded limits, deadlines, WebSocket liveness, absolute staging directory, encryption mode and epoch-aware token callback. +- Inject `~verify_snapshot:Logseq_overlay_db.Database.inspect_snapshot_storage`. Native transport owns downloading, compression layers, framed row import and scratch cleanup; Overlay owns restoring graph storage, encryption metadata and canonical checksum computation. The callback returns `Logseq_overlay_db.Types.Checksum.t`, never a raw database. +- Install `Logseq_sync_transport.Core.capabilities owner capabilities` once. Its dispatch uses the executor's `check_current`, `admit_network` and `handoff`. Use `Core.snapshot_source owner` for the exact original snapshot descriptor. Release with `Core.release_snapshot owner ~identity snapshot`; cancellation/fencing does not transfer the staging ledger. +- Transport must not depend on Worker, the legacy reducer, or Worker fixture directories. The local snapshot test data is an immutable transport fixture, not a runtime dependency. + +## Asset HTTP callbacks: exact methods + +Create a `Logseq_sync_transport.Http_client.t` with `Http_client.create ~backend ~token`; construct the native backend with `Http_client.native_backend ~sw ~clock ~network ~authenticator ~deadlines`. + +The `Logseq_sync_assets.Core.network` record has these callbacks: + +```ocaml +{ get = (fun ~identity ~asset ~version ~maximum_wire_bytes -> + Http_client.get_asset http ~scope ~epoch ~identity + ~asset:asset.Logseq_db_types.Asset_descriptor.uuid ~version ~maximum_wire_bytes) +; put = (fun ~identity ~asset ~version ~wire -> + Http_client.put_asset http ~scope ~epoch ~identity ~asset ~version ~wire) +} +``` + +Here `Http_client` means `Logseq_sync_transport.Http_client`. Bind `scope : Http_client.scope` and `epoch` to the currently admitted graph effect, not a stale graph selection. `scope.check_current` and `scope.admit_network` must use the executor's current effect checks (or the asset owner's active admission context); do not replace them with unconditional success. Refresh the callback context with each authorized epoch rather than retaining an old one in a long-lived cache owner. + +| Callback | HTTP method and path | Completion | +| --- | --- | --- | +| `Http_client.get_asset` | `GET /assets/{graph}/{uuid}.{file_type}` | `Ok None` on 404; otherwise bounded wire bytes, still to be decoded/verified by assets. | +| `Http_client.put_asset` | `PUT /assets/{graph}/{uuid}.{file_type}` with wire bytes | HTTP acknowledgement only; 404 is an error. Never treat it as verified publication. | + +Both authenticate with Bearer credentials; an authenticated 401 refreshes once and rechecks admission. PUT includes `x-amz-meta-checksum`, `x-amz-meta-type` and `content-type: application/octet-stream`. Asset decode, cryptographic verification, cache publication and durable graph activation remain owned by assets/Worker. In particular, do not pass `put_asset` directly as the asset DSL's `upload` callback: that callback requires verified publication, not an HTTP ACK. All backend callbacks must return only after physical network work joins. + +## Acceptance commands + +```sh +dune runtest logseq_sync/test/production_transport --force +dune exec test/source_boundary_test.exe -- . _build/default/logseq_sync.install _build/default/logseq_overlay_db.install +dune exec test/logseq_overlay_db_boundary_test.exe -- . _build/default/logseq_overlay_db.install +``` + +Require a successful exit and inspect/count the source-boundary executable's actual diagnostics, not just its exit status. Native transport and transport must have zero raw database/storage, Worker, legacy-reducer or Worker-fixture violations. Preserve unrelated concurrent owner changes and report their diagnostics separately. Run `ocamlformat --check` on all transport/native `.ml` and `.mli` files; run the full build and regressions on final inputs before integration. + +## Finishing-lane verification + +Verified against HEAD `e9a37592ef513c66acc0c228de056f7fecb04ef7` plus uncommitted work. Tracked diff SHA-256: `5b1aeede85358c72fa58eb5ee7548a8f88a4011d6011f11dc14d1cf90d9d0f09`. Transport/native/native-crypto/production-transport file-manifest SHA-256 (sorted per-file SHA-256 lines, excluding this document): `0dbf053e3b5de3761763d242e217594e12c58937fa4dd34f02c52a5b1d2ee1ef`. + +- `dune build @all app/native_embed.exe.o`: exit 0. +- `dune runtest --force`: exit 0 on the final combined worktree; earlier concurrent asset-owner failures are resolved in that tree, not suppressed by transport. +- Focused production transport suite: exit 0, 18 transport tests and 25 snapshot/HTTP tests, including encryption metadata rejection with staging cleanup. +- Named source-boundary executable: exit 0, `source boundary is clean`; actual diagnostic count drops from 18 at initial inspection to 0 in the final combined worktree. +- Named Overlay package-boundary executable: exit 0, `logseq_overlay_db package boundary: ok`. +- `ocamlformat --check`: exit 0 across all 38 transport/native/native-crypto/production-transport OCaml files; `xcrun clang-format --dry-run --Werror`: exit 0 across all three native C/header files. +- `git diff --check`: clean. No App/assets changes or commits made by this lane; legacy HTTP/snapshot entry points remain until the later App switch. + diff --git a/logseq_sync/lib/transport/core.ml b/logseq_sync/lib/transport/core.ml new file mode 100644 index 00000000..1de2c273 --- /dev/null +++ b/logseq_sync/lib/transport/core.ml @@ -0,0 +1,938 @@ +module Dsl = Sync_dsl.Core +module Protocol = Logseq_sync_protocol.Sync_protocol +module Native = Logseq_sync_native_transport.Websocket_eio +module Domain = Logseq_sync_native_transport.Transport_domain +module Cursor = Logseq_overlay_db.Types.Server_cursor + +let ( let* ) = Result.bind + +type scope = + { check_current : unit -> (unit, Dsl.error) result + ; admit_network : unit -> (unit, Dsl.error) result + ; handoff : Dsl.resource_reference Rrbvec.t -> (unit, Dsl.error) result + } + +type socket = + { activate : unit -> (unit, Dsl.error) result + ; send : string -> (unit, Dsl.error) result + ; stop : unit -> unit + ; await_stopped : unit -> unit + } + +type connect_error = + | Unauthorized + | Failed of Dsl.error + +type backend = + { connect : + token:string + -> on_message:(string -> unit) + -> on_close:(Dsl.connection_end -> unit) + -> (socket, connect_error) result + } + +let native_error = function + | Native.Unauthorized | Forbidden -> + Dsl.Authentication_failure "WebSocket authorization failed" + | Native.Authentication_failed _ -> + Dsl.Authentication_failure "WebSocket TLS authentication failed" + | Native.Protocol_failed _ -> Dsl.Protocol_failure "WebSocket protocol failed" + | Native.Connection_failed { message; _ } -> Dsl.Network_failure message +;; + +let native_end = function + | Domain.Local_close -> Dsl.Locally_closed + | Authentication_failure _ -> Dsl.Access_revoked "WebSocket authentication failed" + | Protocol_failure _ -> Dsl.Invalid_protocol "WebSocket protocol failed" + | Recoverable_close { message = None; _ } -> Dsl.Remote_closed + | Recoverable_close { message = Some message; _ } -> Dsl.Network_lost message +;; + +let native_backend + ~sw + ~clock + ~network + ~authenticator + ~identity + ~limits + ~deadlines + ~liveness + = + let uri = + Uri.of_string identity.Dsl.origin + |> fun origin -> + Uri.with_scheme origin (Some "wss") + |> fun origin -> Uri.with_path origin ("/sync/" ^ Uri.pct_encode identity.graph) + in + let clock = Logseq_sync_native_transport.Transport_time.monotonic clock in + let liveness = + match liveness with + | Execute.Core.Peer_close_only -> Native.Disabled + | Ping_pong { interval_seconds; timeout_seconds } -> + Native.Ping_pong { interval_seconds; timeout_seconds } + in + { connect = + (fun ~token ~on_message ~on_close -> + let* socket = + Native.connect + ~sw + ~clock + ~network + ~authenticator + ~uri + ~token + ~liveness + ~setup_seconds:deadlines.Execute.Core.setup_seconds + ~tls_handshake_seconds:deadlines.tls_handshake_seconds + ~http_response_seconds:deadlines.http_response_seconds + ~maximum_frame_bytes:(min limits.Dsl.maximum_message_bytes (8 * 1024 * 1024)) + ~on_message + ~on_close:(fun reason -> on_close (native_end reason)) + |> Result.map_error (function + | Native.Unauthorized -> Unauthorized + | error -> Failed (native_error error)) + in + Ok + { activate = + (fun () -> + Native.activate socket ~on_open:(fun () -> ()) + |> Result.map_error native_error) + ; send = + (fun wire -> + Native.send socket wire + |> Result.map_error (fun _ -> Dsl.Network_failure "WebSocket send failed")) + ; stop = (fun () -> Native.abort socket) + ; await_stopped = (fun () -> Native.await_retired socket) + }) + } +;; + +type family = + | Head + | Pull + | Push + +type invocation = + { cancellation : Eio.Cancel.t + ; stopped : unit Eio.Promise.t + } + +type entry = + { handle : Dsl.connection + ; epoch : int64 + ; changed : Eio.Condition.t + ; finished : unit Eio.Promise.t + ; finish : unit Eio.Promise.u + ; mutable invocations : invocation list + ; mutable joining : (unit, Dsl.error) result Eio.Promise.t option + ; mutable socket : socket option + ; mutable setup : Eio.Cancel.t option + ; mutable connecting : bool + ; mutable retiring : bool + ; mutable joined : bool + ; mutable terminal : Dsl.error option + ; mutable ending : Dsl.connection_end option + ; mutable request : family option + ; mutable response : Protocol.Server.message option + ; mutable receiving : bool + ; mutable notifications : Dsl.remote_message list + ; mutable notification_bytes : int + } + +type t = + { identity : Dsl.identity + ; limits : Dsl.limits + ; backend : backend + ; token : epoch:int64 -> refresh:bool -> (string, Dsl.error) result + ; fetch_snapshot : + scope:scope -> epoch:int64 -> Cursor.t -> (Dsl.snapshot, Dsl.error) result + ; snapshots : Snapshot.t option + ; timeout : 'a. float -> (unit -> 'a) -> 'a + ; connect_seconds : float + ; download_seconds : float + ; response_seconds : float + ; cleanup_seconds : float + ; mutable epoch : int64 option + ; mutable fenced_epoch : int64 option + ; mutable generation : int64 + ; mutable entries : entry list + ; mutable retired : Dsl.connection Weak.t list + } + +let same_identity t identity = + if t.identity = identity then Ok () else Error (Dsl.Identity_mismatch "Transport owner") +;; + +let pulse entry = Eio.Condition.broadcast entry.changed + +let terminal entry error = + if entry.terminal = None + then ( + entry.terminal <- Some error; + pulse entry) +;; + +let error_of_end = function + | Dsl.Remote_closed -> Dsl.Network_failure "WebSocket peer closed" + | Network_lost message -> Dsl.Network_failure message + | Access_revoked message -> Dsl.Authentication_failure message + | Invalid_protocol message -> Dsl.Protocol_failure message + | Locally_closed -> Dsl.Network_closed +;; + +let check t scope (entry : entry) = + let* () = scope.check_current () in + match t.epoch with + | None -> Error Dsl.Network_closed + | Some epoch when epoch <> entry.epoch -> Error Dsl.Stale_context + | Some _ when entry.retiring -> Error Dsl.Network_closed + | Some _ -> + (match entry.terminal with + | None -> Ok () + | Some error -> Error error) +;; + +let rec await entry ready = + match ready () with + | Some result -> result + | None -> + Eio.Condition.await_no_mutex entry.changed; + await entry ready +;; + +let message_bytes = function + | Dsl.Remote_changed _ -> 32 + | Remote_error message -> String.length message + | Connection_ended _ -> 32 +;; + +let notify t entry message = + let bytes = message_bytes message in + let rec add () = + if entry.retiring || entry.joined + then () + else ( + match List.rev entry.notifications, message with + | Dsl.Remote_changed previous :: rest, Dsl.Remote_changed next -> + let head = + if Cursor.compare previous.through next.through >= 0 then previous else next + in + entry.notifications <- List.rev (Dsl.Remote_changed head :: rest); + pulse entry + | _ + when List.length entry.notifications < t.limits.maximum_messages + && entry.notification_bytes <= t.limits.maximum_bytes - bytes -> + entry.notifications <- entry.notifications @ [ message ]; + entry.notification_bytes <- entry.notification_bytes + bytes; + pulse entry + | _ -> + Eio.Condition.await_no_mutex entry.changed; + add ()) + in + add () +;; + +let on_message t entry wire = + if (not entry.retiring) && entry.terminal = None + then + if String.length wire > t.limits.maximum_message_bytes + then + terminal entry (Dsl.Protocol_failure "WebSocket message exceeds configured bound") + else ( + match Protocol.decode_server_message wire with + | Error _ -> terminal entry (Dsl.Protocol_failure "Invalid WebSocket message") + | Ok (Protocol.Server.Hello _ as response) -> + if entry.request = Some Head && entry.response = None + then ( + entry.response <- Some response; + pulse entry) + else terminal entry (Dsl.Protocol_failure "Unmatched hello response") + | Ok (Changed { t = through }) -> + notify + t + entry + (Dsl.Remote_changed + { through; snapshot_available = true; expected_checksum = None }) + | Ok (Error _) -> + terminal entry (Dsl.Protocol_failure "Remote server error"); + notify t entry (Dsl.Remote_error "Remote server error") + | Ok (Pull_ok _ as response) -> + if entry.request = Some Pull && entry.response = None + then ( + entry.response <- Some response; + pulse entry) + else terminal entry (Dsl.Protocol_failure "Unmatched pull response") + | Ok ((Tx_batch_ok _ | Tx_reject _) as response) -> + if entry.request = Some Push && entry.response = None + then ( + entry.response <- Some response; + pulse entry) + else terminal entry (Dsl.Protocol_failure "Unmatched submission response") + | Ok (Online_users _ | Presence _ | Pong) -> ()) +;; + +let on_close entry ending = + if (not entry.retiring) && entry.terminal = None + then ( + entry.ending <- Some ending; + terminal entry (error_of_end ending); + pulse entry) +;; + +let remember_retired t entry = + let reference = Weak.create 1 in + Weak.set reference 0 (Some entry.handle); + t.retired <- reference :: List.filter (fun item -> Weak.check item 0) t.retired +;; + +let with_invocation entry callback = + let stopped, resolve = Eio.Promise.create () in + Eio.Cancel.sub (fun cancellation -> + let invocation = { cancellation; stopped } in + entry.invocations <- invocation :: entry.invocations; + Fun.protect + ~finally:(fun () -> + entry.invocations + <- List.filter (fun current -> current != invocation) entry.invocations; + Eio.Promise.resolve resolve ()) + callback) +;; + +let stop_entry t entry = + if entry.joined + then Ok () + else + Eio.Cancel.protect (fun () -> + match entry.joining with + | Some promise -> Eio.Promise.await promise + | None -> + let promise, resolve = Eio.Promise.create () in + entry.joining <- Some promise; + let result = + entry.retiring <- true; + entry.notifications <- []; + entry.notification_bytes <- 0; + pulse entry; + Option.iter (fun context -> Eio.Cancel.cancel context Exit) entry.setup; + let cleanup () = + let invocations = entry.invocations in + List.iter + (fun invocation -> Eio.Cancel.cancel invocation.cancellation Exit) + invocations; + Option.iter (fun socket -> socket.stop ()) entry.socket; + if entry.connecting then Eio.Promise.await entry.finished; + Option.iter (fun socket -> socket.await_stopped ()) entry.socket; + List.iter (fun invocation -> Eio.Promise.await invocation.stopped) invocations + in + try + t.timeout t.cleanup_seconds cleanup; + entry.joined <- true; + remember_retired t entry; + t.entries <- List.filter (fun current -> current != entry) t.entries; + pulse entry; + Ok () + with + | Eio.Time.Timeout -> + Error + (Dsl.Cleanup_failure + { primary = entry.terminal + ; failures = + Rrbvec.singleton + (Dsl.Network_failure "WebSocket retirement deadline expired") + }) + | _ -> + Error + (Dsl.Cleanup_failure + { primary = entry.terminal + ; failures = + Rrbvec.singleton (Dsl.Network_failure "WebSocket retirement failed") + }) + in + Eio.Promise.resolve resolve result; + result) +;; + +let find t handle = + match List.find_opt (fun entry -> entry.handle == handle) t.entries with + | Some entry -> Ok entry + | None -> Error Dsl.Stale_context +;; + +let validate_resource t = function + | Dsl.Connection_reference handle -> + let* _ = find t handle in + Ok () + | Dsl.Snapshot_reference _ as reference -> + (match t.snapshots with + | Some snapshots -> Snapshot.validate_resource snapshots reference + | None -> Error Dsl.Stale_context) + | _ -> Error (Dsl.Invalid_input "Not a transport resource") +;; + +let fence t ~identity = + let* () = same_identity t identity in + (match t.epoch with + | Some epoch -> t.fenced_epoch <- Some epoch + | None -> ()); + t.epoch <- None; + List.iter + (fun entry -> + pulse entry; + Option.iter (fun context -> Eio.Cancel.cancel context Exit) entry.setup) + t.entries; + Ok () +;; + +let close_network t ~identity = + let* () = fence t ~identity in + let failures = + List.filter_map + (fun entry -> + match stop_entry t entry with + | Ok () -> None + | Error error -> Some error) + t.entries + in + match failures with + | [] -> + (match t.snapshots with + | None -> Ok () + | Some snapshots -> Snapshot.drain snapshots) + | failures -> + Error (Dsl.Cleanup_failure { primary = None; failures = Rrbvec.of_list failures }) +;; + +let authorize t ~identity ~epoch = + let* () = same_identity t identity in + if epoch < 0L + then Error (Dsl.Invalid_input "Negative authorization epoch") + else ( + match t.epoch, t.fenced_epoch with + | Some current, _ when current = epoch -> Ok () + | Some _, _ -> Error Dsl.Stale_context + | None, Some previous when epoch <= previous -> Error Dsl.Stale_context + | None, _ -> + if t.entries <> [] + then Error Dsl.Stale_context + else ( + t.epoch <- Some epoch; + Ok ())) +;; + +let cleanup_result t entry result = + match stop_entry t entry with + | Ok () -> result + | Error (Dsl.Cleanup_failure failure) -> + Error + (Dsl.Cleanup_failure + { failure with + primary = + (match result with + | Error error -> Some error + | Ok _ -> failure.primary) + }) + | Error error -> Error error +;; + +let send scope entry message = + let* () = scope.admit_network () in + let* wire = + Protocol.encode_client_message message + |> Result.map_error (fun _ -> Dsl.Protocol_failure "Invalid client message") + in + match entry.socket with + | None -> Error Dsl.Stale_context + | Some socket -> socket.send wire +;; + +exception Unauthorized_challenge + +let connect t scope = + let* () = scope.check_current () in + let* epoch = + match t.epoch with + | None -> Error Dsl.Network_closed + | Some epoch -> Ok epoch + in + if List.length t.entries >= t.limits.maximum_resources + then Error (Dsl.Capacity_exceeded "Transport connections") + else if t.generation = Int64.max_int + then Error (Dsl.Capacity_exceeded "Connection generations") + else ( + t.generation <- Int64.succ t.generation; + let* handle = + Dsl.connection + ~owner:t.identity + ~token:("native-websocket-" ^ Int64.to_string t.generation) + ~generation:t.generation + in + let finished, finish = Eio.Promise.create () in + let entry = + { handle + ; epoch + ; changed = Eio.Condition.create () + ; finished + ; finish + ; socket = None + ; setup = None + ; connecting = true + ; invocations = [] + ; joining = None + ; retiring = false + ; joined = false + ; terminal = None + ; ending = None + ; request = None + ; response = None + ; receiving = false + ; notifications = [] + ; notification_bytes = 0 + } + in + t.entries <- entry :: t.entries; + let attempt refresh = + let* () = check t scope entry in + let* token = t.token ~epoch ~refresh in + let* () = check t scope entry in + let* () = scope.admit_network () in + let* socket = + match + t.backend.connect + ~token + ~on_message:(on_message t entry) + ~on_close:(on_close entry) + with + | Ok socket -> Ok socket + | Error Unauthorized -> raise Unauthorized_challenge + | Error (Failed error) -> Error error + in + entry.socket <- Some socket; + let* () = check t scope entry in + let* () = socket.activate () in + let* () = check t scope entry in + Ok handle + in + let result = + try + Fun.protect + ~finally:(fun () -> + entry.setup <- None; + entry.connecting <- false; + Eio.Promise.resolve entry.finish ()) + (fun () -> + t.timeout t.connect_seconds (fun () -> + Eio.Cancel.sub (fun context -> + entry.setup <- Some context; + try attempt false with + | Unauthorized_challenge -> + (try attempt true with + | Unauthorized_challenge -> + Error + (Dsl.Authentication_failure + "WebSocket token rejected after refresh"))))) + with + | Eio.Cancel.Cancelled _ as cancelled -> + ignore (stop_entry t entry : (unit, Dsl.error) result); + raise cancelled + | _ -> Error (Dsl.Network_failure "WebSocket setup failed") + in + match result with + | Error _ -> cleanup_result t entry result + | Ok handle -> + let transferred = + Eio.Cancel.protect (fun () -> + scope.handoff (Rrbvec.singleton (Dsl.Connection_reference handle))) + in + (match transferred with + | Error error -> cleanup_result t entry (Error error) + | Ok () -> Ok handle)) +;; + +let bounded t scope entry ?(seconds = t.response_seconds) ~write work = + try + let result = + with_invocation entry (fun () -> + t.timeout seconds (fun () -> + let* () = check t scope entry in + work ())) + in + match result with + | Ok _ -> result + | Error (Dsl.Capacity_exceeded _) -> result + | Error (Dsl.Network_failure _) when write -> + cleanup_result t entry (Error (Dsl.Outcome_unknown "Submission response lost")) + | Error _ -> cleanup_result t entry result + with + | Eio.Time.Timeout -> + cleanup_result + t + entry + (Error + (if write + then Dsl.Outcome_unknown "Submission response deadline expired" + else Dsl.Network_failure "WebSocket response deadline expired")) + | Eio.Cancel.Cancelled _ as cancelled -> + let retiring = entry.retiring in + let result = stop_entry t entry in + if retiring + then + let* () = result in + Error Dsl.Network_closed + else raise cancelled + | _ -> + cleanup_result + t + entry + (Error + (if write + then Dsl.Outcome_unknown "Submission interrupted" + else Dsl.Network_failure "WebSocket request failed")) +;; + +let response t scope entry family message decode = + if entry.request <> None + then Error (Dsl.Capacity_exceeded "WebSocket response family busy") + else + Fun.protect + ~finally:(fun () -> + entry.request <- None; + entry.response <- None; + pulse entry) + (fun () -> + entry.request <- Some family; + bounded t scope entry ~write:(family = Push) (fun () -> + let* () = send scope entry message in + let* response = + await entry (fun () -> + match check t scope entry with + | Error error -> Some (Error error) + | Ok () -> Option.map (fun message -> Ok message) entry.response) + in + decode response)) +;; + +let head t scope entry = + response t scope entry Head (Protocol.Client.Hello { client = "journal" }) Wire.head +;; + +let receive t scope entry = + if entry.receiving + then Error (Dsl.Capacity_exceeded "WebSocket receiver busy") + else + Fun.protect + ~finally:(fun () -> + entry.receiving <- false; + pulse entry) + (fun () -> + entry.receiving <- true; + try + with_invocation entry (fun () -> + await entry (fun () -> + match scope.check_current (), t.epoch with + | Error error, _ -> Some (Error error) + | Ok (), None -> Some (Error Dsl.Network_closed) + | Ok (), Some epoch when epoch <> entry.epoch -> + Some (Error Dsl.Stale_context) + | Ok (), Some _ when entry.retiring -> Some (Error Dsl.Network_closed) + | Ok (), Some _ -> + (match entry.notifications with + | message :: rest -> + entry.notifications <- rest; + entry.notification_bytes + <- entry.notification_bytes - message_bytes message; + pulse entry; + Some (Ok Dsl.{ connection = entry.handle; message }) + | [] -> + (match entry.ending, entry.terminal with + | Some ending, _ -> + entry.ending <- None; + Some + (Ok + Dsl. + { connection = entry.handle + ; message = Connection_ended ending + }) + | None, Some error -> Some (Error error) + | None, None -> None)))) + with + | Eio.Cancel.Cancelled _ as cancelled -> + let retiring = entry.retiring in + let result = stop_entry t entry in + if retiring + then + let* () = result in + Error Dsl.Network_closed + else raise cancelled) +;; + +let scoped : type a. t -> scope -> entry -> a Dsl.transport -> (a, Dsl.error) result = + fun t scope entry -> function + | Dsl.Head -> head t scope entry + | Dsl.Pull since -> + response + t + scope + entry + Pull + (Protocol.Client.Pull { since = Some since }) + (Wire.pull ~limits:t.limits ~since) + | Dsl.Push batch -> + if + Logseq_overlay_db.Types.submission_batch_total_bytes batch > t.limits.maximum_bytes + || List.length (Logseq_overlay_db.Types.submission_batch_wires batch) + > t.limits.maximum_items + then Error (Dsl.Capacity_exceeded "Submission transport bounds") + else response t scope entry Push (Wire.submission batch) (Wire.push ~batch) + | Dsl.Fetch_snapshot through -> + bounded t scope entry ~seconds:t.download_seconds ~write:false (fun () -> + let snapshot_scope = + { scope with + check_current = (fun () -> check t scope entry) + ; admit_network = + (fun () -> + let* () = check t scope entry in + scope.admit_network ()) + } + in + let* snapshot = t.fetch_snapshot ~scope:snapshot_scope ~epoch:entry.epoch through in + if + snapshot.Dsl.resource.owner <> t.identity + || not (Cursor.equal snapshot.through through) + then Error (Dsl.Protocol_failure "Snapshot source identity or baseline mismatch") + else Ok snapshot) + | Dsl.Connect | Receive _ | Retire_connection _ | On_connection _ -> + Error (Dsl.Invalid_input "Invalid scoped transport") +;; + +let run + : type a. + t -> scope:scope -> identity:Dsl.identity -> a Dsl.transport -> (a, Dsl.error) result + = + fun t ~scope ~identity operation -> + match same_identity t identity with + | Error error -> Error error + | Ok () -> + (match operation with + | Dsl.Retire_connection handle -> + (match find t handle with + | Ok entry -> stop_entry t entry + | Error error -> + if + List.exists + (fun weak -> + match Weak.get weak 0 with + | Some original -> original == handle + | None -> false) + t.retired + then Ok () + else Error error) + | Dsl.Connect -> connect t scope + | Dsl.Receive handle -> + let* entry = find t handle in + receive t scope entry + | Dsl.On_connection (handle, request) -> + let* entry = find t handle in + scoped t scope entry request + | Dsl.Head | Fetch_snapshot _ | Pull _ | Push _ -> + let live = List.filter (fun entry -> not entry.retiring) t.entries in + (match live with + | [ entry ] -> scoped t scope entry operation + | [] -> Error Dsl.Network_closed + | _ -> Error (Dsl.Invalid_input "Transport connection is ambiguous"))) +;; + +let dispatch t ~scope ~identity operation = + let scope = + { check_current = (fun () -> Execute.Core.check_current scope) + ; admit_network = (fun () -> Execute.Core.admit_network scope) + ; handoff = Execute.Core.handoff scope + } + in + run t ~scope ~identity operation +;; + +let capabilities t (caps : Execute.Core.capabilities) = + { caps with + dispatch = + { caps.dispatch with + transport = + (fun ~scope ~identity operation -> dispatch t ~scope ~identity operation) + } + ; authorize_network = (fun identity ~epoch -> authorize t ~identity ~epoch) + ; fence_network = (fun identity -> fence t ~identity) + ; close_network = (fun identity -> close_network t ~identity) + ; validate_resource = + (fun reference -> + match reference, t.snapshots with + | Dsl.Connection_reference _, _ | Dsl.Snapshot_reference _, Some _ -> + validate_resource t reference + | _ -> caps.validate_resource reference) + ; release_snapshot = + (fun identity snapshot -> + match t.snapshots with + | Some snapshots -> Snapshot.release snapshots ~identity snapshot + | None -> caps.release_snapshot identity snapshot) + } +;; + +let create_owner + ~sw + ~clock + ~identity + ~limits + ~(deadlines : Execute.Core.network_deadlines) + ~cleanup_seconds + ~backend + ~token + ~fetch_snapshot + ~snapshots + = + let response_seconds = deadlines.http_response_seconds in + let download_seconds = deadlines.download_seconds in + let connect_seconds = + deadlines.setup_seconds +. deadlines.tls_handshake_seconds +. response_seconds + in + if + not + (List.for_all + (fun value -> Float.is_finite value && value > 0.) + [ deadlines.setup_seconds + ; deadlines.tls_handshake_seconds + ; response_seconds + ; download_seconds + ; connect_seconds + ; cleanup_seconds + ]) + then Error (Dsl.Invalid_input "Transport deadlines") + else ( + let t = + { identity + ; limits + ; backend + ; token + ; fetch_snapshot + ; snapshots + ; timeout = + (fun seconds callback -> + Eio.Time.Timeout.run_exn (Eio.Time.Timeout.seconds clock seconds) callback) + ; connect_seconds + ; download_seconds + ; response_seconds + ; cleanup_seconds + ; epoch = None + ; fenced_epoch = None + ; generation = 0L + ; entries = [] + ; retired = [] + } + in + Eio.Switch.on_release sw (fun () -> + let result = + let* () = close_network t ~identity in + match snapshots with + | None -> Ok () + | Some owner -> Snapshot.close owner + in + match result with + | Ok () -> () + | Error error -> failwith (Dsl.error_message error)); + Ok t) +;; + +let create + ~sw + ~clock + ~identity + ~limits + ~deadlines + ~cleanup_seconds + ~backend + ~token + ~fetch_snapshot + = + create_owner + ~sw + ~clock + ~identity + ~limits + ~deadlines + ~cleanup_seconds + ~backend + ~token + ~snapshots:None + ~fetch_snapshot:(fun ~scope ~epoch:_ through -> fetch_snapshot ~scope through) +;; + +let create_native + ~verify_snapshot + ~sw + ~clock + ~network + ~authenticator + ~identity + ~limits + ~deadlines + ~cleanup_seconds + ~liveness + ~staging_directory + ~encrypted + ~token + = + let http = + Http_client.create + ~backend:(Http_client.native_backend ~sw ~clock ~network ~authenticator ~deadlines) + ~token + in + let* snapshots = + Snapshot.create + ~verify:verify_snapshot + ~root:staging_directory + ~identity + ~limits + ~encrypted + ~http + in + let backend = + native_backend + ~sw + ~clock + ~network + ~authenticator + ~identity + ~limits + ~deadlines + ~liveness + in + create_owner + ~sw + ~clock + ~identity + ~limits + ~deadlines + ~cleanup_seconds + ~backend + ~token + ~snapshots:(Some snapshots) + ~fetch_snapshot:(fun ~scope ~epoch through -> + Snapshot.acquire + snapshots + ~scope: + { Http_client.check_current = scope.check_current + ; admit_network = scope.admit_network + } + ~epoch + ~handoff:scope.handoff + through) +;; + +let snapshot_source t snapshot = + match t.snapshots with + | Some snapshots -> Snapshot.source snapshots snapshot + | None -> Error (Dsl.Invalid_input "No native snapshot producer") +;; + +let release_snapshot t ~identity snapshot = + match t.snapshots with + | Some snapshots -> Snapshot.release snapshots ~identity snapshot + | None -> Error (Dsl.Invalid_input "No native snapshot producer") +;; diff --git a/logseq_sync/lib/transport/core.mli b/logseq_sync/lib/transport/core.mli new file mode 100644 index 00000000..2d493082 --- /dev/null +++ b/logseq_sync/lib/transport/core.mli @@ -0,0 +1,104 @@ +module Dsl = Sync_dsl.Core + +type scope = + { check_current : unit -> (unit, Dsl.error) result + ; admit_network : unit -> (unit, Dsl.error) result + ; handoff : Dsl.resource_reference Rrbvec.t -> (unit, Dsl.error) result + } + +type socket = + { activate : unit -> (unit, Dsl.error) result + ; send : string -> (unit, Dsl.error) result + ; stop : unit -> unit + ; await_stopped : unit -> unit + } + +type connect_error = + | Unauthorized + | Failed of Dsl.error + +type backend = + { connect : + token:string + -> on_message:(string -> unit) + -> on_close:(Dsl.connection_end -> unit) + -> (socket, connect_error) result + } + +val native_backend + : sw:Eio.Switch.t + -> clock:_ Eio.Time.Mono.t + -> network:_ Eio.Net.t + -> authenticator:X509.Authenticator.t + -> identity:Dsl.identity + -> limits:Dsl.limits + -> deadlines:Execute.Core.network_deadlines + -> liveness:Execute.Core.websocket_liveness + -> backend + +type t + +val create + : sw:Eio.Switch.t + -> clock:_ Eio.Time.Mono.t + -> identity:Dsl.identity + -> limits:Dsl.limits + -> deadlines:Execute.Core.network_deadlines + -> cleanup_seconds:float + -> backend:backend + -> token:(epoch:int64 -> refresh:bool -> (string, Dsl.error) result) + -> fetch_snapshot: + (scope:scope + -> Dsl.Overlay.Types.server_cursor + -> (Dsl.snapshot, Dsl.error) result) + -> (t, Dsl.error) result + +val create_native + : verify_snapshot:Logseq_sync_native_transport.Snapshot_checksum.verifier + -> sw:Eio.Switch.t + -> clock:_ Eio.Time.Mono.t + -> network:_ Eio.Net.t + -> authenticator:X509.Authenticator.t + -> identity:Dsl.identity + -> limits:Dsl.limits + -> deadlines:Execute.Core.network_deadlines + -> cleanup_seconds:float + -> liveness:Execute.Core.websocket_liveness + -> staging_directory:string + -> encrypted:bool + -> token:(epoch:int64 -> refresh:bool -> (string, Dsl.error) result) + -> (t, Dsl.error) result + +val snapshot_source + : t + -> Dsl.snapshot + -> (Execute.Core.snapshot_source, Dsl.error) result + +val release_snapshot + : t + -> identity:Dsl.identity + -> Dsl.snapshot + -> (unit, Dsl.error) result + +val authorize : t -> identity:Dsl.identity -> epoch:int64 -> (unit, Dsl.error) result +val fence : t -> identity:Dsl.identity -> (unit, Dsl.error) result +val close_network : t -> identity:Dsl.identity -> (unit, Dsl.error) result +val validate_resource : t -> Dsl.resource_reference -> (unit, Dsl.error) result + +(** The snapshot producer retains its original cleanup ledger until handoff. *) +val run + : t + -> scope:scope + -> identity:Dsl.identity + -> 'a Dsl.transport + -> ('a, Dsl.error) result + +val dispatch + : t + -> scope:Execute.Core.effect_scope + -> identity:Dsl.identity + -> 'a Dsl.transport + -> ('a, Dsl.error) result + +(** Install only transport dispatch, network authority and connection validation. *) +val capabilities : t -> Execute.Core.capabilities -> Execute.Core.capabilities diff --git a/logseq_sync/lib/transport/dune b/logseq_sync/lib/transport/dune new file mode 100644 index 00000000..a5a98537 --- /dev/null +++ b/logseq_sync/lib/transport/dune @@ -0,0 +1,5 @@ +(library + (name logseq_sync_transport) + (public_name logseq_sync.transport) + (modules core wire http_client snapshot) + (libraries unix yojson eio execute sync_dsl logseq_sync.native_transport logseq_sync.protocol logseq_overlay_db logseq_db_types rrbvec uri x509)) diff --git a/logseq_sync/lib/transport/http_client.ml b/logseq_sync/lib/transport/http_client.ml new file mode 100644 index 00000000..17565f74 --- /dev/null +++ b/logseq_sync/lib/transport/http_client.ml @@ -0,0 +1,189 @@ +module Dsl = Sync_dsl.Core +module Http = Logseq_sync_native_transport.Http +module Native = Logseq_sync_native_transport.Http_eio + +let ( let* ) = Result.bind + +type scope = + { check_current : unit -> (unit, Dsl.error) result + ; admit_network : unit -> (unit, Dsl.error) result + } + +type backend = + { perform : Http.request -> (Native.response, Native.error) result + ; download : + request:Http.request + -> destination:string + -> maximum_bytes:int + -> (Native.response, Native.error) result + } + +let native_backend ~sw ~clock ~network ~authenticator ~deadlines = + let clock = Logseq_sync_native_transport.Transport_time.monotonic clock in + { perform = + (fun request -> + Native.perform_with_deadlines + ~setup_seconds:deadlines.Execute.Core.setup_seconds + ~tls_handshake_seconds:deadlines.tls_handshake_seconds + ~http_response_seconds:deadlines.http_response_seconds + ~sw + ~clock + ~network + ~authenticator + request) + ; download = + (fun ~request ~destination ~maximum_bytes -> + Native.download_with_deadlines + ~setup_seconds:deadlines.setup_seconds + ~tls_handshake_seconds:deadlines.tls_handshake_seconds + ~download_seconds:deadlines.download_seconds + ~http_response_seconds:deadlines.http_response_seconds + ~sw + ~clock + ~network + ~authenticator + ~request + ~destination + ~maximum_bytes + ~on_progress:(fun _ -> ())) + } +;; + +let domain_error = function + | Native.Network message -> Dsl.Network_failure message + | Native.Protocol message -> Dsl.Protocol_failure message + | Native.Authentication message -> Dsl.Authentication_failure message + | Native.Storage message -> Dsl.Storage_failure message + | Native.Crypto message -> Dsl.Crypto_failure message + | Native.Invalid_input message -> Dsl.Invalid_input message +;; + +type t = + { backend : backend + ; token : epoch:int64 -> refresh:bool -> (string, Dsl.error) result + } + +let create ~backend ~token = { backend; token } + +let request t ~scope ~epoch ~authenticated make_request perform = + let attempt refresh = + let* () = scope.check_current () in + let* token = + if authenticated then Result.map Option.some (t.token ~epoch ~refresh) else Ok None + in + let* () = scope.check_current () in + let request = make_request token in + let* () = scope.admit_network () in + let* response = perform request |> Result.map_error domain_error in + let* () = scope.check_current () in + Ok response + in + try + let* response = attempt false in + let* response = + if authenticated && response.Native.status = 401 then attempt true else Ok response + in + match response.Native.status with + | 401 | 403 -> Error (Dsl.Authentication_failure "HTTP authorization failed") + | status when status >= 200 && status < 300 -> Ok response + | 404 -> Ok response + | 429 -> Error (Dsl.Protocol_failure "HTTP rate limit exceeded (429)") + | status when status >= 500 && status < 600 -> + Error (Dsl.Protocol_failure (Printf.sprintf "HTTP server failure (%d)" status)) + | status -> + Error (Dsl.Protocol_failure (Printf.sprintf "HTTP request rejected (%d)" status)) + with + | Eio.Cancel.Cancelled _ as cancelled -> raise cancelled + | Eio.Time.Timeout -> Error (Dsl.Network_failure "HTTP request deadline expired") + | Invalid_argument _ -> Error (Dsl.Invalid_input "Invalid HTTP request") + | _ -> Error (Dsl.Protocol_failure "Unexpected HTTP client failure") +;; + +let perform t ~scope ~epoch ~authenticated make_request = + request t ~scope ~epoch ~authenticated make_request t.backend.perform +;; + +let download t ~scope ~epoch ~authenticated ~destination ~maximum_bytes make_request = + request t ~scope ~epoch ~authenticated make_request (fun request -> + t.backend.download ~request ~destination ~maximum_bytes) +;; + +let asset_request + ~identity + ~asset + ~(version : Logseq_db_types.Asset_descriptor.version) + ~operation + ~maximum_response_bytes + token + = + let base_url = Uri.of_string identity.Dsl.origin in + (match Http.validate_base_url base_url with + | Ok () -> () + | Error message -> invalid_arg message); + let uri = + Uri.with_path + base_url + (Printf.sprintf + "/assets/%s/%s.%s" + (Uri.pct_encode identity.graph) + (Logseq_db_types.Graph_types.Uuid.to_string asset) + (Uri.pct_encode version.file_type)) + in + let headers = Http.authorization (Option.get token) in + let headers = + match operation with + | Http.Get -> headers + | Put _ -> + [ "x-amz-meta-checksum", version.checksum + ; "x-amz-meta-type", version.file_type + ; "content-type", "application/octet-stream" + ] + @ headers + in + Http. + { operation + ; uri + ; headers + ; maximum_response_bytes + ; expected_content_type = Asset_binary + } +;; + +let get_asset t ~scope ~epoch ~identity ~asset ~version ~maximum_wire_bytes = + if maximum_wire_bytes <= 0 + then Error (Dsl.Invalid_input "Asset HTTP bound") + else + let* response = + perform + t + ~scope + ~epoch + ~authenticated:true + (asset_request + ~identity + ~asset + ~version + ~operation:Http.Get + ~maximum_response_bytes:maximum_wire_bytes) + in + if response.status = 404 then Ok None else Ok (Some response.body) +;; + +let put_asset t ~scope ~epoch ~identity ~asset ~version ~wire = + let* response = + perform + t + ~scope + ~epoch + ~authenticated:true + (asset_request + ~identity + ~asset + ~version + ~operation:(Http.Put wire) + ~maximum_response_bytes:65536) + in + if response.status = 404 + then Error (Dsl.Protocol_failure "Asset upload endpoint unavailable") + else Ok () +;; diff --git a/logseq_sync/lib/transport/http_client.mli b/logseq_sync/lib/transport/http_client.mli new file mode 100644 index 00000000..fe0ab0cc --- /dev/null +++ b/logseq_sync/lib/transport/http_client.mli @@ -0,0 +1,72 @@ +module Dsl = Sync_dsl.Core +module Http = Logseq_sync_native_transport.Http +module Native = Logseq_sync_native_transport.Http_eio + +type scope = + { check_current : unit -> (unit, Dsl.error) result + ; admit_network : unit -> (unit, Dsl.error) result + } + +type backend = + { perform : Http.request -> (Native.response, Native.error) result + ; download : + request:Http.request + -> destination:string + -> maximum_bytes:int + -> (Native.response, Native.error) result + } + +val native_backend + : sw:Eio.Switch.t + -> clock:_ Eio.Time.Mono.t + -> network:_ Eio.Net.t + -> authenticator:X509.Authenticator.t + -> deadlines:Execute.Core.network_deadlines + -> backend + +type t + +val create + : backend:backend + -> token:(epoch:int64 -> refresh:bool -> (string, Dsl.error) result) + -> t + +(** Every attempt rechecks admission; only authenticated 401 refreshes, once. Backends return after physical join. *) +val perform + : t + -> scope:scope + -> epoch:int64 + -> authenticated:bool + -> (string option -> Http.request) + -> (Native.response, Dsl.error) result + +val download + : t + -> scope:scope + -> epoch:int64 + -> authenticated:bool + -> destination:string + -> maximum_bytes:int + -> (string option -> Http.request) + -> (Native.response, Dsl.error) result + +val get_asset + : t + -> scope:scope + -> epoch:int64 + -> identity:Dsl.identity + -> asset:Logseq_db_types.Graph_types.Uuid.t + -> version:Logseq_db_types.Asset_descriptor.version + -> maximum_wire_bytes:int + -> (string option, Dsl.error) result + +(** PUT completion acknowledges HTTP only; the asset owner retains verification and publication. *) +val put_asset + : t + -> scope:scope + -> epoch:int64 + -> identity:Dsl.identity + -> asset:Logseq_db_types.Graph_types.Uuid.t + -> version:Logseq_db_types.Asset_descriptor.version + -> wire:string + -> (unit, Dsl.error) result diff --git a/logseq_sync/lib/transport/snapshot.ml b/logseq_sync/lib/transport/snapshot.ml new file mode 100644 index 00000000..0dd789c3 --- /dev/null +++ b/logseq_sync/lib/transport/snapshot.ml @@ -0,0 +1,352 @@ +module Dsl = Sync_dsl.Core +module Native = Logseq_sync_native_transport +module Http = Native.Http +module Cursor = Logseq_overlay_db.Types.Server_cursor + +let ( let* ) = Result.bind + +type artifact = + { snapshot : Dsl.snapshot + ; directory : string + ; source : Execute.Core.snapshot_source + ; reservation : int64 + ; mutable handed : bool + } + +type t = + { root : string + ; identity : Dsl.identity + ; limits : Dsl.limits + ; encrypted : bool + ; verify : Native.Snapshot_checksum.verifier + ; http : Http_client.t + ; mutable artifacts : artifact list + ; mutable pending_cleanup : (string * int64) list + ; mutable released : Dsl.snapshot Weak.t list + ; mutable reserved : int64 + ; mutable acquiring : bool + ; mutable closed : bool + } + +let create ~verify ~root ~identity ~limits ~encrypted ~http = + try + if Filename.is_relative root || (Unix.lstat root).st_kind <> Unix.S_DIR + then Error (Dsl.Invalid_input "Snapshot staging directory") + else + Ok + { root + ; identity + ; limits + ; encrypted + ; verify + ; http + ; artifacts = [] + ; pending_cleanup = [] + ; released = [] + ; reserved = 0L + ; acquiring = false + ; closed = false + } + with + | Unix.Unix_error _ -> + Error (Dsl.Storage_failure "Snapshot staging directory unavailable") +;; + +let find t snapshot = + match List.find_opt (fun artifact -> artifact.snapshot == snapshot) t.artifacts with + | Some artifact -> Ok artifact + | None -> Error Dsl.Stale_context +;; + +let source t snapshot = + let* artifact = find t snapshot in + Ok artifact.source +;; + +let validate_resource t = function + | Dsl.Snapshot_reference snapshot -> Result.map (fun _ -> ()) (find t snapshot) + | _ -> Error (Dsl.Invalid_input "Not a snapshot resource") +;; + +let cleanup directory = + try + Native.Snapshot_verify.remove_tree directory; + Ok () + with + | _ -> Error (Dsl.Storage_failure "Snapshot staging cleanup failed") +;; + +let release t ~identity snapshot = + if identity <> t.identity + then Error Dsl.Stale_context + else ( + match find t snapshot with + | Error error -> + if + List.exists + (fun weak -> + match Weak.get weak 0 with + | Some original -> original == snapshot + | None -> false) + t.released + then Ok () + else Error error + | Ok artifact -> + let* () = cleanup artifact.directory in + t.reserved <- Int64.sub t.reserved artifact.reservation; + t.artifacts <- List.filter (fun item -> item != artifact) t.artifacts; + let weak = Weak.create 1 in + Weak.set weak 0 (Some snapshot); + t.released <- weak :: List.filter (fun item -> Weak.check item 0) t.released; + Ok ()) +;; + +let drain t = + if t.acquiring + then Error (Dsl.Capacity_exceeded "Snapshot acquisition still active") + else ( + let errors = + List.filter_map + (fun artifact -> + if artifact.handed + then None + else ( + match release t ~identity:t.identity artifact.snapshot with + | Ok () -> None + | Error error -> Some error)) + t.artifacts + in + let pending = t.pending_cleanup in + let failures = + List.filter_map + (fun (directory, reservation) -> + match cleanup directory with + | Error error -> Some error + | Ok () -> + t.pending_cleanup + <- List.filter (fun (path, _) -> path <> directory) t.pending_cleanup; + t.reserved <- Int64.sub t.reserved reservation; + None) + pending + in + match errors @ failures with + | [] -> Ok () + | failures -> + Error (Dsl.Cleanup_failure { primary = None; failures = Rrbvec.of_list failures })) +;; + +let close t = + t.closed <- true; + let* () = drain t in + let errors = + List.filter_map + (fun artifact -> + match release t ~identity:t.identity artifact.snapshot with + | Ok () -> None + | Error error -> Some error) + t.artifacts + in + match errors with + | [] -> Ok () + | failures -> + Error (Dsl.Cleanup_failure { primary = None; failures = Rrbvec.of_list failures }) +;; + +let metadata body = + let invalid () = Error (Dsl.Protocol_failure "Invalid snapshot metadata") in + try + match Yojson.Safe.from_string body with + | `Assoc fields -> + (match + ( List.assoc_opt "ok" fields + , List.assoc_opt "url" fields + , List.assoc_opt "t" fields ) + with + | Some (`Bool true), Some (`String url), Some (`Int value) + when value >= 0 && Int64.of_int value <= 9007199254740991L -> + let uri = Uri.of_string url in + if not (Http.valid_artifact uri) + then invalid () + else + Result.map (fun through -> uri, through) (Cursor.of_int64 (Int64.of_int value)) + |> Result.map_error (fun _ -> Dsl.Protocol_failure "Invalid snapshot cursor") + | _ -> invalid ()) + | _ -> invalid () + with + | Yojson.Json_error _ | Invalid_argument _ -> invalid () +;; + +let acquire t ~scope ~epoch ~handoff through = + let* () = scope.Http_client.check_current () in + if t.closed + then Error Dsl.Network_closed + else if + t.acquiring + || t.pending_cleanup <> [] + || List.length t.artifacts >= t.limits.maximum_resources + then Error (Dsl.Capacity_exceeded "Snapshot producers") + else ( + let maximum64 = + min + t.limits.snapshot_maximum_artifact_bytes + (Int64.div t.limits.snapshot_staging_budget_bytes 3L) + in + let reservation = Int64.mul maximum64 3L in + if + maximum64 <= 0L + || maximum64 > Int64.of_int max_int + || t.reserved > Int64.sub t.limits.snapshot_staging_budget_bytes reservation + then Error (Dsl.Capacity_exceeded "Snapshot staging budget") + else ( + let maximum_bytes = Int64.to_int maximum64 in + t.acquiring <- true; + t.reserved <- Int64.add t.reserved reservation; + let directory = ref None + and retained = ref false + and result = ref None in + let work () = + let base_url = Uri.of_string t.identity.origin in + let* graph_id = + Logseq_db_types.Graph_types.Uuid.of_string t.identity.graph + |> Result.map_error (fun _ -> Dsl.Invalid_input "Snapshot graph identity") + in + let* response = + Http_client.perform t.http ~scope ~epoch ~authenticated:true (fun token -> + Http.snapshot_metadata ~base_url ~graph_id ~token:(Option.get token)) + in + let* uri, baseline = metadata response.body in + let* () = + if Cursor.equal baseline through + then Ok () + else Error (Dsl.Protocol_failure "Snapshot baseline changed") + in + let* () = scope.check_current () in + let path = Filename.temp_file ~temp_dir:t.root "snapshot-" "" in + Sys.remove path; + Unix.mkdir path 0o700; + directory := Some path; + let raw = Filename.concat path "download" + and artifact = Filename.concat path "artifact" in + let temporary_paths = + [ Filename.concat path "gzip-1"; Filename.concat path "gzip-2" ] + in + let* response = + Http_client.download + t.http + ~scope + ~epoch + ~authenticated:(Http.same_origin base_url uri) + ~destination:raw + ~maximum_bytes + (fun token -> Http.artifact ~base_url ~uri ~token) + in + let* () = + if response.status >= 200 && response.status < 300 + then Ok () + else Error (Dsl.Protocol_failure "Snapshot artifact unavailable") + in + let* expected_rows = + Native.Snapshot_artifact.artifact_row_count response.headers + |> Result.map_error (fun _ -> Dsl.Protocol_failure "Invalid snapshot row count") + in + let* () = + if expected_rows <= maximum_bytes + then Ok () + else Error (Dsl.Capacity_exceeded "Snapshot rows") + in + let* () = scope.check_current () in + let* () = + Native.Snapshot_artifact.peel_gzip_layers + ~decompress_gzip:Native.Artifact_decoder.decompress_gzip + ~maximum_bytes + ~source:raw + ~destination:artifact + ~temporary_paths + |> Result.map_error (fun _ -> + Dsl.Protocol_failure "Invalid snapshot compression") + in + List.iter + (fun file -> if Sys.file_exists file then Sys.remove file) + (raw :: temporary_paths); + let* checksum = + Native.Snapshot_verify.inspect + ~verify:t.verify + ~check_current:(fun () -> + Eio.Fiber.yield (); + Eio.Fiber.check ()) + ~scratch_bytes:t.limits.snapshot_scratch_budget_bytes + ~maximum_frame_bytes:(min maximum_bytes (64 * 1024 * 1024)) + ~directory:path + ~encrypted:t.encrypted + ~path:artifact + ~expected_rows + |> Result.map_error (fun _ -> + Dsl.Protocol_failure "Invalid snapshot framing, storage or encryption") + in + let* () = scope.check_current () in + let* resource = Dsl.resource ~owner:t.identity ~token:(Filename.basename path) in + let* snapshot = + Dsl.snapshot + ~resource + ~through:baseline + ~checksum:(Logseq_overlay_db.Types.Checksum.to_string checksum) + ~encrypted:t.encrypted + in + let size = Int64.of_int (Unix.stat artifact).st_size in + let entry = + { snapshot + ; directory = path + ; source = { path = artifact; expected_rows } + ; reservation = size + ; handed = false + } + in + t.reserved <- Int64.sub t.reserved (Int64.sub reservation size); + t.artifacts <- entry :: t.artifacts; + retained := true; + let transferred = + Eio.Cancel.protect (fun () -> + handoff (Rrbvec.singleton (Dsl.Snapshot_reference snapshot))) + in + match transferred with + | Ok () -> + entry.handed <- true; + Ok snapshot + | Error error -> + (match release t ~identity:t.identity snapshot with + | Ok () -> Error error + | Error cleanup -> + Error + (Dsl.Cleanup_failure + { primary = Some error; failures = Rrbvec.singleton cleanup })) + in + let finish () = + t.acquiring <- false; + if not !retained + then ( + match !directory with + | None -> t.reserved <- Int64.sub t.reserved reservation + | Some path -> + (match cleanup path with + | Ok () -> t.reserved <- Int64.sub t.reserved reservation + | Error error -> + t.pending_cleanup <- (path, reservation) :: t.pending_cleanup; + result + := Some + (Error + (Dsl.Cleanup_failure + { primary = + (match !result with + | Some (Error primary) -> Some primary + | _ -> None) + ; failures = Rrbvec.singleton error + })))) + in + Fun.protect ~finally:finish (fun () -> + result + := Some + (try work () with + | Eio.Cancel.Cancelled _ as cancelled -> raise cancelled + | _ -> Error (Dsl.Storage_failure "Snapshot acquisition failed"))); + Option.get !result)) +;; diff --git a/logseq_sync/lib/transport/snapshot.mli b/logseq_sync/lib/transport/snapshot.mli new file mode 100644 index 00000000..b66e1e27 --- /dev/null +++ b/logseq_sync/lib/transport/snapshot.mli @@ -0,0 +1,29 @@ +module Dsl = Sync_dsl.Core + +type t + +val create + : verify:Logseq_sync_native_transport.Snapshot_checksum.verifier + -> root:string + -> identity:Dsl.identity + -> limits:Dsl.limits + -> encrypted:bool + -> http:Http_client.t + -> (t, Dsl.error) result + +(** No preliminary pull. Metadata cursor must match the requested baseline; live rows do not advance it. *) +val acquire + : t + -> scope:Http_client.scope + -> epoch:int64 + -> handoff:(Dsl.resource_reference Rrbvec.t -> (unit, Dsl.error) result) + -> Dsl.Overlay.Types.server_cursor + -> (Dsl.snapshot, Dsl.error) result + +(** Exact original descriptors only; files remain owned across network fencing until checked release. *) +val source : t -> Dsl.snapshot -> (Execute.Core.snapshot_source, Dsl.error) result + +val validate_resource : t -> Dsl.resource_reference -> (unit, Dsl.error) result +val release : t -> identity:Dsl.identity -> Dsl.snapshot -> (unit, Dsl.error) result +val drain : t -> (unit, Dsl.error) result +val close : t -> (unit, Dsl.error) result diff --git a/logseq_sync/lib/transport/wire.ml b/logseq_sync/lib/transport/wire.ml new file mode 100644 index 00000000..de80f57d --- /dev/null +++ b/logseq_sync/lib/transport/wire.ml @@ -0,0 +1,187 @@ +module Dsl = Sync_dsl.Core +module Protocol = Logseq_sync_protocol.Sync_protocol +module O = Logseq_overlay_db.Types +module Cursor = O.Server_cursor + +let ( let* ) = Result.bind +let invalid message = Error (Dsl.Protocol_failure message) + +let checksum value = + O.Checksum.of_string ("checksum:v1:" ^ value) + |> Result.map_error (fun _ -> Dsl.Protocol_failure "Invalid checksum") +;; + +let optional_checksum = function + | None -> Ok None + | Some value -> Result.map Option.some (checksum value) +;; + +let operation_name = function + | O.Save_block_operation -> "save-block" + | Append_journal_operation -> "append-journal" + | Insert_blocks_operation -> "insert-blocks" + | Delete_blocks_operation -> "delete-blocks" + | Create_journal_page_operation -> "create-journal-page" + | Set_task_status_operation -> "set-task-status" + | Clear_task_status_operation -> "clear-task-status" + | Publish_asset_operation -> "publish-asset" + | Set_asset_reference_operation -> "set-asset-reference" +;; + +let submission batch = + Protocol.Client.Tx_batch + { client_revision = + Some (O.Submission_batch_id.to_string (O.submission_batch_id batch)) + ; t_before = O.submission_batch_t_before batch + ; txs = + List.map + (fun wire -> + Protocol.Client. + { tx = O.submission_wire_protected_transaction wire + ; tx_id = Some (O.submission_wire_mutation_id wire) + ; outliner_op = Some (operation_name (O.submission_wire_operation wire)) + }) + (O.submission_batch_wires batch) + } +;; + +let head = function + | Protocol.Server.Hello { t; checksum } -> + let* expected_checksum = optional_checksum checksum in + Ok Dsl.{ through = t; snapshot_available = true; expected_checksum } + | _ -> invalid "Expected hello response" +;; + +let pull ~limits ~since = function + | Protocol.Server.Pull_ok { t; checksum = value; txs } -> + if Cursor.compare t since < 0 || List.length txs > limits.Dsl.maximum_items + then invalid "Invalid pull bounds" + else ( + let rec validate previous bytes accumulated = function + | [] -> Ok (List.rev accumulated) + | (entry : Protocol.Server.pull_transaction) :: rest -> + let length = String.length entry.tx in + if + (match previous with + | Some cursor -> Cursor.compare entry.t cursor <= 0 + | None -> false) + || Cursor.compare entry.t t > 0 + || length > limits.maximum_message_bytes + || bytes > limits.maximum_bytes - length + then invalid "Invalid authoritative transaction sequence" + else + let* encoded = + O.encoded_transaction_of_string + ~maximum_bytes:limits.maximum_message_bytes + entry.tx + |> Result.map_error (fun _ -> + Dsl.Protocol_failure "Invalid authoritative transaction") + in + let accumulated = + if Cursor.compare entry.t since > 0 + then + O.authoritative_transaction ~cursor:entry.t ~transaction:encoded + :: accumulated + else accumulated + in + validate (Some entry.t) (bytes + length) accumulated rest + in + let* transactions = validate None 0 [] txs in + let* checksum = optional_checksum value in + match transactions with + | [] -> Ok (Dsl.No_progress { since; through = t; expected_checksum = checksum }) + | _ -> + let* batch = + O.authoritative_batch + ~maximum_count:limits.maximum_items + ~maximum_bytes:limits.maximum_bytes + ~transactions + ~through:t + ~checksum + |> Result.map_error (fun _ -> + Dsl.Protocol_failure "Invalid authoritative batch") + in + Ok (Dsl.Pulled batch)) + | _ -> invalid "Expected pull response" +;; + +let barrier ~baseline through value = + if Cursor.compare through baseline <= 0 + then invalid "Acceptance cursor did not advance" + else ( + match value with + | None -> invalid "Acceptance omitted checksum" + | Some value -> + let* checksum = checksum value in + Ok O.{ through; checksum }) +;; + +let rejection batch (rejection : Protocol.rejection) = + let baseline = O.submission_batch_t_before batch in + let members = List.map O.submission_wire_mutation_id (O.submission_batch_wires batch) in + let equal = O.Graph.Uuid.equal in + let rec prefix members = function + | [] -> Ok members + | item :: rest -> + (match members with + | member :: members when equal member item -> prefix members rest + | _ -> invalid "Accepted IDs are not an ordered submission prefix") + in + let* remaining = prefix members rejection.success_tx_ids in + let* failed_member, unexecuted_suffix = + match rejection.failed_tx_id, remaining with + | Some failed, expected :: rest when equal failed expected -> Ok (Some failed, rest) + | None, expected :: rest when rejection.success_tx_ids = [] -> Ok (Some expected, rest) + | _ -> invalid "Rejected member does not match frozen submission" + in + match rejection.reason with + | Protocol.Stale -> + (match rejection.t with + | Some through + when Cursor.compare through baseline > 0 && rejection.success_tx_ids = [] -> + Ok (O.Stale { through }) + | _ -> invalid "Invalid stale rejection cursor or partition") + | Empty_tx_data + | Invalid_tx + | Invalid_t_before + | Db_transact_failed + | Snapshot_upload_in_progress -> + let* acceptance_barrier = + match rejection.success_tx_ids with + | [] -> Ok None + | _ -> + (match rejection.t with + | None -> invalid "Partial rejection omitted acceptance cursor" + | Some through -> + Result.map Option.some (barrier ~baseline through rejection.checksum)) + in + let reason = + match rejection.reason with + | Protocol.Empty_tx_data | Invalid_tx | Invalid_t_before -> O.Invalid_request + | Db_transact_failed when rejection.missing_block_uuids <> [] -> + O.Missing_dependencies + | Db_transact_failed | Snapshot_upload_in_progress | Stale -> O.Operational_failure + in + Ok + (O.Definitive + { reason + ; partition = + { accepted_prefix = rejection.success_tx_ids + ; failed_member + ; unexecuted_suffix + ; acceptance_barrier + ; missing_uuids = rejection.missing_block_uuids + ; diagnostics = [] + } + }) +;; + +let push ~batch = function + | Protocol.Server.Tx_batch_ok { t; checksum } -> + Result.map + (fun value -> Dsl.Accepted value) + (barrier ~baseline:(O.submission_batch_t_before batch) t checksum) + | Protocol.Server.Tx_reject value -> + Result.map (fun value -> Dsl.Rejected value) (rejection batch value) + | _ -> invalid "Expected submission response" +;; diff --git a/logseq_sync/lib/transport/wire.mli b/logseq_sync/lib/transport/wire.mli new file mode 100644 index 00000000..a3607efe --- /dev/null +++ b/logseq_sync/lib/transport/wire.mli @@ -0,0 +1,16 @@ +module Dsl = Sync_dsl.Core +module Protocol = Logseq_sync_protocol.Sync_protocol + +val submission : Dsl.Overlay.Types.submission_batch -> Protocol.Client.message +val head : Protocol.Server.message -> (Dsl.remote_head, Dsl.error) result + +val pull + : limits:Dsl.limits + -> since:Dsl.Overlay.Types.server_cursor + -> Protocol.Server.message + -> (Dsl.pull_result, Dsl.error) result + +val push + : batch:Dsl.Overlay.Types.submission_batch + -> Protocol.Server.message + -> (Dsl.submission_reply, Dsl.error) result diff --git a/logseq_sync/lib/worker_owner/core.ml b/logseq_sync/lib/worker_owner/core.ml new file mode 100644 index 00000000..8752124d --- /dev/null +++ b/logseq_sync/lib/worker_owner/core.ml @@ -0,0 +1,774 @@ +module Dsl = Sync_dsl.Core +module Execute = Execute.Core +module Intent = Logseq_db_types.Asset_upload_intent +module Input = Logseq_db_types.Asset_import +module Store = Logseq_db_storage.Asset_upload_store +module Assets = Logseq_sync_assets.Core +module Cache = Logseq_sync_assets.Cache +module Uuid = Logseq_db_types.Graph_types.Uuid +module Database = Logseq_overlay_db.Database + +let ( let* ) = Result.bind + +type store = + { connection : Sqlite3.db + ; mutex : Eio.Mutex.t + } + +let store connection = { connection; mutex = Eio.Mutex.create () } + +type binding = + { owner : Execute.database_owner + ; network : Execute.worker_network + ; cleanup_seconds : float + } + +type t = + { namespace : string + ; assets : Assets.t + ; store : store + ; staging_mutex : Eio.Mutex.t + ; notify : + owner:Execute.database_owner -> identity:Dsl.identity -> (unit, Dsl.error) result + ; mutable binding : binding option + ; scopes : (Uuid.t, scope) Hashtbl.t + ; mutable leases : int + ; changed : Eio.Condition.t + ; sync_changed : Eio.Condition.t + ; mutable sync_revision : int64 + ; mutable fenced : bool + ; join_with_timeout : + float -> (unit -> (unit, Dsl.error) result) -> (unit, Dsl.error) result + } + +and scope = + { owner : t + ; operation : Uuid.t + ; mutable active : bool + ; mutable cancelled : bool + ; cancel_context : Eio.Cancel.t + } + +type imported_file = + { owner : t + ; handle : Cache.handle + ; path : string + ; mutable released : bool + } + +exception Scope_cancelled + +let create ~clock ~namespace ~assets ~store ~notify_outbox_changed = + let join_with_timeout seconds callback = + Eio.Fiber.first callback (fun () -> + Eio.Time.Mono.sleep clock seconds; + Error + (Dsl.Cleanup_failure + { primary = None + ; failures = + Rrbvec.singleton + (Dsl.Storage_failure + "Worker scopes or consumer leases have not physically joined") + })) + in + { namespace + ; assets + ; store + ; staging_mutex = Eio.Mutex.create () + ; notify = notify_outbox_changed + ; binding = None + ; scopes = Hashtbl.create 8 + ; leases = 0 + ; changed = Eio.Condition.create () + ; sync_changed = Eio.Condition.create () + ; sync_revision = 0L + ; fenced = false + ; join_with_timeout + } +;; + +let identity t = Assets.identity t.assets + +let same_owner t requested = + if requested = identity t then Ok () else Error (Dsl.Identity_mismatch "Worker owner") +;; + +let binding t = + match t.binding with + | Some binding -> Ok binding + | None -> Error Dsl.Stale_context +;; + +let current scope = + if scope.cancelled + then Error Dsl.Cancelled + else if (not scope.active) || scope.owner.fenced + then Error Dsl.Stale_context + else ( + match Hashtbl.find_opt scope.owner.scopes scope.operation with + | Some registered when registered == scope -> Ok () + | Some _ | None -> Error Dsl.Stale_context) +;; + +let with_store t callback = + Eio.Mutex.use_rw ~protect:true t.store.mutex (fun () -> + try callback t.store.connection with + | Sqlite3.Error message | Sqlite3.SqliteError message -> + Error (Dsl.Storage_failure message)) +;; + +let storage result = Result.map_error (fun message -> Dsl.Storage_failure message) result + +let uncertain result = + Result.map_error + (fun message -> + Dsl.Outcome_unknown ("Import store commit response is unknown: " ^ message)) + result +;; + +let input_owner t = + let owner = identity t in + let* graph = + Uuid.of_string owner.graph + |> Result.map_error (fun message -> Dsl.Invalid_input message) + in + let namespace = t.namespace in + Ok Store.Inputs.{ namespace; origin = owner.origin; account = owner.account; graph } +;; + +let checked_input t bytes = + let* owner = input_owner t in + let* input = + Input.Pure_intent.restore ~namespace:owner.namespace bytes + |> Result.map_error (fun _ -> Dsl.Invalid_input "Invalid original import input") + in + Ok (owner, input) +;; + +let matches_owner t (intent : Intent.t) = + let owner = identity t in + intent.origin = owner.origin + && intent.account = owner.account + && Uuid.to_string intent.graph = owner.graph +;; + +let proof scope (intent : Intent.t) = + let* () = current scope in + if intent.operation_id <> scope.operation || not (matches_owner scope.owner intent) + then Error (Dsl.Identity_mismatch "Import operation") + else + let* owner = input_owner scope.owner in + with_store scope.owner (fun db -> + let* row = storage (Store.Inputs.read db ~owner ~operation:scope.operation) in + match row with + | Some row -> + (match row.prepared with + | Some prepared -> + let* restored = + Intent.restore prepared ~phase:intent.phase ~revision:intent.revision + |> Result.map_error (fun message -> Dsl.Invalid_input message) + in + if restored = intent + then Ok prepared + else Error (Dsl.Identity_mismatch "Original Prepared import proof") + | None -> Error (Dsl.Storage_failure "Original Prepared import proof is missing")) + | None -> Error (Dsl.Identity_mismatch "Unreserved import operation")) +;; + +let durable_intent scope intent = + let* _ = proof scope intent in + with_store scope.owner (fun db -> + let* stored = storage (Store.read db ~operation:scope.operation) in + match stored with + | Some stored when stored = intent -> Ok () + | Some _ -> Error (Dsl.Invalid_input "Import checkpoint revision is stale") + | None -> Error (Dsl.Storage_failure "Original import checkpoint is missing")) +;; + +let with_database scope callback = + let* () = current scope in + let* bound = binding scope.owner in + let* database = Execute.database bound.owner in + let* () = current scope in + callback database +;; + +let database_changed t ~owner ~identity:requested = + let* () = same_owner t requested in + let* bound = binding t in + if bound.owner != owner + then Error (Dsl.Identity_mismatch "Worker database owner") + else ( + t.sync_revision <- Int64.succ t.sync_revision; + Eio.Condition.broadcast t.sync_changed; + Ok ()) +;; + +let projection_changed t ~identity:requested = + let* bound = binding t in + database_changed t ~owner:bound.owner ~identity:requested +;; + +let notify (scope : scope) = + let* bound = binding scope.owner in + scope.owner.notify ~owner:bound.owner ~identity:(identity scope.owner) +;; + +let bind_runtime t ~identity:requested ~database_owner ~limits ~cleanup_seconds ~network = + let* () = same_owner t requested in + if + limits <> Assets.limits t.assets + || (not (Float.is_finite cleanup_seconds)) + || cleanup_seconds <= 0. + then Error (Dsl.Invalid_input "Worker owner configuration mismatch") + else ( + match t.binding with + | Some _ -> Error (Dsl.Invalid_input "Worker owner is already bound") + | None when t.fenced -> Error Dsl.Stale_context + | None -> + t.binding <- Some { owner = database_owner; network; cleanup_seconds }; + Ok ()) +;; + +let with_scope t ~identity:requested ~operation callback = + let* () = same_owner t requested in + let* _ = binding t in + if t.fenced + then Error Dsl.Stale_context + else if Hashtbl.mem t.scopes operation + then Error (Dsl.Capacity_exceeded "Import operation already has a live scope") + else if Hashtbl.length t.scopes >= (Assets.limits t.assets).maximum_tasks + then Error (Dsl.Capacity_exceeded "Worker scopes") + else ( + try + Eio.Cancel.sub (fun cancel_context -> + let scope = + { owner = t; operation; active = true; cancelled = false; cancel_context } + in + Hashtbl.add t.scopes operation scope; + Fun.protect + ~finally:(fun () -> + scope.active <- false; + Hashtbl.remove t.scopes operation; + Eio.Condition.broadcast t.changed) + (fun () -> callback scope)) + with + | Eio.Cancel.Cancelled Scope_cancelled -> Error Dsl.Cancelled) +;; + +let crypto (scope : scope) operation = + Logseq_sync_crypto.Core.run + (Assets.crypto scope.owner.assets) + ~check_current:(fun () -> current scope) + ~identity:(identity scope.owner) + operation +;; + +let prepare_import_unlocked scope (request : Input.t) ~current:request_current = + let current scope = + let* () = current scope in + if request_current () then Ok () else Error Dsl.Stale_context + in + let* () = current scope in + if scope.operation <> request.operation + then Error (Dsl.Identity_mismatch "Import scope operation") + else + let* owner = input_owner scope.owner in + let* input = + Input.Pure_intent.create ~namespace:owner.namespace request + |> Result.map_error (fun _ -> Dsl.Invalid_input "Invalid complete import input") + in + let* reserved = + with_store scope.owner (fun db -> + let* () = current scope in + uncertain (Store.Inputs.reserve db ~owner ~input)) + in + let* () = current scope in + let* existing = + with_store scope.owner (fun db -> + storage (Store.read db ~operation:request.operation)) + in + let* () = current scope in + match existing, reserved.prepared with + | Some intent, Some _ -> + let* _ = proof scope intent in + Ok intent + | None, Some _ | Some _, None -> + Error + (Dsl.Storage_failure + "Checkpoint and original Prepared proof disagree; staging is retained") + | None, None -> + let* staged = + Assets.stage scope.owner.assets ~check_current:(fun () -> current scope) request + in + let prepare () = + let* () = current scope in + let* version = + Logseq_db_types.Asset_descriptor.version + ~checksum:staged.checksum + ~file_type:request.file_type + |> Result.map_error (fun message -> Dsl.Invalid_input message) + in + let* intent = + Intent.prepare + ~operation_id:request.operation + ~origin:owner.origin + ~account:owner.account + ~graph:owner.graph + ~asset:request.asset + ~version + ~title:request.title + ~size:staged.size + ~staged_file:staged.file + ~replace_reference:request.replace_reference + ~target:request.target + ~local_mutation:request.local_mutation + ~metadata_mutation:request.metadata_mutation + |> Result.map_error (fun message -> Dsl.Invalid_input message) + in + Ok intent + in + let discard error = + match Cache.release_staged (Assets.cache scope.owner.assets) ~file:staged.file with + | Ok () -> Error error + | Error cleanup -> Error (Dsl.Cleanup_failure + { primary = Some error + ; failures = Rrbvec.singleton (Assets.cache_error cleanup) }) + in + (match prepare () with + | Error error -> discard error + | Ok intent -> + let admitted = ref false in + match with_store scope.owner (fun db -> + let* () = current scope in + admitted := true; + uncertain (Store.Inputs.commit_prepared db ~owner ~input intent)) with + | Ok _ -> Ok intent + | Error error when not !admitted -> discard error + | Error error -> Error error) +;; + +let prepare_import (scope : scope) request ~current = + Eio.Mutex.use_rw ~protect:true scope.owner.staging_mutex (fun () -> + prepare_import_unlocked scope request ~current) +;; + +let reconcile_staging t ~check_current = + Eio.Mutex.use_rw ~protect:true t.staging_mutex (fun () -> + let* () = check_current () in + let* owner = input_owner t in + with_store t (fun db -> + let keep file = + let result = + let* () = check_current () in + let* operation = + match String.split_on_char '.' file with + | operation :: _ -> Uuid.of_string operation |> storage + | [] -> Error (Dsl.Invalid_input "Invalid staged file") + in + let* intent = storage (Store.read db ~operation) in + let* input = storage (Store.Inputs.read db ~owner ~operation) in + let matches (intent : Intent.t) = + intent.origin = owner.origin && intent.account = owner.account + && intent.graph = owner.graph && intent.staged_file = file + in + Ok (Option.exists matches intent + || Option.exists (fun row -> Option.exists matches row.Store.Inputs.prepared) input) + in + Result.map_error Dsl.error_message result + in + Cache.prune_staged (Assets.cache t.assets) ~keep + |> Result.map_error Assets.cache_error)) +;; + +let publish_import_preview scope intent = + let* () = durable_intent scope intent in + Assets.publish_preview + scope.owner.assets + ~check_current:(fun () -> current scope) + intent +;; + +let persist_import scope intent ~expected = + let* _ = proof scope intent in + with_store scope.owner (fun db -> + let* () = current scope in + uncertain (Store.save db ~expected intent)) +;; + +let inspect_import scope intent = + let* () = durable_intent scope intent in + let* observation = + with_database scope (fun database -> Import_database.inspect database intent) + in + Ok + (match observation with + | Import_database.U.Import_absent -> `Import_absent + | Local_present -> `Local_present + | Metadata_present -> `Metadata_present + | Publication_acknowledged -> `Publication_acknowledged + | Entity_cancelled -> `Entity_cancelled) +;; + +let apply_local_import scope intent = + let* () = durable_intent scope intent in + with_database scope (fun database -> + Eio.Cancel.protect (fun () -> + let* () = current scope in + let* () = Import_database.apply_local database intent in + notify scope)) +;; + +let put_import scope intent = + let* () = durable_intent scope intent in + let* bound = binding scope.owner in + Assets.put_staged + scope.owner.assets + ~check_current:(fun () -> current scope) + ~network:bound.network + intent +;; + +let apply_import_metadata scope intent = + let* () = durable_intent scope intent in + with_database scope (fun database -> + Eio.Cancel.protect (fun () -> + let* () = current scope in + let* () = Import_database.apply_metadata database intent in + notify scope)) +;; + +let await_import_publication scope intent = + let* () = durable_intent scope intent in + with_database scope (fun database -> + let sync_result = Result.map_error (fun error -> Dsl.Sync_read_failure error) in + let rec wait () = + let* () = current scope in + let* before = Database.inspect_sync database |> sync_result in + let* observation = inspect_import scope intent in + match observation with + | `Publication_acknowledged -> Ok () + | `Import_absent | `Entity_cancelled -> + Error (Dsl.Invalid_input "Original import mutation was not published") + | `Local_present | `Metadata_present -> + let* _ = + Database.await_sync_change + database + ~after:(Logseq_overlay_db.Types.sync_view_token before) + |> sync_result + in + wait () + in + wait ()) +;; + +let cancel_import scope intent = + let* () = durable_intent scope intent in + match intent.Intent.phase with + | Complete | Cancelled -> Ok () + | Prepared | Local_committed | Uploading | Remote_stored | Metadata_pending -> + let* cancelled = + Intent.advance intent Cancelled + |> Result.map_error (fun message -> Dsl.Invalid_input message) + in + persist_import scope cancelled ~expected:(Some intent.revision) +;; + +let release_import_staging scope intent = + let* () = durable_intent scope intent in + let* () = + match intent.Intent.phase with + | Cancelled -> Ok () + | Complete -> + let* observed = inspect_import scope intent in + if observed = `Publication_acknowledged + then ( + let cache = Assets.cache scope.owner.assets in + let* cached = + Cache.lookup cache ~asset:intent.asset ~version:intent.version + |> Result.map_error Assets.cache_error + in + match cached with + | Some handle -> + Cache.release_checked cache handle |> Result.map_error Assets.cache_error + | None when Cache.staged_path cache ~file:intent.staged_file = None -> Ok () + | None -> + Assets.publish_preview + scope.owner.assets + ~check_current:(fun () -> current scope) + intent) + else Error (Dsl.Invalid_input "Terminal import has no publication proof") + | Prepared | Local_committed | Uploading | Remote_stored | Metadata_pending -> + Error (Dsl.Invalid_input "Import staging is not terminal") + in + Cache.release_staged (Assets.cache scope.owner.assets) ~file:intent.staged_file + |> Result.map_error Assets.cache_error +;; + +let input_wire (row : Store.Inputs.record) : Input.Recovery.item = + { sequence = row.sequence + ; input = Input.Pure_intent.encode row.input + ; prepared = row.evidence + ; acknowledged = row.acknowledged + } +;; + +let recover_import_inputs scope ~after ~maximum_items ~maximum_bytes = + let* () = current scope in + if + maximum_items <= 0 + || maximum_items + > min Input.Recovery.maximum_items (Assets.limits scope.owner.assets).maximum_items + || maximum_bytes <= 0 + || maximum_bytes + > min Input.Recovery.maximum_bytes (Assets.limits scope.owner.assets).maximum_bytes + then Error (Dsl.Invalid_input "Import recovery bounds") + else + let* owner = input_owner scope.owner in + with_store scope.owner (fun db -> + let* page = + storage (Store.Inputs.pending db ~owner ~after ~maximum_items ~maximum_bytes) + in + Ok Input.Recovery.{ items = List.map input_wire page.items; next = page.next }) +;; + +let confirm_import_input scope bytes = + let* () = current scope in + let* owner, input = checked_input scope.owner bytes in + with_store scope.owner (fun db -> + let* row = + storage (Store.Inputs.read db ~owner ~operation:(Input.Pure_intent.operation input)) + in + match row with + | None -> Ok None + | Some row when Input.Pure_intent.encode row.input = bytes -> + Ok (Some (input_wire row)) + | Some _ -> Error (Dsl.Identity_mismatch "Original complete import input")) +;; + +let acknowledge_import_input scope ~input:bytes ~prepared = + let* () = current scope in + let* owner, input = checked_input scope.owner bytes in + with_store scope.owner (fun db -> + let* () = current scope in + uncertain (Store.Inputs.acknowledge db ~owner ~input ~evidence:prepared)) +;; + +let cancel scope = + if not scope.active + then Error Dsl.Stale_context + else ( + scope.cancelled <- true; + Eio.Cancel.cancel scope.cancel_context Scope_cancelled; + Ok ()) +;; + +let retain_imported_file scope intent = + let* () = durable_intent scope intent in + let t = scope.owner in + if t.leases >= (Assets.limits t.assets).maximum_resources + then Error (Dsl.Capacity_exceeded "Imported file leases") + else ( + match Cache.retain_staged (Assets.cache t.assets) ~file:intent.Intent.staged_file with + | None -> Ok None + | Some handle -> + (match Cache.path (Assets.cache t.assets) handle with + | Some path -> + t.leases <- t.leases + 1; + Ok (Some { owner = t; handle; path; released = false }) + | None -> + let* () = + Cache.release_checked (Assets.cache t.assets) handle + |> Result.map_error Assets.cache_error + in + Error (Dsl.Storage_failure "Retained import staging disappeared"))) +;; + +let imported_file_path file = file.path + +let release_imported_file file = + if file.released + then Ok () + else + let* () = + Cache.release_checked (Assets.cache file.owner.assets) file.handle + |> Result.map_error Assets.cache_error + in + file.released <- true; + file.owner.leases <- file.owner.leases - 1; + Eio.Condition.broadcast file.owner.changed; + Ok () +;; + +let fence t ~identity:requested = + let* () = same_owner t requested in + t.fenced <- true; + Assets.fence t.assets; + Hashtbl.iter + (fun _ scope -> + scope.cancelled <- true; + Eio.Cancel.cancel scope.cancel_context Scope_cancelled) + t.scopes; + Ok () +;; + +let await_io t ~identity:requested = + let* () = same_owner t requested in + if not t.fenced then Error (Dsl.Invalid_input "Fence Worker before joining") + else + let* bound = binding t in + let rec wait () = + if Hashtbl.length t.scopes = 0 then Ok () + else (Eio.Condition.await_no_mutex t.changed; wait ()) + in + t.join_with_timeout bound.cleanup_seconds wait +;; + +let await_quiescent t ~identity:requested = + let* () = same_owner t requested in + if not t.fenced + then Error (Dsl.Invalid_input "Fence Worker before joining") + else ( + let rec wait () = + if Hashtbl.length t.scopes = 0 && t.leases = 0 + then Ok () + else ( + Eio.Condition.await_no_mutex t.changed; + wait ()) + in + let* bound = binding t in + t.join_with_timeout bound.cleanup_seconds wait) +;; + +let original_import t (asset : Dsl.asset) = + match asset.source, asset.current_checksum with + | Logseq_db_types.Asset_descriptor.External _, _ | Managed _, None -> Ok None + | Managed _, Some checksum -> + let* owner = input_owner t in + with_store t (fun db -> + storage + (Store.find_original + db + ~origin:owner.origin + ~account:owner.account + ~graph:owner.graph + ~asset:asset.uuid + ~checksum)) +;; + +let upload_asset t asset = + let* original = original_import t asset in + match original with + | None -> Error (Dsl.Storage_failure "No original durable import for asset upload") + | Some intent -> + with_scope t ~identity:(identity t) ~operation:intent.operation_id (fun scope -> + let step intent phase action = + let* () = action intent in + let* next = + Intent.advance intent phase + |> Result.map_error (fun message -> Dsl.Invalid_input message) + in + let* () = persist_import scope next ~expected:(Some intent.revision) in + Ok next + in + let rec resume intent = + match intent.Intent.phase with + | Prepared -> + let* next = step intent Local_committed (apply_local_import scope) in + resume next + | Local_committed -> + let* next = step intent Uploading (fun _ -> Ok ()) in + resume next + | Uploading -> + let* next = step intent Remote_stored (put_import scope) in + resume next + | Remote_stored -> + let* next = step intent Metadata_pending (apply_import_metadata scope) in + resume next + | Metadata_pending -> + let* next = step intent Complete (await_import_publication scope) in + resume next + | Complete -> release_import_staging scope intent + | Cancelled -> + let* () = release_import_staging scope intent in + Error Dsl.Cancelled + in + resume intent) +;; + +let run_assets : type value. t -> identity:Dsl.identity -> operation:Uuid.t -> value Dsl.assets -> (value, Dsl.error) result = + fun t ~identity ~operation action -> + match action with + | Dsl.Release_file file -> Assets.release_file t.assets ~identity file + | Release_lease lease -> Assets.release_lease t.assets ~identity lease + | _ -> + with_scope t ~identity ~operation (fun scope -> + let* bound = binding t in + let run () : (value, Dsl.error) result = + match action with + | Dsl.Reconcile_staging -> reconcile_staging t ~check_current:(fun () -> current scope) + | _ -> Assets.run t.assets + ~check_current:(fun () -> current scope) + ~admit_network:(fun () -> current scope) + ~original:(original_import t) ~upload:(upload_asset t) ~identity action in + match action with + | Dsl.Download _ | Upload _ -> bound.network.with_admission run + | _ -> run ()) +;; + +let asset_file_path t file = Assets.file_path t.assets file + +let assets_dispatch (type value) t ~scope ~identity (operation : value Dsl.assets) + : (value, Dsl.error) result = + match operation with + | Dsl.Reconcile_staging -> + let* () = same_owner t identity in + let* _ = binding t in + reconcile_staging t ~check_current:(fun () -> + if t.fenced then Error Dsl.Stale_context else Execute.check_current scope) + | _ -> Assets.dispatch + t.assets + ~original:(original_import t) + ~upload:(upload_asset t) + ~scope + ~identity + operation +;; + +let crypto_dispatch t ~scope ~identity operation = + Logseq_sync_crypto.Core.dispatch (Assets.crypto t.assets) ~scope ~identity operation +;; + +let worker_owner t = + (module struct + type nonrec scope = scope + type nonrec imported_file = imported_file + + let bind_runtime = bind_runtime t + + let with_scope ~identity ~operation callback = + with_scope t ~identity ~operation callback + ;; + + let crypto = crypto + let prepare_import = prepare_import + let publish_import_preview = publish_import_preview + let persist_import = persist_import + let inspect_import = inspect_import + let apply_local_import = apply_local_import + let put_import = put_import + let apply_import_metadata = apply_import_metadata + let await_import_publication = await_import_publication + let cancel_import = cancel_import + let release_import_staging = release_import_staging + let recover_import_inputs = recover_import_inputs + let confirm_import_input = confirm_import_input + let acknowledge_import_input = acknowledge_import_input + let cancel = cancel + let retain_imported_file = retain_imported_file + let imported_file_path = imported_file_path + let release_imported_file = release_imported_file + let fence = fence t + let await_quiescent = await_quiescent t + end : Execute.WORKER_OWNER) +;; diff --git a/logseq_sync/lib/worker_owner/core.mli b/logseq_sync/lib/worker_owner/core.mli new file mode 100644 index 00000000..ffec2258 --- /dev/null +++ b/logseq_sync/lib/worker_owner/core.mli @@ -0,0 +1,62 @@ +module Dsl = Sync_dsl.Core +module Intent = Logseq_db_types.Asset_upload_intent +module Assets = Logseq_sync_assets.Core + +type store + +(** Borrow the existing upload-store connection; this owner never opens a second store or closes it. *) +val store : Sqlite3.db -> store + +type t + +(** Namespace comes from the application's authoritative logical-owner mapper. *) +val create + : clock:'clock Eio.Time.Mono.t + -> namespace:string + -> assets:Assets.t + -> store:store + -> notify_outbox_changed: + (owner:Execute.Core.database_owner + -> identity:Dsl.identity + -> (unit, Dsl.error) result) + -> t + +val database_changed + : t + -> owner:Execute.Core.database_owner + -> identity:Dsl.identity + -> (unit, Dsl.error) result + +val projection_changed : t -> identity:Dsl.identity -> (unit, Dsl.error) result + +val worker_owner : t -> (module Execute.Core.WORKER_OWNER) + +val run_assets + : t + -> identity:Dsl.identity + -> operation:Logseq_db_types.Graph_types.Uuid.t + -> 'a Dsl.assets + -> ('a, Dsl.error) result + +val asset_file_path : t -> Dsl.asset_file -> (string, Dsl.error) result + +val assets_dispatch + : t + -> scope:Execute.Core.effect_scope + -> identity:Dsl.identity + -> 'a Dsl.assets + -> ('a, Dsl.error) result + +val crypto_dispatch + : t + -> scope:Execute.Core.effect_scope + -> identity:Dsl.identity + -> 'a Dsl.crypto + -> ('a, Dsl.error) result + +val original_import : t -> Dsl.asset -> (Intent.t option, Dsl.error) result +val fence : t -> identity:Dsl.identity -> (unit, Dsl.error) result + +(** Join admitted scopes after fence; consumer leases remain owned. *) +val await_io : t -> identity:Dsl.identity -> (unit, Dsl.error) result +val await_quiescent : t -> identity:Dsl.identity -> (unit, Dsl.error) result diff --git a/logseq_sync/lib/worker_owner/dune b/logseq_sync/lib/worker_owner/dune new file mode 100644 index 00000000..9abde614 --- /dev/null +++ b/logseq_sync/lib/worker_owner/dune @@ -0,0 +1,5 @@ +(library + (name logseq_sync_worker_owner) + (public_name logseq_sync.worker_owner) + (modules core import_database) + (libraries eio execute sync_dsl logseq_sync.assets logseq_sync.crypto logseq_db_types logseq_db_storage logseq_overlay_db rrbvec sqlite3 unix uri)) diff --git a/logseq_sync/lib/worker_owner/import_database.ml b/logseq_sync/lib/worker_owner/import_database.ml new file mode 100644 index 00000000..060ea806 --- /dev/null +++ b/logseq_sync/lib/worker_owner/import_database.ml @@ -0,0 +1,208 @@ +module U = struct + type observation = + | Import_absent + | Local_present + | Metadata_present + | Publication_acknowledged + | Entity_cancelled +end + +module I = Logseq_db_types.Asset_upload_intent +module A = Logseq_db_types.Asset_descriptor +module D = Logseq_overlay_db.Database +module T = Logseq_overlay_db.Types +module Diagnostic = Logseq_db_types.Failure_diagnostic + +let ( let* ) = Result.bind + +let reject on_diagnostic reason source = + let detail = + match source with + | None -> Diagnostic.create ~operation:"assetUpload" ~reason () + | Some source -> Diagnostic.of_source ~operation:"assetUpload" ~reason source + in + (try on_diagnostic detail with + | _ -> ()); + Error + (if reason = "persistenceFailed" + then Sync_dsl.Core.Outcome_unknown "Import mutation commit response is unknown" + else Sync_dsl.Core.Asset_failure detail) +;; + +let map_error on_diagnostic classify = function + | Ok value -> Ok value + | Error error -> + let reason, source = classify error in + reject on_diagnostic reason source +;; + +let read on_diagnostic = + map_error on_diagnostic (function + | T.Database_closed | Snapshot_released | Snapshot_generation_invalidated -> + "databaseClosed", None + | Read_limit_exceeded -> "limitExceeded", None + | Stale_read_cursor -> "staleCommit", None + | Invalid_read_request source -> "invalidRequest", Some source + | Fatal_read_state source -> "fatalState", Some source) +;; + +let local_result on_diagnostic = + map_error on_diagnostic (function + | T.Local_database_closed -> "databaseClosed", None + | Invalid_local_mutation source -> "invalidMutation", Some source + | Mutation_identity_conflict -> "identityConflict", None + | Missing_precondition _ -> "missingPrecondition", None + | Target_precondition_conflict -> "preconditionConflict", None + | Planner_limit_exceeded -> "limitExceeded", None + | Delete_unsupported_for_footprint -> "unsupportedFootprint", None + | Local_commit_busy -> "busy", None + | Local_commit_persistence_failed source -> "persistenceFailed", Some source + | Local_commit_fatal_state source -> "fatalState", Some source) +;; + +let sync_result on_diagnostic = + map_error on_diagnostic (function + | T.Sync_database_closed -> "databaseClosed", None + | Sync_fatal_state source -> "fatalState", Some source) +;; + +let precondition on_diagnostic = + map_error on_diagnostic (function + | T.Duplicate_block_precondition _ + | Duplicate_page_precondition _ + | Duplicate_scope_precondition _ + -> "preconditionConflict", None) +;; + +let local (i : I.t) = + T.Insert_blocks + { mutation_id = i.local_mutation + ; parent = i.target + ; tree = { uuid = i.asset; title = i.title; children = [] } + ; asset = + Some + { replace_reference = i.replace_reference; version = i.version; size = i.size } + } +;; + +let metadata (i : I.t) = + T.Publish_asset + { mutation_id = i.metadata_mutation; block = i.asset; version = i.version } +;; + +let snapshot on_diagnostic db f = + let* s = read on_diagnostic (D.current_snapshot db) in + Fun.protect ~finally:(fun () -> D.release_snapshot s) (fun () -> f s) +;; + +let applied = function + | Some (T.Existing_applied _) -> true + | _ -> false +;; + +let rejected = function + | Some (T.Existing_blocked _ | Existing_discarded _ | Existing_remote_won _) -> true + | _ -> false +;; + +let inspect ?(on_diagnostic = fun _ -> ()) db (i : I.t) = + let* insertion = local_result on_diagnostic (D.inspect_local_mutation db (local i)) in + let* publication = + local_result on_diagnostic (D.inspect_local_mutation db (metadata i)) + in + if rejected insertion || rejected publication + then Ok U.Entity_cancelled + else if not (applied insertion) + then Ok U.Import_absent + else + let* assets = + snapshot on_diagnostic db (fun s -> + read on_diagnostic (D.get_asset_descriptors s [ i.asset ])) + in + match assets with + | [] -> Ok (if applied insertion then U.Entity_cancelled else U.Import_absent) + | [ asset ] when asset.current_checksum = Some i.version.checksum -> + (match asset.source with + | A.Managed (Some version) when A.equal_version version i.version -> + let* sync = sync_result on_diagnostic (D.inspect_sync db) in + let pending = + List.exists + (fun (s : T.submission_descriptor) -> s.mutation_id = i.metadata_mutation) + (T.sync_view_submissions sync) + in + Ok + (if not (applied publication) + then U.Local_present + else if not pending + then U.Publication_acknowledged + else U.Metadata_present) + | A.Managed None -> Ok U.Local_present + | _ -> Ok U.Entity_cancelled) + | _ -> Ok U.Entity_cancelled +;; + +let commit on_diagnostic db mutation expected = + let* outcome = local_result on_diagnostic (D.commit_local db ~expected mutation) in + match outcome with + | T.Local_committed _ | Local_existing (Existing_applied _) -> Ok () + | Local_existing _ -> reject on_diagnostic "preconditionConflict" None +;; + +let apply_local ?(on_diagnostic = fun _ -> ()) db (i : I.t) = + let mutation = local i in + let* receipt = local_result on_diagnostic (D.inspect_local_mutation db mutation) in + if applied receipt + then Ok () + else if rejected receipt + then reject on_diagnostic "preconditionConflict" None + else + let* expected = + snapshot on_diagnostic db (fun s -> + let* pages = read on_diagnostic (D.get_pages s [ i.target ]) in + let* blocks, pages = + match pages with + | [ T.Present_page { revision; _ } ] -> Ok ([], [ i.target, revision ]) + | _ -> + let* blocks = read on_diagnostic (D.get_outline_blocks s [ i.target ]) in + (match blocks with + | [ T.Present_block { revision; _ } ] -> Ok ([ i.target, revision ], []) + | _ -> reject on_diagnostic "missingPrecondition" None) + in + let* structure = + read + on_diagnostic + (D.get_structure + s + (T.Children { parent = i.target; limit = 1; cursor = None })) + in + match structure with + | T.Children_result { revision_scope; scope_revision; _ } -> + precondition + on_diagnostic + (D.write_precondition + ~blocks + ~pages + ~scopes:[ revision_scope, scope_revision ]) + | _ -> reject on_diagnostic "invalidContent" None) + in + commit on_diagnostic db mutation expected +;; + +let apply_metadata ?(on_diagnostic = fun _ -> ()) db (i : I.t) = + let* observation = inspect ~on_diagnostic db i in + match observation with + | U.Metadata_present | Publication_acknowledged -> Ok () + | Local_present -> + let* expected = + snapshot on_diagnostic db (fun s -> + let* blocks = read on_diagnostic (D.get_outline_blocks s [ i.asset ]) in + match blocks with + | [ T.Present_block { revision; _ } ] -> + precondition + on_diagnostic + (D.write_precondition ~blocks:[ i.asset, revision ] ~pages:[] ~scopes:[]) + | _ -> reject on_diagnostic "missingPrecondition" None) + in + commit on_diagnostic db (metadata i) expected + | _ -> reject on_diagnostic "missingPrecondition" None +;; diff --git a/logseq_sync/lib/worker_owner/import_database.mli b/logseq_sync/lib/worker_owner/import_database.mli new file mode 100644 index 00000000..cf977620 --- /dev/null +++ b/logseq_sync/lib/worker_owner/import_database.mli @@ -0,0 +1,26 @@ +module U : sig + type observation = + | Import_absent + | Local_present + | Metadata_present + | Publication_acknowledged + | Entity_cancelled +end + +val inspect + : ?on_diagnostic:(Logseq_db_types.Failure_diagnostic.t -> unit) + -> Logseq_overlay_db.Database.t + -> Logseq_db_types.Asset_upload_intent.t + -> (U.observation, Sync_dsl.Core.error) result + +val apply_local + : ?on_diagnostic:(Logseq_db_types.Failure_diagnostic.t -> unit) + -> Logseq_overlay_db.Database.t + -> Logseq_db_types.Asset_upload_intent.t + -> (unit, Sync_dsl.Core.error) result + +val apply_metadata + : ?on_diagnostic:(Logseq_db_types.Failure_diagnostic.t -> unit) + -> Logseq_overlay_db.Database.t + -> Logseq_db_types.Asset_upload_intent.t + -> (unit, Sync_dsl.Core.error) result diff --git a/logseq_sync/spec/effect_runner/asset_cache.mli b/logseq_sync/spec/effect_runner/asset_cache.mli deleted file mode 100644 index 234c4732..00000000 --- a/logseq_sync/spec/effect_runner/asset_cache.mli +++ /dev/null @@ -1,30 +0,0 @@ -(** {1 TLDR} - - Name the opaque cache resources owned by the Sync effect runner. - There is no public cache constructor or execution API here: business requests - enter [Logseq_sync_pure_reducer.Core.step], and the runner performs cache I/O. - Cache handles and staged values are process-local resources, not durable proofs. *) - -type t - -type handle -type error -type staged - -(** {1 Minimal usage example} - - The caller supplies a current reducer [state], an [event], and the actual - [runner]. This is the public execution path, not direct cache access: - {[ - let transition = Logseq_sync_pure_reducer.Core.step !state event in - state := transition.next; - List.iter - (function - | Logseq_sync_pure_reducer.Core.Run runner_effect -> - Effect_runner.submit runner runner_effect - | instruction -> interpret_other instruction) - transition.effects - ]} - The caller supplies [interpret_other] for delegated work and publications. - Runner callbacks queue events for a later serialized reducer turn; the runner - owns cache cleanup and is shut down by its lifecycle owner. *) diff --git a/logseq_sync/spec/effect_runner/asset_codec.mli b/logseq_sync/spec/effect_runner/asset_codec.mli deleted file mode 100644 index 7a9b6265..00000000 --- a/logseq_sync/spec/effect_runner/asset_codec.mli +++ /dev/null @@ -1,61 +0,0 @@ -(** {1 TLDR} - - Encode and verify bounded asset file bytes, using caller-supplied crypto. - [encode] returns wire bytes or an error; [decode] alone creates {!verified} - plaintext bound to the expected asset version. No file I/O or persistence - occurs here. Envelopes encrypt raw file bytes, not Transit title values. - An envelope failure must never fall back to raw bytes. *) - -type crypto = - { encrypt : key:string -> plaintext:string -> (string * string, string) result - ; decrypt : key:string -> iv:string -> ciphertext:string -> (string, string) result - } - -val encode - : maximum_plaintext_bytes:int - -> crypto:crypto - -> key:string option - -> string - -> (string, string) result - -type decode_error = - | Invalid_limit - | Wire_too_large - | Plaintext_too_large - | Malformed_envelope - | Malformed_transit of string - | Decryption_failed of string - | Checksum_mismatch - -(** Only a successful bounded decode creates this value. It binds plaintext to - the expected version; there is no unchecked bytes-to-verified constructor. *) -type verified -val verified_bytes : verified -> string -val verified_version : verified -> Logseq_db_types.Asset_descriptor.version - -(** Preserve conservative raw/envelope classification. A candidate's parsing or - decryption failure never becomes raw data, even if its wire checksum matches. *) -val decode - : maximum_plaintext_bytes:int - -> crypto:crypto - -> key:string option - -> expected:Logseq_db_types.Asset_descriptor.version - -> string - -> (verified, decode_error) result - -(** {1 Minimal usage example} - - The caller supplies [crypto], [key], [expected], [wire] and a positive - [maximum_plaintext_bytes] matching its resource budget: - {[ - match decode ~maximum_plaintext_bytes ~crypto ~key ~expected wire with - | Ok verified -> Ok (verified_version verified, verified_bytes verified) - | Error error -> Error (decode_error_message error) - ]} - Verification failure leaves the caller responsible for rejecting the input; - no cache handle or file is acquired by this operation. *) - -(** {1 Observation and helpers} *) - -val checksum : string -> string -val decode_error_message : decode_error -> string diff --git a/logseq_sync/spec/effect_runner/dune b/logseq_sync/spec/effect_runner/dune deleted file mode 100644 index d674904e..00000000 --- a/logseq_sync/spec/effect_runner/dune +++ /dev/null @@ -1,7 +0,0 @@ -(library - (name logseq_sync_effect_runner) - (public_name logseq_sync.effect_runner) - (modules effect_runner asset_codec asset_cache) - (virtual_modules effect_runner asset_codec asset_cache) - (default_implementation logseq_sync_effect_runner_impl) - (libraries eio logseq_db_types logseq_sync.pure_reducer uri x509)) diff --git a/logseq_sync/spec/effect_runner/effect_runner.mli b/logseq_sync/spec/effect_runner/effect_runner.mli deleted file mode 100644 index 2a4f853c..00000000 --- a/logseq_sync/spec/effect_runner/effect_runner.mli +++ /dev/null @@ -1,185 +0,0 @@ -(** {1 TLDR} - - Interpret [Logseq_sync_pure_reducer.Core.runner_effect] with Eio and - caller-supplied validated dependencies. {!create} binds a runner to an Eio - switch; {!submit} performs I/O and queues exact Core completion events through - [post]. The serial reducer remains the state owner. Asset and protected-value - tickets execute at most once during this runner's lifetime; that process-local - rule is not durable replay evidence. The lifecycle owner calls {!shutdown}. - - Construction depends on the capability types below. Primary excerpts: - {[ - val create : - ?on_asset_diagnostic:(scope:Logseq_sync_pure_reducer.Core.graph_scope - -> operation:string -> request_operation:string - -> Logseq_db_types.Failure_diagnostic.t -> unit) - -> sw:Eio.Switch.t -> dependencies - -> post:(Logseq_sync_pure_reducer.Core.event -> unit) - -> (t, create_error) result - val submit : t -> Logseq_sync_pure_reducer.Core.runner_effect -> unit - val shutdown : t -> unit - ]} *) - -type t -type dependency_error = Invalid_dependency of string -type create_error = Invalid_create of string - -(** {1 Minimal usage example} - - The caller supplies [sw], validated [dependencies], a Core-produced [runner_effect], - and [post] that queues later serialized reducer turns: - {[ - match create ~sw dependencies ~post with - | Error error -> Error error - | Ok runner -> - Fun.protect - ~finally:(fun () -> shutdown runner) - (fun () -> - submit runner runner_effect; - run_until_retired runner; - Ok ()) - ]} - [run_until_retired] is the caller's event loop, not a runner API. It keeps the - switch, pump and callback delivery alive through the owner's retirement - protocol. [submit] returning is not the effect's completion acknowledgement. *) - -(** {1 Construction and validation} *) - -type runtime - -val runtime - : fork:(sw:Eio.Switch.t -> (unit -> unit) -> unit) - -> sleep:(float -> unit) - -> (runtime, dependency_error) result - -type transport -type tls_authenticator -type id_token_provider - -val id_token_provider - : acquire:(Logseq_sync_pure_reducer.Core.account_scope -> (string, string) result) - -> invalidate:(Logseq_sync_pure_reducer.Core.account_scope -> token:string -> unit) - -> id_token_provider - -type authenticated_failure = - | Unauthorized - | Forbidden - | Request_failed of string - -val tls_authenticator : X509.Authenticator.t -> tls_authenticator -val system_tls_authenticator : unit -> (tls_authenticator, dependency_error) result - -type websocket_liveness = - | Disabled - | Ping_pong of - { interval_seconds : float - ; timeout_seconds : float - } - -val transport - : tls_authenticator:tls_authenticator - -> network:_ Eio.Net.t - -> clock:_ Eio.Time.clock - -> websocket_liveness:websocket_liveness - -> (transport, dependency_error) result - -(** {2 Storage, secrets and crypto} *) - -type local_store - -val local_store - : ?asset_cache_budget_bytes:int64 - -> ?asset_maximum_file_bytes:int - -> ?asset_pending_budget_bytes:int64 - -> application_support_directory:string - -> unit - -> (local_store, dependency_error) result - -type artifact_store - -val artifact_store : staging_directory:string -> (artifact_store, dependency_error) result - -type wrapped_key_load_error = - | Wrapped_graph_key_unavailable of string - | Local_private_key_unavailable of string - -type secrets - -val secrets - : unlock_private_key: - (managed_sync_origin:Uri.t - -> user_id:string - -> password:string - -> private_key_package:string - -> (unit, string) result) - -> unlock_graph_key: - (managed_sync_origin:Uri.t - -> user_id:string - -> encrypted_graph_key:string - -> (string, string) result) - -> load_wrapped_graph_key: - (managed_sync_origin:Uri.t - -> user_id:string - -> graph_id:Logseq_sync_pure_reducer.Core.graph_id - -> (string, wrapped_key_load_error) result) - -> verify_and_save_wrapped_graph_key: - (managed_sync_origin:Uri.t - -> user_id:string - -> graph_id:Logseq_sync_pure_reducer.Core.graph_id - -> encrypted_graph_key:string - -> (unit, string) result) - -> delete_account_secrets: - (managed_sync_origin:Uri.t -> user_id:string -> (unit, string) result) - -> (secrets, dependency_error) result - -type crypto - -val crypto - : encrypt_aes_gcm:(key:string -> plaintext:string -> (string * string, string) result) - -> decrypt_aes_gcm: - (key:string -> iv:string -> ciphertext:string -> (string, string) result) - -> (crypto, dependency_error) result - -val apple_secrets : unit -> (secrets, dependency_error) result -val apple_crypto : unit -> (crypto, dependency_error) result - -type dependencies - -val dependencies - : runtime:runtime - -> transport:transport - -> local_store:local_store - -> artifact_store:artifact_store - -> secrets:secrets - -> crypto:crypto - -> id_token_provider:id_token_provider - -> (dependencies, dependency_error) result - -val create - : ?on_asset_diagnostic:(scope:Logseq_sync_pure_reducer.Core.graph_scope - -> operation:string - -> request_operation:string - -> Logseq_db_types.Failure_diagnostic.t - -> unit) - -> sw:Eio.Switch.t - -> dependencies - -> post:(Logseq_sync_pure_reducer.Core.event -> unit) - -> (t, create_error) result - -(** {1 Capability details} *) - -(** Asset and protected-value tickets execute at most once for the runner's lifetime. - Replaying an already submitted ticket does not execute I/O or post a second completion. *) -val submit : t -> Logseq_sync_pure_reducer.Core.runner_effect -> unit - -(** {1 Lifecycle and recovery} *) - -(** Checked physical cleanup for the original retired logical account. The - caller must hold its lifecycle fence and join admitted IO first. Retained - native leases preserve their cache and return Error; this issues no authority. *) -val retire_account:t -> account:Logseq_sync_pure_reducer.Core.account_scope -> (unit,string)result - -(** Stop this runner's admission and cancel owned work. The enclosing switch and - lifecycle owner still own actual I/O/callback joining; this unit return is not - a checked per-account cleanup receipt. *) -val shutdown : t -> unit diff --git a/logseq_sync/spec/execute/core.mli b/logseq_sync/spec/execute/core.mli new file mode 100644 index 00000000..3f0050e2 --- /dev/null +++ b/logseq_sync/spec/execute/core.mli @@ -0,0 +1,414 @@ +(** Contexts are runtime-linear: rejection preserves one; each admitted outcome returns the sole next context. Install it before starting again. *) + +module Dsl = Sync_dsl.Core + +type t +type 'state graph_context +type context = Context : 'state Dsl.phase * 'state graph_context -> context + +type ('after, 'value) outcome = + | Completed of 'after graph_context * 'value + | Failed of context * Dsl.error + +type ('after, 'value) running + +(** Fresh invocation bound to runtime/owner/incarnation/context version. *) +type effect_scope + +(** Atomic non-suspending masked handoff to reserved cleanup ledger: success transfers all references, + rejection none; prior transfers persist. Exact repeated transfers are idempotent. + Validate original owner/kind/metadata; revoked scopes accept original resources for cleanup only. + ACK forbids producer use/release. Handoff must precede returning resources. + Original owner records acquisitions before suspension, including raise/cancel before delivery; + producer resolves untransferred resources. Resource-owned services survive invocation join until release. + Charges persist through physical cleanup; consumer leases have independent lifetimes. *) +val handoff : effect_scope -> Dsl.resource_reference Rrbvec.t -> (unit, Dsl.error) result + +module Callback_bridge : sig + type 'a producer = + { start : + deliver:(('a, Dsl.error) result -> (unit, Dsl.error) result) + -> (unit, Dsl.error) result + ; stop : unit -> (unit, Dsl.error) result + ; await_stopped : deadline:float -> (unit, Dsl.error) result + } + + (** Register producer and bounded terminal/cleanup capacity before start, even if start fails/raises. + deliver is thread-safe/non-suspending: select one terminal, hand off resources before ACK. + Error transfers nothing new; duplicate/late originals transfer only for cleanup, foreign remain producer-owned. + resources is pure/total and lists every owned reference. Callback-before-start-return is legal. + start/stop return promptly; await_stopped respects Execute's absolute monotonic cleanup deadline. + Before return/cancel propagation, physically stop callbacks/native work and resolve untransferred resources. + Failed/timed-out join retains producer/owner/charges, forces draining and reports all cleanup causes; + no implicit repair retry. Cancellation is not rollback; uncertain writes retain IDs/Outcome_unknown. *) + val run + : scope:effect_scope + -> resources:('a -> Dsl.resource_reference Rrbvec.t) + -> 'a producer + -> ('a, Dsl.error) result +end + +val check_current : effect_scope -> (unit, Dsl.error) result + +(** Check scope before irreversible work; admit_network registers one actual attempt after gate/epoch/cancel checks, never a reusable permit. Cleanup validates original ownership, not current authority. *) +val admit_network : effect_scope -> (unit, Dsl.error) result + +type database_owner + +(** Dispatch may suspend; return joins subfibers/producers except retained Cleanup_failure, which forces draining. + Mask only bounded handoff/commit bookkeeping; resource returns require handoff before cancellation checks. + External work uses Callback_bridge. Database mutations use the sole owner; Applied/Activated require commit. + Deferred is domain data; lost commit response preserves durable IDs/Outcome_unknown, never fresh replay. + Scoped crypto matches request IDs/counts/generation; stream snapshot verification within byte/item budgets, + charging staging/scratch through cleanup. Validate producer row count/checksum/encryption before activation. + Release snapshots on every path. Transport obeys Dsl matching/No_progress rules; policy stays in Dsl. + Reject foreign identities/resources; preserve typed causes without secrets or adapter retry loops. *) +type dispatch = + { database : + 'a. + owner:database_owner + -> scope:effect_scope + -> identity:Dsl.identity + -> 'a Dsl.database + -> ('a, Dsl.error) result + ; transport : + 'a. + scope:effect_scope + -> identity:Dsl.identity + -> 'a Dsl.transport + -> ('a, Dsl.error) result + ; crypto : + 'a. + scope:effect_scope + -> identity:Dsl.identity + -> 'a Dsl.crypto + -> ('a, Dsl.error) result + ; assets : + 'a. + scope:effect_scope + -> identity:Dsl.identity + -> 'a Dsl.assets + -> ('a, Dsl.error) result + } + +type websocket_liveness = + | Peer_close_only + | Ping_pong of + { interval_seconds : float + ; timeout_seconds : float + } + (** Finite positive monotonic timings; ping timeout is Network_lost. Peer_close_only has no silent-peer deadline. *) + +(** Finite positive monotonic deadlines: setup spans DNS/TCP addresses; TLS bounds each handshake. + HTTP spans write/headers/body/redirects/one refresh; download spans the entire transfer. Progress never resets. + Expiry cancels/joins; failed cleanup retains owners, unknown writes return Outcome_unknown. + Read/setup expiry is Network_failure; certificate/protocol errors never become retryable timeouts. *) +type network_deadlines = + { setup_seconds : float + ; tls_handshake_seconds : float + ; http_response_seconds : float + ; download_seconds : float + } + +(** Runtime-bound single attempt: register before callback, cancel on fence, join before return. No reusable permission or retry; dispatch cleanup rules apply. *) +type worker_network = + { with_admission : 'a. (unit -> ('a, Dsl.error) result) -> ('a, Dsl.error) result } + +(** One concrete owner shared by Worker and crypto/assets/database preparation; create binds after exclusive acquire. + Before binding/after fence, scopes reject; no rebinding while obligations live. Failed create retires binding + or retains failed cleanup under sw. Use supplied network admission for every remote attempt. + Scope reserves capacity and exact operation/intent ownership before callback/I/O; one live scope per operation. + Recheck generation/cancel after suspension and before commit; cancel/join tasks on every exit. + Imported files retain separate original-owner leases beyond scope exit; paths grant no ownership. + Release remains legal after fence/replacement, idempotent only after success; failed cleanup retains handles/causes. + Fence forbids new work but never reclaims consumer bytes. Quiescence joins work and waits for leases; + cleanup deadlines cannot force consumer release. Durable input/intents/staging survive restart; + prune only proven unreferenced terminal staging. *) +module type WORKER_OWNER = sig + type scope + type imported_file + + val bind_runtime + : identity:Dsl.identity + -> database_owner:database_owner + -> limits:Dsl.limits + -> cleanup_seconds:float + -> network:worker_network + -> (unit, Dsl.error) result + + val with_scope + : identity:Dsl.identity + -> operation:Logseq_db_types.Graph_types.Uuid.t + -> (scope -> ('a, Dsl.error) result) + -> ('a, Dsl.error) result + + val crypto : scope -> 'a Dsl.crypto -> ('a, Dsl.error) result + + (** Reserve original input before copying; journal Prepared proof before return. + Exact input recovers original intent; changed input/owner rejects. *) + val prepare_import + : scope + -> Logseq_db_types.Asset_import.t + -> current:(unit -> bool) + -> (Logseq_db_types.Asset_upload_intent.t, Dsl.error) result + + (** Preview is not remote publication. *) + val publish_import_preview + : scope + -> Logseq_db_types.Asset_upload_intent.t + -> (unit, Dsl.error) result + + (** Exact prior revision; None means absent. Uncertainty retains original proof/staging, never re-stages picker paths. *) + val persist_import + : scope + -> Logseq_db_types.Asset_upload_intent.t + -> expected:int option + -> (unit, Dsl.error) result + + val inspect_import + : scope + -> Logseq_db_types.Asset_upload_intent.t + -> ( [ `Import_absent + | `Local_present + | `Metadata_present + | `Publication_acknowledged + | `Entity_cancelled + ] + , Dsl.error ) + result + + val apply_local_import + : scope + -> Logseq_db_types.Asset_upload_intent.t + -> (unit, Dsl.error) result + + (** ACK means storage, not metadata/publication. *) + val put_import + : scope + -> Logseq_db_types.Asset_upload_intent.t + -> (unit, Dsl.error) result + + val apply_import_metadata + : scope + -> Logseq_db_types.Asset_upload_intent.t + -> (unit, Dsl.error) result + + (** Local/metadata writes use the sole queue/original mutation IDs; inspect/wait require original proof, not matching content. *) + val await_import_publication + : scope + -> Logseq_db_types.Asset_upload_intent.t + -> (unit, Dsl.error) result + + (** Cancellation preserves commits/unknown outcomes. *) + val cancel_import + : scope + -> Logseq_db_types.Asset_upload_intent.t + -> (unit, Dsl.error) result + + (** Release only with terminal proof and no users. Decoded intent/path/operation ID grants no authority. *) + val release_import_staging + : scope + -> Logseq_db_types.Asset_upload_intent.t + -> (unit, Dsl.error) result + + val recover_import_inputs + : scope + -> after:int64 option + -> maximum_items:int + -> maximum_bytes:int + -> (Logseq_db_types.Asset_import.Recovery.page, Dsl.error) result + + val confirm_import_input + : scope + -> string + -> (Logseq_db_types.Asset_import.Recovery.item option, Dsl.error) result + + (** Recovery is bounded/read-only; ACK matches original Prepared proof. Worker sequences durable phases. *) + val acknowledge_import_input + : scope + -> input:string + -> prepared:string + -> (unit, Dsl.error) result + + val cancel : scope -> (unit, Dsl.error) result + + val retain_imported_file + : scope + -> Logseq_db_types.Asset_upload_intent.t + -> (imported_file option, Dsl.error) result + + val imported_file_path : imported_file -> string + val release_imported_file : imported_file -> (unit, Dsl.error) result + val fence : identity:Dsl.identity -> (unit, Dsl.error) result + val await_quiescent : identity:Dsl.identity -> (unit, Dsl.error) result +end + +(** Validate identity/configuration locally before acquire: exclusive writer, finite positive timings, + exact owner budgets and bounded chunks/frames; mismatch is Invalid_input, never graph/file enumeration. + Limits remain immutable; Restore accounts disk recovery incrementally. Authenticate each remote attempt, + at most one checked refresh. fence_network is nonblocking; close_network physically joins all socket work. + Gate closes even on fence error; failed join prevents reopen. authorize_network checks actual epoch each time. + Validate original resource ledger even after revocation. Observe bounded atomic metadata, never graph scans. + Fence failure stays draining; retire ACKs delegation/resources, not database close/deletion. + No retirement with outstanding producers/leases/snapshots/cleanup; caller keeps original owners alive. + Release failure retains exact handle/cause. No global cache or new durable store. *) +type capabilities = + { validate_identity : Dsl.identity -> (unit, Dsl.error) result + ; validate_limits : Dsl.limits -> (unit, Dsl.error) result + ; websocket_liveness : websocket_liveness + ; network_deadlines : network_deadlines + ; cleanup_seconds : float + ; acquire : Dsl.identity -> (unit, Dsl.error) result + ; fence_network : Dsl.identity -> (unit, Dsl.error) result + ; close_network : Dsl.identity -> (unit, Dsl.error) result + ; authorize_network : Dsl.identity -> epoch:int64 -> (unit, Dsl.error) result + ; validate_resource : Dsl.resource_reference -> (unit, Dsl.error) result + ; restore : scope:effect_scope -> Dsl.identity -> (Dsl.restore_report, Dsl.error) result + ; worker_owner : (module WORKER_OWNER) + ; dispatch : dispatch + ; observe : Dsl.identity -> (Dsl.observation, Dsl.error) result + ; fence : Dsl.identity -> (unit, Dsl.error) result + ; retire : Dsl.identity -> (unit, Dsl.error) result + ; release_file : Dsl.identity -> Dsl.asset_file -> (unit, Dsl.error) result + ; release_lease : Dsl.identity -> Dsl.asset_lease -> (unit, Dsl.error) result + ; release_snapshot : Dsl.identity -> Dsl.snapshot -> (unit, Dsl.error) result + } + +val database_owner : Dsl.Overlay.Database.t -> database_owner + +type snapshot_source = + { path : string + ; expected_rows : int + } + +val activation_owner + : sw:Eio.Switch.t + -> snapshot_source:(Dsl.snapshot -> (snapshot_source, Dsl.error) result) + -> dependencies:Dsl.Overlay.Database.dependencies + -> inspection:Dsl.Overlay.Database.mirror_inspection + -> graph_name:string + -> (database_owner, Dsl.error) result + +(** The snapshot source callback validates its original acquisition ledger, including metadata. + Absent-mirror owner shares the sole queue; database fails Storage_failure until committed activation opens/installs it. Interrupted open recovers committed inspection locally; caller owns close. + All writes share the external transaction queue; revalidate generation/token after suspension. Never hold transactions across network/key/task/callback waits. + None means not yet activated; committed-open recovery failures remain errors. *) +val database_if_open : database_owner -> (Dsl.Overlay.Database.t option, Dsl.error) result +val database : database_owner -> (Dsl.Overlay.Database.t, Dsl.error) result + +(** Validate locally before exclusive acquire; failure leaves no registration except retained cleanup obligations. + Switches own structured runtime/control/task lifetimes; parent cancellation fences and joins owners. + Finish checked retirement before releasing database/scheduling owners. + Runtime-created contexts/handles/revisions are process-local; equal-identity client owners are foreign. + Reserve/register bounded ledgers before delivery; Producer checks do not replace current authority checks. + Runtime owns mechanics only; sync/retry policy remains pure Dsl. *) +val create + : sw:Eio.Switch.t + -> clock:'clock Eio.Time.Mono.t + -> database_owner:database_owner + -> identity:Dsl.identity + -> limits:Dsl.limits + -> capabilities + -> (t * Dsl.cold graph_context, Dsl.error) result + +val start + : t + -> 'before graph_context + -> ('before, 'after, 'a) Dsl.program + -> (('after, 'a) running, Dsl.error) result + +(** One admitted program per runtime; atomically validate phase/identity/version/capacity before reserving version. + Rejection has no effects. await is repeatable after joins or retained cleanup failure. + Only joined ordinary failures enter declared continuations; unjoined producers fence and return actual + draining context with Cleanup_failure/all causes. Durable uncertainty never permits fresh sends. *) +val await : ('after, 'a) running -> ('after, 'a) outcome + +type network_retirement + +val close_network : t -> (network_retirement, Dsl.error) result +val await_network_retirement : network_retirement -> (unit, Dsl.error) result + +(** Close fences/cancels immediately; receipt is not join. Repeat close returns the same outstanding receipt/error. + Await joins exact old generation; failure retains cause/obligations and Closed. Open checks actual epoch, + rejecting without waiting/replay until join succeeds. Controls work during binding/Restore, including cold. + Admission rejection has zero effects; admitted explicit attempts disable initial-install eligibility before + semantic Open checks, even on rejection. Restore never resets eligibility/gate. Indexed Closed also joins. *) +val open_network : t -> authorization_epoch:int64 -> (unit, Dsl.error) result + +(** Durable postcommit invalidation validates exact binding, coalescing in an independent dirty slot. Captured-epoch ACK preserves concurrent commits; notification failure never undoes/replays a user write. *) +val notify_outbox_changed + : t + -> owner:database_owner + -> identity:Dsl.identity + -> (unit, Dsl.error) result + +type ingress_ack = + | Queued + | Replaced + +(** Wrapped-only active Run_task ingress, including Locked/Closed; otherwise Stale_context. + valid_for is monotonic (0,300] seconds. Reserve before copy; one pending revision. + Replacement invalidates unconsumed receipt, not admitted work; rejection retains old input/caller ownership. + Expiry/stop/scope exit discards unconsumed payload. ACK means queued, not unlocked; never log secrets. *) +val submit_unlock + : t + -> identity:Dsl.identity + -> valid_for:float + -> Dsl.unlock_input + -> (ingress_ack, Dsl.error) result + +val update_demand : t -> Dsl.demand_change -> (ingress_ack, Dsl.error) result +val receive_demand : t -> (Dsl.demand_output, Dsl.error) result + +type delivery_status = + | Current + | Revoked + +val demand_status : t -> Dsl.demand_delivery -> (delivery_status, Dsl.error) result + +(** One receiver; dequeue transfers existing use obligation, not a new lease. Cancel before dequeue transfers nothing. + Caller retains/releases deliveries after revoke/stop/cancel; bytes require lease and remain until release ACK. + Stale queued output is cleaned; empty scope exit wakes Cancelled. Revocation never reclaims consumer bytes. + Release uses original owners, lease before file; ACK requires physical release. Failure retains handle/cause/charges; + foreign rejects without cleanup, duplicate succeeds without double release. Keep owners until cleanup resolves. *) +val release_demand : t -> Dsl.demand_delivery -> (unit, Dsl.error) result + +(** Latest same-ID update replaces atomically; other IDs never drop. Rejection preserves accepted work/caller ownership. + Charge active/pending interests until Finish; same-ID net descriptors, old cleanup remains charged. + Foreground reserves/demotion ceilings apply before admission; dedup does not exempt distinct IDs. + Live withdrawal uses control capacity; unknown withdrawal is idempotent. Dequeue reclaims no charges. + Stop/gate have independent slots; one input consumer. Stop is idempotent graceful input, not forced drain. + Run_task return joins scope and returns Ready; forced cancel fences and returns actual draining. *) +val request_stop : t -> (unit, Dsl.error) result + +(** Cancel exact execution and fence; foreign has no effects, accepted repeats are idempotent, terminal rejects. Commits/cleanup persist without await; outcome retains actual phase and causes, not retirement. *) +val cancel : t -> ('after, 'a) running -> (unit, Dsl.error) result + +(** Idle-only recovery returns the same phase/version; explicitly drain/join/retire. Cleanup failure retains draining/original obligations. *) +val current : t -> (context, Dsl.error) result + +type observation_revision + +type runtime_observation = + { revision : observation_revision + ; state : Dsl.observation + ; status : Dsl.service_status + } + +val observe_runtime : t -> (runtime_observation, Dsl.error) result + +(** Bounded coalesced latest snapshot; atomic compare/wait, one waiter, cancel consumes nothing. Foreign revisions reject; actual metadata overrides policy. Retired is readable; unchanged retired await fails Stale_context. *) +val await_observation + : t + -> after:observation_revision + -> (runtime_observation, Dsl.error) result + +val identity : 's graph_context -> Dsl.identity +val phase : 's graph_context -> 's Dsl.phase + +type version + +val version : 's graph_context -> version + +(** Metadata grants no fresh authority; versions/contexts/resources never survive restart or serialize. *) +val equal_version : version -> version -> bool diff --git a/logseq_sync/spec/execute/dune b/logseq_sync/spec/execute/dune new file mode 100644 index 00000000..caa63e7b --- /dev/null +++ b/logseq_sync/spec/execute/dune @@ -0,0 +1,7 @@ +(library + (name execute) + (public_name logseq_sync.execute) + (modules core) + (virtual_modules core) + (default_implementation logseq_sync_execute_impl) + (libraries eio sync_dsl logseq_db_types rrbvec)) diff --git a/logseq_sync/spec/protocol/dune b/logseq_sync/spec/protocol/dune new file mode 100644 index 00000000..e9adcef0 --- /dev/null +++ b/logseq_sync/spec/protocol/dune @@ -0,0 +1,7 @@ +(library + (name logseq_sync_protocol) + (public_name logseq_sync.protocol) + (modules sync_protocol) + (virtual_modules sync_protocol) + (default_implementation logseq_sync_protocol_impl) + (libraries logseq_db_types)) diff --git a/logseq_sync/spec/pure_reducer/sync_protocol.mli b/logseq_sync/spec/protocol/sync_protocol.mli similarity index 100% rename from logseq_sync/spec/pure_reducer/sync_protocol.mli rename to logseq_sync/spec/protocol/sync_protocol.mli diff --git a/logseq_sync/spec/pure_reducer/core.mli b/logseq_sync/spec/pure_reducer/core.mli deleted file mode 100644 index 19ffe466..00000000 --- a/logseq_sync/spec/pure_reducer/core.mli +++ /dev/null @@ -1,770 +0,0 @@ -(** {1 TLDR} - - Own pure account, graph and synchronization policy. The host owns {!t}; - {!initial} validates construction and {!step} returns a {!transition} whose - [next] must be installed before interpreting ordered [effects]. [Run] goes - to the Sync runner, [Delegate] to the Worker and [Publish] to observers. - Actual I/O, persistence and callback posting remain with those interpreters. - {!state} exposes startup, unlock, resident graph, network and failure facts. - - The runner GADT and dependent event families appear below. Primary excerpts: - {[ - val initial : config -> (t, create_error) result - val step : t -> event -> transition - val state : t -> state - val scope_for_graph : t -> graph_id:graph_id -> graph_scope option - ]} - Execution, presentation and lifecycle generations are separate. Cancellation - or [Shutdown] is not rollback or a physical join; exact cleanup completions - and [Websocket_retired] acknowledge their actual retained owners. *) - -(** {1 Minimal usage example} - - The caller supplies validated [config], input [event] and [interpret], which - dispatches Run/Delegate/Publish to their existing owners: - {[ - match initial config with - | Error error -> Error error - | Ok first -> - let reducer = ref first in - let transition = step !reducer event in - reducer := transition.next; - List.iter interpret transition.effects; - Ok reducer - ]} - Interpreters queue typed completions for later serialized turns rather than - recursively calling [step]. On retirement feed [Shutdown] and interpret its - effects, while keeping the real I/O and callback owners alive to finish cleanup. *) - -(** {1 Construction and validation} *) - -type limits -type config -type config_error = Invalid_config of string - -val limits - : maximum_response_bytes:int - -> maximum_artifact_bytes:int - -> submission_batch_size:int - -> (limits, config_error) result - -val config : managed_sync_origin:Uri.t -> limits:limits -> (config, config_error) result - -(** {1 Capability details} *) - -type graph_id = Logseq_db_types.Graph_types.Uuid.t -type graph = Logseq_db_types.Managed_graph.t -type account_generation = int -type graph_generation = int -type connection_generation = int -type presentation_generation = int -type lifecycle_generation = int64 - -type sync_phase = - | Offline - | Connecting - | Pulling - | Submitting - | Current - | Paused - | Failed - -type startup_failure_stage = - | During_authentication - | During_catalog - | During_local_restore - | During_bootstrap - | During_e2ee - -type local_unavailability = - | No_local_account - | Local_catalog_unavailable - | Local_mirror_unavailable - | Local_key_unavailable - | Local_restore_failed - -type local_access = - | Local_pending - | Local_available - | Local_unavailable of local_unavailability - -type startup_network_phase = - | Local_only - | Online_after_timeline - | Online_by_user - -type startup_facts = - { authenticated : bool - ; local_access : local_access - ; network_phase : startup_network_phase - ; catalog_loading : bool - ; awaiting_selection : bool - ; restoring_local : bool - ; bootstrapping : bool - ; awaiting_e2ee_password : bool - ; failure : startup_failure_stage option - ; account_generation : account_generation - ; graph_generation : graph_generation - ; presentation_generation : presentation_generation - } - -(** Cleanup of an authorized, non-current graph's local copy. The host must - obtain explicit confirmation that unsynchronized content will be discarded. - The selected graph and catalog entry are retained. Completion or failure - releases the deletion flight so selection and retry remain available. *) -type local_deletion_stage = - | Closing_graph - | Deleting_mirror - | Clearing_selection - -type local_deletion = - | Deletion_in_progress of local_deletion_stage - | Deletion_failed of local_deletion_stage - -(** Evidence is created by Core's package/crypto/attachment owners, never from - UI booleans. Phases are projections; Core rechecks identity and current scope. *) -type unlock_challenge -type unlock_submission -type unlock_attached - -type unlock_recovery = - | Retry_password of unlock_challenge * string - | Retry_preparation of string - -type unlock_phase = - | Unlock_inactive - | Unlock_challenge of unlock_challenge - | Unlock_running of unlock_submission - | Unlock_bootstrapping of unlock_submission - | Unlock_presentable of unlock_attached - | Unlock_failed of unlock_recovery - | Unlock_cancelled of unlock_submission - -val unlock_challenge : unlock_phase -> unlock_challenge option -val equal_unlock_challenge : unlock_challenge -> unlock_challenge -> bool -val unlock_submission_challenge : unlock_submission -> unlock_challenge -val equal_unlock_submission : unlock_submission -> unlock_submission -> bool -val unlock_attached_submission : unlock_attached -> unlock_submission - -(** Presentation only: retry eligibility is still determined by the original outcome. - Unknown and security failures require attention even when transport retries them. *) -type error_attention = Transient_connection | Requires_attention - -type snapshot = - { sync_phase : sync_phase - ; catalog : graph list - ; selected_graph : graph_id option - ; resident_graphs : graph_id list - ; applied_server_t : Logseq_db_types.Server_cursor.t option - ; timeline_presentation_pending : bool - ; unlock : unlock_phase - ; startup : startup_facts - ; last_error : string option - ; last_error_attention : error_attention option - ; local_deletion : local_deletion option - } - -type diagnostic_group = - { title : string - ; entries : (string * string) list - } - -type diagnostics = { groups : diagnostic_group list } - -type state = - { snapshot : snapshot - ; diagnostics : diagnostics - } - -type bootstrap_progress = - { graph_id : graph_id - ; received_bytes : int64 - ; total_bytes : int64 option - } - -type invalidation = - { basis : int64 - ; changed_uuids : graph_id list - ; changed_uuids_truncated : bool - } - -(** {1 Lifecycle and recovery} - - Core allocates execution generations and effect tickets for current owners; - runners hold the original tickets and return matching completions. Scope data - is not authority by itself. Account replacement, graph retirement and lifecycle - changes invalidate ordinary work; retained cleanup owners still require exact - acknowledgements. Process-local tickets are not restart recovery evidence. *) - -type account_scope = - { managed_sync_origin : Uri.t - ; user_id : string - ; account_generation : account_generation - ; presentation_generation : presentation_generation - ; lifecycle_generation : lifecycle_generation - } - -type graph_scope = - { account : account_scope - ; graph_id : graph_id - ; graph_generation : graph_generation - } - -(** Execution admission gate owned by a pending host handoff. It grants no - deletion authority. Release is exact-ID and cannot unfreeze a later round. *) -type host_fence_target=One_graph of graph_id|All_graphs -type host_fence={id:Logseq_db_types.Graph_types.Uuid.t;owner:account_scope; - target:host_fence_target;captured_scopes:graph_scope Rrbvec.t;previous:Logseq_db_types.Graph_types.Uuid.t option} - -type connection_scope = - { graph : graph_scope - ; connection_generation : connection_generation - } - -type effect_scope = - { account_generation : account_generation option - ; graph_generation : graph_generation option - ; connection_generation : connection_generation option - ; presentation_generation : presentation_generation option - ; lifecycle_generation : lifecycle_generation option - } - -type effect_id - -type crypto_failure_kind = - | Invalid_key_material - | Crypto_provider_unavailable - -type effect_error = - | Effect_failed of string - | Crypto_failed of crypto_failure_kind * string - -type graph_key_handle -type snapshot_download -type staged_artifact -type catalog_cache - -val graph_key_handle : id:string -> scope:graph_scope -> graph_key_handle -val graph_key_handle_id : graph_key_handle -> string -val graph_key_handle_scope : graph_key_handle -> graph_scope - -(** The download producer must carry the exact admitted request. This receipt - binds its artifact to the metadata attempt; Core also checks identity/current - scope on completion. Disk/stream validation remains the producer's job. *) -val stage_snapshot_download - : snapshot_download -> path:string -> expected_rows:int -> (staged_artifact, string) result - -val snapshot_download_scope : snapshot_download -> graph_scope -val snapshot_download_uri : snapshot_download -> Uri.t -val snapshot_download_expected_bytes : snapshot_download -> int64 option -val snapshot_download_maximum_bytes : snapshot_download -> int - -val staged_artifact_path : staged_artifact -> string -val staged_artifact_expected_rows : staged_artifact -> int -val unlock_challenge_scope : unlock_challenge -> graph_scope -val unlock_attached_scope : unlock_attached -> graph_scope - -val effect_scope_of_account : account_scope -> effect_scope -val effect_scope_of_graph : graph_scope -> effect_scope - -val catalog_cache - : user_id:string - -> graphs:graph list - -> selected_graph:graph_id option - -> catalog_cache - -val catalog_cache_user_id : catalog_cache -> string -val catalog_cache_graphs : catalog_cache -> graph list -val catalog_cache_selected_graph : catalog_cache -> graph_id option -val encode_catalog_cache : catalog_cache -> string -val decode_catalog_cache : string -> (catalog_cache, string) result - -(** {2 Protected values and assets} *) - -type encryption_batch = - { scope : graph_scope - ; key : graph_key_handle - ; plaintexts : string list - } - -type encrypted_values = (string * string) list - -type decryption_batch = - { scope : graph_scope - ; key : graph_key_handle - ; protected_values : (string * string) list - } - -type decrypted_values = string list -type snapshot_baseline = string -type snapshot_metadata = string - -type graph_key_request = - { scope : graph_scope - ; encrypted_graph_key : string - } - -type private_key_unlock = - { scope : account_scope - ; password : string - ; private_key_package : string - } - -type asset_context = - { scope : graph_scope - ; encrypted : bool - ; key : graph_key_handle option - } - -(** A transaction identity, never the target graph UUID. *) -type retirement_id = Logseq_db_types.Graph_types.Retirement_id.t -val retirement_id_of_string : string -> (retirement_id, string) result -val retirement_id_to_string : retirement_id -> string - -(** A scoped reference to one durable, exclusive retired asset namespace. - Construction grants no access; execution validates the producer journal. *) -type asset_deletion -val asset_deletion_reference : scope:graph_scope -> retirement:retirement_id -> asset_deletion -val asset_deletion_scope : asset_deletion -> graph_scope -val asset_deletion_retirement : asset_deletion -> retirement_id - -(** Business inputs carry identities and resource references, never asset bytes. *) -type asset_action = - | Check_asset_cache of graph_id * Logseq_db_types.Asset_descriptor.version - | Fetch_asset of - { asset : graph_id - ; version : Logseq_db_types.Asset_descriptor.version - ; maximum_plaintext_bytes : int - } - | Stage_asset_file of - { operation : graph_id - ; file_type : string - ; source_file : string - } - | Put_asset_file of - { asset : graph_id - ; version : Logseq_db_types.Asset_descriptor.version - ; file : string - ; maximum_plaintext_bytes : int - } - | Publish_staged_asset of - { asset : Logseq_db_types.Graph_types.Uuid.t - ; version : Logseq_db_types.Asset_descriptor.version - ; file : string - } - | Retain_asset_file of string - | Retain_staged_file of string - | Release_asset_file of string - | Release_staged_file of string - | Prune_asset_staging of string list - (** Join already admitted actual asset I/O tasks and their callback posting. - This read-only barrier neither closes caches nor grants a host permit. *) - | Await_asset_operations - (** Checked scope close preserves retained handles and failed physical cleanup - on error. The caller releases its original handles and retries; cancellation - or scope retirement does not stand in for a physical release ACK. *) - | Close_asset_scope - (** Cancels every target generation and waits for operations/native leases. - New target work remains blocked until its terminal action; lease release - remains available. Preparing retires files without unlinking their bytes. - The worker journals the mirror identity and frozen upload intents first. *) - | Prepare_graph_asset_deletion of retirement_id - (** Collects only this captured namespace. Both terminal actions are replayable; - committed work is never restored, and a replacement cache is never swept. *) - | Commit_graph_asset_deletion of asset_deletion - | Cancel_graph_asset_deletion of asset_deletion - | Delete_account_assets - | Asset_retry_after of float - | Cancel_asset_operation of string - -type asset_failure = - | Asset_network - | Asset_not_found - | Asset_checksum_mismatch - | Asset_authentication - | Asset_locked - | Asset_storage_full - | Asset_missing_source - | Asset_size_rejected - | Asset_revoked_access - | Asset_cancelled - | Asset_invalid_content of string - | Asset_with_cause of asset_failure * Logseq_db_types.Failure_diagnostic.t - -(** Preserve the retry category independently of bounded diagnostic context. *) -val asset_failure_kind : asset_failure -> asset_failure -val asset_failure_cause : asset_failure -> Logseq_db_types.Failure_diagnostic.t option -val asset_failure_with_cause : asset_failure -> Logseq_db_types.Failure_diagnostic.t -> asset_failure - -type asset_value = - | Asset_unit - | Asset_cached of string option - | Asset_downloaded of string - | Asset_staged of - { file : string - ; checksum : string - ; size : int64 - } - | Asset_retained of (string * string) option - | Asset_deletion_prepared of asset_deletion - | Asset_pruned of int - | Asset_retry_elapsed - -type asset_result = (asset_value, asset_failure) result - -type asset_request = - { scope : graph_scope - ; operation : string - ; action : asset_action - } - -type asset_io_request = - { context : asset_context - ; operation : string - ; action : asset_action - } - -type asset_ticket - -val asset_ticket_id : asset_ticket -> string - -type asset_output = - { request : asset_request - ; result : asset_result - } - -type protected_action = - | Encrypt_values of graph_key_handle * string list - | Decrypt_value of graph_key_handle * string - -type protected_request = - { protected_operation : string - ; protected_scope : graph_scope - ; protected_action : protected_action - } - -type protected_value = - | Encrypted_values of (string * string) list - | Decrypted_value of string - -type protected_result = (protected_value, effect_error) result - -type protected_output = - { protected_request : protected_request - ; protected_result : protected_result - } - -type protected_ticket - -val protected_ticket_id : protected_ticket -> string - -(** {2 Runner requests and completions} *) - -type _ runner_request = - | Load_catalog : account_scope -> catalog_cache option runner_request - | Save_catalog : - { account : account_scope - ; cache : catalog_cache - } - -> unit runner_request - | Fetch_catalog : account_scope -> graph list runner_request - | Fetch_snapshot_baseline : graph_scope -> snapshot_baseline runner_request - | Fetch_snapshot_metadata : graph_scope -> snapshot_metadata runner_request - | Download_snapshot : snapshot_download -> staged_artifact runner_request - | Fetch_e2ee_graph_key : graph_scope -> string runner_request - | Fetch_e2ee_user_keys : account_scope -> string runner_request - | Load_and_unlock_graph_key : graph_scope -> graph_key_handle runner_request - | Fetch_and_unlock_graph_key : graph_key_request -> graph_key_handle runner_request - | Unlock_private_key : private_key_unlock -> unit runner_request - | Delete_account_secrets : account_scope -> unit runner_request - | Encrypt_protected_values : encryption_batch -> encrypted_values runner_request - | Decrypt_protected_values : decryption_batch -> decrypted_values runner_request - -type 'a effect_ticket - -val effect_ticket_id : 'a effect_ticket -> effect_id -val effect_ticket_scope : 'a effect_ticket -> effect_scope -val effect_id_to_string : effect_id -> string - -type websocket_request = - { scope : connection_scope - ; uri : Uri.t - } - -type websocket_send = - { scope : connection_scope - ; message : Sync_protocol.Client.message - } - -(** Only recoverable peer closure or transport failure is eligible for retry. - Local closure, authentication rejection and protocol failure do not retry. *) -type websocket_termination = - | Recoverable_close of { message : string option; attention : error_attention } - | Local_close - | Authentication_failure of string - | Protocol_failure of string - -type timer_id - -type timer_request = - { id : timer_id - ; scope : effect_scope - ; delay_seconds : float - } - -type runner_effect = private - (* Snapshot actual I/O owners before fork; wait off the serial consumer and - report after their physical task and callback posting finish. No host, - filesystem or data-admission permission is granted. *) - | Await_io of int64 * effect_scope - | Asset_io of asset_ticket * asset_io_request - | Protected_io of protected_ticket * protected_request - | Request : 'a effect_ticket * 'a runner_request -> runner_effect - | Start_websocket of websocket_request - | Send_websocket of websocket_send - (** Cancel setup or close the exact owner, join its I/O and callback fibers, - then emit [Websocket_retired]. Duplicate requests are idempotent. *) - | Close_websocket of connection_scope - | Schedule_timer of timer_request - | Cancel_effects of effect_scope - | Cancel_network_effects of effect_scope - -type runner_completion = - | Asset_completion of asset_ticket * asset_result - | Protected_completion of protected_ticket * protected_result - | Completion : 'a effect_ticket * ('a, effect_error) result -> runner_completion - -val runner_effect_scope : runner_effect -> effect_scope -val equal_runner_effect : runner_effect -> runner_effect -> bool -val runner_effect_diagnostic : runner_effect -> string - -(** {2 Worker delegation} *) - -type mirror_request = - { graph : graph - ; scope : graph_scope - } - -(** Constructed only by Core after matching the active attempt and its artifact. - Consumers cannot replace its graph scope, metadata cursor or key independently. *) -type snapshot_activation_request -val snapshot_activation_scope : snapshot_activation_request -> graph_scope -val snapshot_activation_cursor : snapshot_activation_request -> Logseq_db_types.Server_cursor.t -val snapshot_activation_artifact : snapshot_activation_request -> staged_artifact -val snapshot_activation_key : snapshot_activation_request -> graph_key_handle option - -type mirror_deletion = - { account : account_scope - ; graph_id : graph_id - ; scope : effect_scope - } - -type mirror_inspection = - | Mirror_available of mirror_request - | Mirror_absent of graph_scope - -type outbox_transition_request = - { expected : Logseq_overlay_db.Types.sync_token - ; key : graph_key_handle option - ; scope : graph_scope - ; transition : Logseq_overlay_db.Types.outbox_transition - } - -(** Failure of the exact delegated outbox request. The original scope, transition, - and expected Sync token fence failures from superseded requests. *) -type outbox_transition_failure = - { request : outbox_transition_request - ; message : string - } - -type authoritative_batch = - { input : Logseq_overlay_db.Types.authoritative_batch - ; key : graph_key_handle option - ; scope : connection_scope - ; presentation_generation : presentation_generation - ; lifecycle_generation : lifecycle_generation - } - -type worker_effect = - | Inspect_mirror of mirror_request - | Activate_snapshot of snapshot_activation_request - | Delete_mirror of mirror_deletion - | Attach_graph of mirror_request - | Detach_graph of graph_scope - | Reset_managed_account of account_scope - | Inspect_sync of graph_scope - | Apply_outbox_transition of outbox_transition_request - | Apply_authoritative_batch of authoritative_batch - -type output = - | Asset_finished of asset_output - | Protected_finished of protected_output - | State_changed of state - | Resident_scopes_changed of graph_scope list - | Session_state_changed of graph_scope * state - | Bootstrap_progressed of bootstrap_progress - -type instruction = - | Run of runner_effect - | Delegate of worker_effect - | Publish of output - -val equal_instruction : instruction -> instruction -> bool -val equal_instructions : instruction list -> instruction list -> bool - -type scoped_error = - { scope : effect_scope - ; message : string - } - -type graph_attachment = - { scope : graph_scope - ; sync : Logseq_overlay_db.Types.sync_view - } - -type sync_inspection = - { scope : graph_scope - ; sync : Logseq_overlay_db.Types.sync_view - } - -type authoritative_commit_result = - { scope : graph_scope - ; commit : Logseq_overlay_db.Types.authoritative_commit - ; sync : Logseq_overlay_db.Types.sync_view - } - -type authoritative_deferred_result = - { scope : graph_scope - ; defer : Logseq_overlay_db.Types.authoritative_defer - } - -type outbox_transition_result = - { scope : graph_scope - ; commit : Logseq_overlay_db.Types.outbox_commit - ; sync : Logseq_overlay_db.Types.sync_view - } - -(** Completion/failure producers must return the exact delegated activation. - Same-scope retries still have different attempt identities. *) -type snapshot_activation -type snapshot_activation_error -val snapshot_activation_completed : snapshot_activation_request -> snapshot_activation -val snapshot_activation_failed : snapshot_activation_request -> message:string -> snapshot_activation_error - -type startup_scope = - { account_generation : account_generation - ; presentation_generation : presentation_generation - } - -type timeline_scope = - { startup : startup_scope - ; graph_id : graph_id - ; graph_generation : graph_generation - } - -(** {2 Serialized events and transitions} *) - -type event = - | Host_fence_started of host_fence - | Host_fence_released of Logseq_db_types.Graph_types.Uuid.t - | IO_fence_requested of int64 * effect_scope - | IO_fence_completed of int64 * effect_scope * (unit,string) result - | Asset_requested of asset_request - | Protected_requested of protected_request - | Restore_local_account of { user_id : string } - | Account_authenticated of { user_id : string option } - | Local_feed_acknowledged - | Local_outbox_changed - | Resident_graphs_changed of graph_id list - | Visible_graph_changed of graph_id - | Local_graph_outbox_changed of graph_scope - | Timeline_presented of timeline_scope - | Graph_selected of graph_id - | Graph_picker_requested - | Sign_in_requested of startup_scope - | Catalog_refresh_requested of startup_scope - | Online_recovery_requested of timeline_scope - | Online_cancelled of startup_scope - | Local_account_unavailable of startup_scope * local_unavailability - | E2ee_password_submitted of unlock_challenge * string - | Local_cache_deletion_requested of graph_id - | Foreground_changed of - { foreground : bool - ; lifecycle_generation : lifecycle_generation - } - (** Acknowledges the captured graph close after outstanding database operations, - subscriptions, storage, and ownership have been released. *) - | Graph_detached of graph_scope * (unit, string) result - (** Acknowledges the exact deletion request; stale or repeated results are inert. *) - | Mirror_deleted of mirror_deletion * (unit, string) result - | Mirror_inspected of mirror_inspection - | Graph_attached of graph_attachment - | Graph_attachment_failed of scoped_error - | Sync_inspected of sync_inspection - | Outbox_transition_applied of outbox_transition_result - (** An uncommitted preparation lost its token to a newer durable revision. - The exact pending request and a changed token fence recovery; this is not - evidence of a commit or permission to repeat an unchanged failed request. *) - | Outbox_transition_superseded of - { request : outbox_transition_request; sync : Logseq_overlay_db.Types.sync_view } - | Outbox_transition_failed of outbox_transition_failure - | Authoritative_batch_applied of authoritative_commit_result - | Authoritative_batch_deferred of authoritative_deferred_result - | Authoritative_batch_failed of scoped_error - | Snapshot_activated of snapshot_activation - | Snapshot_activation_failed of snapshot_activation_error - | Runner_completed of runner_completion - | Snapshot_download_progress of bootstrap_progress - | Websocket_opened of connection_scope - | Websocket_message of connection_scope * Sync_protocol.Server.message - | Websocket_protocol_error of connection_scope * Sync_protocol.codec_error - (** Termination classification; replacement waits for [Websocket_retired]. *) - | Websocket_closed of connection_scope * websocket_termination - (** The exact owner's resources are closed and its I/O and callback fibers have - completed. No further callbacks may be emitted. Match the retained owner - even after a graph/lifecycle change; unrelated or duplicate events are inert. *) - | Websocket_retired of connection_scope - | Timer_elapsed of timer_id - | Shutdown - -type t -type create_error = Invalid_create of string - -val initial : config -> (t, create_error) result -val state : t -> state -val admitted_graph_scope : t -> graph_scope option -val asset_context : t -> asset_context option - -type transition = - { next : t - ; effects : instruction list - } - -val step : t -> event -> transition - -(** {1 Observation and helpers} *) - -(** Ordered, currently resident graphs; at most three unique authorized entries. *) -val resident_graphs : t -> graph_id list -val scope_for_graph : t -> graph_id:graph_id -> graph_scope option -val catalog_cache_resident_graphs : catalog_cache -> graph_id list - -val state_for_scope : t -> graph_scope -> state option -val asset_context_for_scope : t -> graph_scope -> asset_context option -val event_graph_scope : t -> event -> graph_scope option - -type coordination_state = - { account : account_scope option - ; desired_residents : graph_id list - ; residents : (graph_scope * state) list - ; selected : timeline_scope option - ; deletion : (graph_scope * local_deletion) option - } - -(** One immutable owner read, including non-admitted resident lifetimes and the - exact local deletion flight. Execution identity excludes focus generations; - selected carries the current presentation separately. No IO or registration. *) -val coordination_state : t -> coordination_state - -val host_fence_blocks : t -> graph_id -> bool -val host_fence_active : t -> host_fence option diff --git a/logseq_sync/spec/pure_reducer/dune b/logseq_sync/spec/pure_reducer/dune deleted file mode 100644 index 4c2c622a..00000000 --- a/logseq_sync/spec/pure_reducer/dune +++ /dev/null @@ -1,7 +0,0 @@ -(library - (name logseq_sync_pure_reducer) - (public_name logseq_sync.pure_reducer) - (modules core sync_protocol asset_transfer) - (virtual_modules core sync_protocol asset_transfer) - (default_implementation logseq_sync_pure_reducer_impl) - (libraries logseq_db_types logseq_overlay_db rrbvec uri)) diff --git a/logseq_sync/spec/sync_dsl/core.mli b/logseq_sync/spec/sync_dsl/core.mli new file mode 100644 index 00000000..b39a5838 --- /dev/null +++ b/logseq_sync/spec/sync_dsl/core.mli @@ -0,0 +1,746 @@ +(** Indices express lifecycle, not runtime authority, linearity or rollback. *) + +type cold +type ready +type draining +type retired + +type _ phase = + | Cold : cold phase + | Ready : ready phase + | Draining : draining phase + | Retired : retired phase + +type _ drainable = + | Cold_context : cold drainable + | Ready_context : ready drainable + | Draining_context : draining drainable + +type identity = private + { origin : string + ; account : string + ; graph : string + ; account_generation : int64 + ; graph_generation : int64 + } + +module Overlay = Logseq_overlay_db + +type error = + | Invalid_input of string + | Identity_mismatch of string + | Stale_context + | Network_closed + | Network_failure of string + | Authentication_failure of string + | Locked + | Crypto_failure of string + | Protocol_failure of string + | Storage_failure of string + | Cancelled + | Outcome_unknown of string + | Cleanup_failure of + { primary : error option + ; failures : error Rrbvec.t + } + | Capacity_exceeded of string + | Sync_read_failure of Overlay.Types.sync_read_error + | Outbox_failure of Overlay.Types.outbox_transition_error + | Authoritative_failure of Overlay.Types.authoritative_transition_error + | Activation_failure of Overlay.Types.snapshot_activation_error + | Asset_failure of Logseq_db_types.Failure_diagnostic.t + | Unexpected_failure of string + +type ('before, 'after, 'value) program + +val return : 'a -> ('s, 's, 'a) program +val abort : error -> ('s, 'final, 'a) program + +val bind + : ('before, 'middle, 'a) program + -> ('a -> ('middle, 'after, 'b) program) + -> ('before, 'after, 'b) program + +(** Canonical HTTPS origin, nonempty account/graph, nonnegative generations; effects recheck actual authority. *) +val identity + : origin:string + -> account:string + -> graph:string + -> account_generation:int64 + -> graph_generation:int64 + -> (identity, error) result + +type limits = private + { maximum_items : int + ; maximum_bytes : int + ; maximum_pending : int + ; maximum_resources : int + ; steps_per_turn : int + ; maximum_tasks : int + ; maximum_messages : int + ; maximum_message_bytes : int + ; maximum_demands : int + ; maximum_active_assets : int + ; foreground_reserve : int + ; foreground_demand_reserve : int + ; foreground_descriptor_reserve_bytes : int + ; asset_cache_budget_bytes : int64 + ; asset_maximum_file_bytes : int + ; asset_pending_budget_bytes : int64 + ; snapshot_maximum_artifact_bytes : int64 + ; snapshot_staging_budget_bytes : int64 + ; snapshot_scratch_budget_bytes : int64 + } + +(** Positive bounds: items/pending/messages/demands <= 4096, bytes/message_bytes <= 16777216, + resources <= 1024, steps <= 65536, tasks <= 256; active_assets <= pending. + Tasks/pending >= active_assets + 4; reserve three pending control slots plus foreground work. + Foreground reserves: work [1..active_assets], interests [1..demands], descriptors [1..message_bytes]; + messages >= foreground_demand_reserve + 3. Background cannot borrow reserves. + Charge interests per ID, shared transfers once; replacement uses atomic net descriptors. + Dequeue reclaims nothing; cancelled/failed cleanup remains charged through physical join. + Reserve before acquisition; control/release need no new slot. Exhaustion never evicts accepted work. + Snapshot artifact/staging/scratch budgets are independent positive int64s, without a 16 MiB artifact cap; + staging below artifact is legal. Charge encrypted/plaintext coexistence until cleanup; + stream verification/activation within maximum_bytes/items. + Asset cache/pending budgets are nonnegative; file bytes [0..104857600]; zero disables that class. + Count pinned/recovered files and actual sizes before admission; over-budget permits only recovery/prune/release. + Never prune pinned bytes or durable unpublished originals. Validation reads configuration only. *) +val limits + : maximum_items:int + -> maximum_bytes:int + -> maximum_pending:int + -> maximum_resources:int + -> steps_per_turn:int + -> maximum_tasks:int + -> maximum_messages:int + -> maximum_message_bytes:int + -> maximum_demands:int + -> maximum_active_assets:int + -> foreground_reserve:int + -> foreground_demand_reserve:int + -> foreground_descriptor_reserve_bytes:int + -> asset_cache_budget_bytes:int64 + -> asset_maximum_file_bytes:int + -> asset_pending_budget_bytes:int64 + -> snapshot_maximum_artifact_bytes:int64 + -> snapshot_staging_budget_bytes:int64 + -> snapshot_scratch_budget_bytes:int64 + -> (limits, error) result + +type retry_policy = private + { initial_seconds : float + ; maximum_seconds : float + ; multiplier : float + ; maximum_attempts : int + } + +(** Finite positive delays, maximum >= initial, multiplier >= 1, attempts [0..10]. + maximum_attempts counts actual retry admissions after the initial attempt; zero disables retry. + Only network failures retry, using capped exponential monotonic Sleep. *) +val retry_policy + : initial_seconds:float + -> maximum_seconds:float + -> multiplier:float + -> maximum_attempts:int + -> (retry_policy, error) result + +type network_gate = + | Closed + | Open of { authorization_epoch : int64 } + +type key_status = + | Unencrypted + | Key_locked + | Key_available + +type observation = + { identity : identity + ; keys : key_status + ; network : network_gate + ; checkpoint : Overlay.Types.server_cursor option + ; pending_submissions : int + ; pending_assets : int + ; cleanup_pending : int + ; last_failure : error option + } + +type restore_report = + { keys : key_status + ; checkpoint : Overlay.Types.server_cursor option + ; recovery_pending : bool + } + +type activation = + | Activated of Overlay.Types.generation + | Activation_deferred of string + +type authoritative_result = + | Applied of Overlay.Types.authoritative_commit + | Deferred of Overlay.Types.authoritative_defer + +type resource_release + +type resource = private + { owner : identity + ; token : string + ; release : resource_release + } + +(** Shape-only process-local descriptor; use/release checks the registered original-owner ledger, even after revocation. *) +val resource : owner:identity -> token:string -> (resource, error) result + +type snapshot = private + { resource : resource + ; through : Overlay.Types.server_cursor + ; checksum : string + ; encrypted : bool + } + +(** Staged bytes remain owed until checked release; activation verifies producer metadata, baseline, checksum and encryption. *) +val snapshot + : resource:resource + -> through:Overlay.Types.server_cursor + -> checksum:string + -> encrypted:bool + -> (snapshot, error) result + +type _ database = + | Inspect : Overlay.Types.sync_view database + | Verify_checksum : + Overlay.Types.server_cursor * Overlay.Types.checksum + -> unit database + | Prepare_submission : + Overlay.Types.sync_token * Overlay.Types.outbox_transition + -> Overlay.Types.outbox_commit database + | Apply_authoritative : + Overlay.Types.sync_token * Overlay.Types.authoritative_batch + -> authoritative_result database + | Activate_snapshot : snapshot -> activation database + | Release_snapshot : snapshot -> unit database + +(** Inspect rejects oversized views, never truncates. Prepare durably freezes wires before send; + conflicts need fresh tokens. Deferred retains the original batch and exact submission outcome. + Activation checks generation/baseline/local intent atomically; cancellation preserves commits. *) + +type remote_head = + { through : Overlay.Types.server_cursor + ; snapshot_available : bool + ; expected_checksum : Overlay.Types.checksum option + } + +type pull_result = + | Pulled of Overlay.Types.authoritative_batch + | No_progress of + { since : Overlay.Types.server_cursor + ; through : Overlay.Types.server_cursor + ; expected_checksum : Overlay.Types.checksum option + } + (** Validate generation, issued baseline and all cursors before filtering; Pulled advances baseline. + No_progress matches since with through >= since; neither result advances checkpoint without apply. + Retain greatest target, recover Submitted, then settle or delay before re-pull; never ACK a barrier. *) + +type submission_reply = + | Accepted of Overlay.Types.acceptance_barrier + | Rejected of Overlay.Types.rejection_resolution + +(** Fresh registered connection generation; every use checks its original producer. Retirement invalidates the connection, not the graph. *) +type connection + +val connection + : owner:identity + -> token:string + -> generation:int64 + -> (connection, error) result + +type connection_end = + | Remote_closed + | Network_lost of string + | Access_revoked of string + | Invalid_protocol of string + | Locally_closed + +type remote_message = + | Remote_error of string + | Remote_changed of remote_head + | Connection_ended of connection_end + +type received = + { connection : connection + ; message : remote_message + } + +type _ transport = + | Connect : connection transport + | Receive : connection -> received transport + | Retire_connection : connection -> unit transport + | On_connection : connection * 'a transport -> 'a transport + | Head : remote_head transport + | Fetch_snapshot : Overlay.Types.server_cursor -> snapshot transport + | Pull : Overlay.Types.server_cursor -> pull_result transport + | Push : Overlay.Types.submission_batch -> submission_reply transport + +(** Validate framing/authentication/cursors/bounds before delivery. Wrap only Head/Fetch_snapshot/Pull/Push + once in On_connection; one unresolved response-family request and one Receive per connection. + Match generation/baseline/frozen IDs/barriers; ambiguity fails the connection, retaining Submitted. + Empty/replayed Pull returns No_progress. Only Remote_changed hints coalesce; overflow backpressures. + Retire physically joins readers/writers/requests/callbacks before reconnect. Only remote network loss retries. + Authentication allows one checked refresh, then requires corrected authority. + Retry_group uses original batch ID, t_before and exact durable wires with a fresh Inspect token; + unknown sends never become fresh submissions. Accept/Reject then Pull reconcile uncertain execution. + Rejection validates exact ordered prefix/failed member/suffix and prefix barrier; invalid partitions retain wires. + Omitted failed ID requires empty prefix; Stale needs advancing cursor, never proves non-execution. + Preserve typed rejection causes/bounded diagnostics; Remote_error is non-retryable and retains uncertainty. *) + +type unlock_input + +val unlock_input : wrapped_graph_key:string -> (unlock_input, error) result + +(** Wrapped material only; never log keys. Still-locked Unlock returns Locked; caller owns key acquisition. *) +val wrapped_graph_key : unlock_input -> string + +type _ crypto = + | Unlock : unlock_input -> key_status crypto + | Encrypt_values : string Rrbvec.t -> (string * string) Rrbvec.t crypto + | Decrypt_value : string -> string crypto + | Protect : + Overlay.Database.protection_request + -> (Overlay.Types.crypto_item_id * string) list crypto + | Unprotect : + Overlay.Database.unprotection_request + -> (Overlay.Types.crypto_item_id * string) list crypto + (** Graph keys stay with their owner. Encrypt_values preserves order; Decrypt_value validates Transit. + Protect/Unprotect preserve exact IDs/cardinality; bound all bytes/items, never plaintext fallback. *) + +type asset = Logseq_db_types.Asset_descriptor.t + +type asset_file = private + { file : resource + ; asset : asset + } + +type asset_lease = private + { lease : resource + ; source : asset_file + } + +val asset_file : resource -> asset -> (asset_file, error) result +val asset_lease : resource -> asset_file -> (asset_lease, error) result + +type resource_reference = + | Snapshot_reference of snapshot + | Connection_reference of connection + | File_reference of asset_file + | Lease_reference of asset_lease + +type asset_result = + | Available of asset_file + | Waiting_remote + | Missing_local + +(** retained_bytes is included in resident/pending, not additive. *) +type cache_report = + { resident_bytes : int64 + ; pending_bytes : int64 + ; retained_bytes : int64 + ; cleanup_pending : int + ; over_budget : bool + } + +type _ assets = + | Prune_cache : cache_report assets + (** Original owner proves retention from durable inputs/checkpoints, serialized with preparation. *) + | Reconcile_staging : int assets + | Lookup : asset -> asset_file option assets + | Download : asset -> asset_result assets + | Upload : asset -> unit assets + | Retain : asset_file -> asset_lease assets + | Release_file : asset_file -> unit assets + | Release_lease : asset_lease -> unit assets + +(** Registered files/leases are process-local and original-owner checked. Upload ACK means verified publication. + Lookup is offline: Managed remote B/current A tries original A then cached B; + Managed None/current A checks A, then waits (never GET A); External never transfers. + Download requires explicit remote version and verifies encryption/checksum, without plaintext fallback. + Results preserve selection; Retain/Publish check actual byte metadata. Sharing includes incarnation, + UUID/checksums/file_type; transfer dedup omits demand ID, but each interest needs its own lease. + Replacement invalidates old publication rights. Locked/missing versions wait for correction. + Prune only unpinned cache/proven orphan staging; unpublished originals survive. + Release is idempotent only after success; failed cleanup retains handles/paths/charges/causes. *) + +type 'a task +type scope +type !'a child +type demand_id + +val demand_id : string -> (demand_id, error) result +val compare_demand_id : demand_id -> demand_id -> int + +type priority = + | Foreground + | Background + +type demand_change = + | Demand of + { id : demand_id + ; asset : asset + ; priority : priority + } + | Withdraw of demand_id + +(** Exact ingress revision; replacement/scope exit invalidates it. *) +type demand_receipt + +(** Single-use wrapped revision; replacement/expiry/scope exit invalidates it. Never persists or prints keys. *) +type unlock_receipt + +type service_input = + | Unlock_requested of unlock_receipt + | Demand_changed of demand_receipt * demand_change + | Network_changed of network_gate + | Stop_requested + (** IDs are nonempty, <= 256 bytes; withdrawal cancels only its interest, not shared users. *) + +type asset_failure_reason = + | Asset_network of string + | Asset_network_closed + | Asset_not_found + | Asset_checksum_mismatch + | Asset_authentication of string + | Asset_locked + | Asset_storage_full + | Asset_missing_source + | Asset_size_rejected + | Asset_revoked_access + | Asset_cancelled + | Asset_invalid_content of string + | Asset_other of error + +(** Joined result; attempts counts actual scoped remote admissions, not tasks. No implicit retry; unjoined cleanup remains an effect error. *) +type demand_transfer = + { result : (asset_result, asset_failure_reason) result + ; attempts : int + } + +type demand_availability = + | Demand_queued + | Demand_downloading + | Demand_waiting_network + | Demand_waiting_unlock + | Demand_failed of + { reason : asset_failure_reason + ; attempts : int + ; retry_scheduled : bool + } + | Demand_available of asset_file * asset_lease + | Demand_waiting_remote + | Demand_missing_local + | Demand_revoked + +(** Received delivery retains original cleanup obligation through revocation until release ACK. *) +type demand_delivery + +type demand_output = private + { delivery : demand_delivery + ; id : demand_id + ; demand : demand_receipt + ; availability : demand_availability + } + +(** Every state requires delivery release; retry_scheduled requires a bounded network timer. Locked/Closed wait, then fresh Lookup. *) + +type outbox_check + +(** Runtime-bound capacity revision. *) +type capacity_revision + +type 'a wake = + | Input of service_input + | Child_ready of 'a child + +type notifications + +type notification = + | Policy_failure of error + | Connected of connection + | Remote_received of received + | Connection_retired of connection + +type service_status = + | Service_idle + | Service_connecting + | Service_pulling + | Service_submitting + | Service_online + | Service_waiting_network + | Service_waiting_unlock + | Service_backoff of + { attempt : int + ; seconds : float + ; cause : error + } + | Service_blocked of error + +type _ task_operation = + | Notifications : scope -> notifications task_operation + | Notify : notifications * notification -> unit task_operation + | Next_notification : notifications -> notification task_operation + | Await_outbox_changed : outbox_check task_operation + | Acknowledge_outbox_check : outbox_check -> unit task_operation + | Abandon_outbox_check : outbox_check -> unit task_operation + | Resample_outbox : unit task_operation + | Capacity_revision : capacity_revision task_operation + | Await_capacity_changed : capacity_revision -> unit task_operation + | Task_observe : observation task_operation + | Publish_status : service_status -> unit task_operation + | Task_database : 'a database -> 'a task_operation + | Task_transport : 'a transport -> 'a task_operation + | Task_crypto : 'a crypto -> 'a task_operation + | Task_assets : 'a assets -> 'a task_operation + | Consume_unlock : unlock_receipt -> key_status task_operation + | Publish_demand : demand_receipt * demand_availability -> unit task_operation + | Download_demand : demand_receipt -> demand_transfer task_operation + | Finish_demand : demand_receipt -> unit task_operation + | Wait_network_open : unit task_operation + | Next_input : service_input task_operation + | Await_input_or_child : 'a child Rrbvec.t -> 'a wake task_operation + | Sleep : float -> unit task_operation + | Yield : unit task_operation + (** Capture capacity revision before admission; atomically wait for change or join owned children, then recheck. + Waiters cancel with scope; wakeup grants no capacity and cannot discard accepted work. + Outbox dirtiness survives without waiters. One await-to-ACK owner captures epoch before Inspect; + ACK only after bounded plan installation; concurrent commits stay dirty. Abandon/cancel releases reservation only. + Start/restore/unlock/activation/reopen mark dirty; Connected resamples. Re-inspect after waiting for Open. + Sleep is finite nonnegative, monotonic/cancellable; effects may suspend without blocking control. + Wait_network_open also waits for old join but grants no permit; admission rechecks authority. + Consume_unlock consumes live receipt before crypto; stale/expired performs no crypto. Failure needs fresh input. + Publish atomically validates revision/selection/resources/capacity: success transfers file+lease, + failure transfers nothing. Never release/republish transferred resources. Available needs live Demand; + Revoked needs live Withdraw; waiting/missing carry none. Reject duplicate references. + Replace unreceived output only after cleanup; received bytes remain until consumer release ACK. + Scope exit retains received/failed cleanup, blocking retirement; Finish cannot erase it or newer revisions. + Download_demand performs one Download after Lookup; Closed/Locked/no-version admits no remote attempt. + Active work stays charged through join; replacement reclassifies shared priority before admission. + Notifications share message budgets; Notify backpressures, coalescing only adjacent same-connection hints. + One consumer per channel; exit joins children before queue release. + Await_input_or_child requires unique live same-scope children; shares Next_input's sole consumer. + Atomically consume only winning input; readiness is not join, cancellation consumes nothing. + Stop wins, otherwise choice is fair. *) + +val task_return : 'a -> 'a task +val task_bind : 'a task -> ('a -> 'b task) -> 'b task +val task_fail : error -> 'a task + +val task_perform + : 'a task_operation + -> success:('a -> 'b task) + -> failure:(error -> 'b task) + -> 'b task + +val with_scope : (scope -> 'a task) -> 'a task +val spawn : scope -> 'a task -> ('a child, error) result task + +(** spawn_asset binds revision/priority; replacement reclassifies before admission. Slots remain charged through join; background eventually runs. *) +val spawn_asset : scope -> demand_receipt -> 'a task -> ('a child, error) result task + +val equal_child : 'a child -> 'a child -> bool +val join : 'a child -> ('a, error) result task +val cancel_child : 'a child -> unit task + +(** Scope exit cancels/joins all children/producers. One join consumes a completed child's slot; + foreign/expired/reused joins fail. Report unobserved failures, retaining cleanup/unknown outcomes. + Child cancellation affects only its subtree; Network_closed is not graph cancellation. *) + +type _ task_node = + | Task_return : 'a -> 'a task_node + | Task_fail : error -> 'a task_node + | Task_perform : + 'a task_operation * ('a -> 'b task) * (error -> 'b task) + -> 'b task_node + | Task_scope : (scope -> 'a task) -> 'a task_node + | Task_spawn : scope * 'a task -> ('a child, error) result task_node + | Task_spawn_asset : + scope * demand_receipt * 'a task + -> ('a child, error) result task_node + | Task_join : 'a child -> ('a, error) result task_node + | Task_cancel : 'a child -> unit task_node + | Task_bind : 'a task * ('a -> 'b task) -> 'b task_node + +(** Pure total projection; continuations terminate without I/O. Yield after steps_per_turn, fairly including input/completions. *) +val task_view : 'a task -> 'a task_node + +(** Interpreter-only process-local handles, not authority or resource ACKs; equal identity is not equal ownership. + Execute keeps its fresh owner private, reserves/registers before delivery and validates current ledgers. + Foreign/closed checks fail before callbacks; original cleanup remains legal. *) +module Producer : sig + type owner + + val create : identity -> owner + val scope : owner -> scope + val check_scope : owner -> scope -> (unit, error) result + + (** Exact live owner/scope; close is idempotent only after Execute joins producers. Received-delivery cleanup survives close. *) + val close_scope : owner -> scope -> (unit, error) result + + val child + : owner + -> scope + -> 'a task + -> poll:(unit -> ('a, error) result option) + -> ('a child, error) result + + val check_child : owner -> scope -> 'a child -> (unit, error) result + + val child_result + : owner + -> scope + -> 'a child + -> (('a, error) result option, error) result + + (** Bind poll to the exact typed task after reservation; check owner/live scope/unconsumed registration before polling. + None is unfinished; repeatable Some is not join. Forget only after physical join; + repeated original forget succeeds, consumed check/result fails. Prune callbacks without growing tombstones. *) + val forget_child : owner -> scope -> 'a child -> (unit, error) result + + val demand_receipt : owner -> scope -> (demand_receipt, error) result + val unlock_receipt : owner -> scope -> (unlock_receipt, error) result + val outbox_check : owner -> scope -> (outbox_check, error) result + val notifications : owner -> scope -> (notifications, error) result + + val capacity_revision + : owner + -> scope + -> sequence:int64 + -> (capacity_revision, error) result + + (** Validate owner/live scope, not latest-sequence equality; old captures remain valid atomic-wait baselines without a revision registry. *) + val capacity_sequence : owner -> scope -> capacity_revision -> (int64, error) result + + val check_demand_receipt : owner -> scope -> demand_receipt -> (unit, error) result + val check_unlock_receipt : owner -> scope -> unlock_receipt -> (unit, error) result + val check_outbox_check : owner -> scope -> outbox_check -> (unit, error) result + + (** Fresh registered identities; Execute atomically checks current selection/revision/expiry/epoch/queue and consumption. Never resurrect handles. *) + val check_notifications : owner -> scope -> notifications -> (unit, error) result + + val demand_output + : owner + -> scope + -> demand_receipt + -> id:demand_id + -> demand_availability + -> (demand_output, error) result + + val check_delivery : owner -> demand_delivery -> (unit, error) result + val delivery_released : owner -> demand_delivery -> (bool, error) result + val acknowledge_delivery : owner -> demand_delivery -> (unit, error) result + val same_reference_registration : resource_reference -> resource_reference -> bool + + val reference_state + : owner + -> resource_reference + -> ([ `Unregistered | `Retained | `Released ], error) result + + val register_reference : owner -> resource_reference -> (unit, error) result + + (** Execute reserves before registration and ACKs only physical release. Exact-owner/metadata ACKs + stay on original handles, not owner tombstone lists; released references cannot be registered again. + Producer constructs only; Execute validates publication/resources/capacity. Cleanup survives scope close. *) + val acknowledge_reference : owner -> resource_reference -> (unit, error) result +end + +type (_, _, _, _) operation = + | Run_task : 'a task -> (ready, ready, ready, 'a) operation + | Restore : (cold, ready, cold, restore_report) operation + | Install_initial_network : network_gate -> (ready, ready, ready, unit) operation + | Set_network : network_gate -> (ready, ready, ready, unit) operation + | Database : 'a database -> (ready, ready, ready, 'a) operation + | Transport : 'a transport -> (ready, ready, ready, 'a) operation + | Crypto : 'a crypto -> (ready, ready, ready, 'a) operation + | Assets : 'a assets -> (ready, ready, ready, 'a) operation + | Observe : ('s, 's, 's, observation) operation + | Begin_drain : 's drainable -> ('s, draining, draining, unit) operation + | Finish_retirement : (draining, retired, draining, unit) operation + | Release_file : asset_file -> ('s, 's, 's, unit) operation + | Release_lease : asset_lease -> ('s, 's, 's, unit) operation + | Release_snapshot : snapshot -> ('s, 's, 's, unit) operation + +val perform + : ('before, 'after, 'failed, 'a) operation + -> success:('a -> ('after, 'final, 'b) program) + -> failure:(error -> ('failed, 'final, 'b) program) + -> ('before, 'final, 'b) program + +type (_, _, _) node = + | Return : 'a -> ('s, 's, 'a) node + | Abort : error -> ('s, 'final, 'a) node + | Perform : + ('before, 'after, 'failed, 'a) operation + * ('a -> ('after, 'final, 'b) program) + * (error -> ('failed, 'final, 'b) program) + -> ('before, 'final, 'b) node + +(** Pure total projection; callbacks terminate without I/O/yield. Exceptions report actual phase. *) +val view : ('before, 'after, 'a) program -> ('before, 'after, 'a) node + +val run_task : 'a task -> (ready, ready, ready, 'a) operation +val restore : (cold, ready, cold, restore_report) operation + +(** Atomic one-shot install; admitted explicit controls supersede it, so later install is a no-op. + Admission rejection has zero effects, including eligibility. Admitted install/control consumes eligibility + before semantic Open epoch/old-generation-join checks, even on failure. Never queue or retry Open. *) +val install_initial_network : network_gate -> (ready, ready, ready, unit) operation + +val set_network : network_gate -> (ready, ready, ready, unit) operation +val database : 'a database -> (ready, ready, ready, 'a) operation +val transport : 'a transport -> (ready, ready, ready, 'a) operation +val crypto : 'a crypto -> (ready, ready, ready, 'a) operation +val assets : 'a assets -> (ready, ready, ready, 'a) operation + +(** Runtime starts Closed; local Restore preserves controls/eligibility. Missing mirror yields checkpoint None; + Inspect waits for activation. Locked restore reads metadata only; partial failure stays cold with cleanup owed. + Open checks actual epoch; Closed fences/joins without rollback. Remote effects recheck permission; + local inspection/reconciliation/unlock/release need none. Protected work fails Locked before bytes. *) +val observe : ('s, 's, 's, observation) operation + +val begin_drain : 's drainable -> ('s, draining, draining, unit) operation +val finish_retirement : (draining, retired, draining, unit) operation +val release_file : asset_file -> ('s, 's, 's, unit) operation +val release_lease : asset_lease -> ('s, 's, 's, unit) operation + +(** Drain fences irreversibly; failure stays draining. Retirement requires physical joins and cleanup, + retaining original owners after replacement. Retired allows release/cached observation only; + cancellation reports actual phase/cause, never implies rollback. *) +val release_snapshot : snapshot -> ('s, 's, 's, unit) operation + +type policy_config = private + { sync_retry : retry_policy + ; asset_retry : retry_policy + ; no_progress_seconds : float + } + +(** Positive finite no-progress delay; no default retry counts/timings. *) +val policy_config + : sync_retry:retry_policy + -> asset_retry:retry_policy + -> no_progress_seconds:float + -> (policy_config, error) result + +(** Restore, Install_initial_network, then Run_task; never replay the initial gate. Stop joins and returns Ready, + leaving retirement to the owner. Recover Submitted exact wires before fresh submissions; persist before Push. + Unknown outcomes reconcile original IDs. Deferred retains exact batch/outcome ID, pauses Pull/fresh sends, + then re-inspects/reapplies after settlement; scope exit drops only volatile bytes, preserving restart recovery. + No_progress uses configured delay; snapshots release on every path, deferred activation waits for change. + Only Network_failure/Remote_closed/Network_lost/Asset_network retry; join old connection before reconnect. + Retry Sleep cancels on stop/gate close. Reconnect budget resets after authoritative catch-up and Submitted + recovery; request budget after matched success; asset budget after available bytes or selection replacement. + Exhaustion retains cause until new authorized gate epoch; unlock restarts only key-blocked work. + Closed/Locked consume no retries; other failures require repair, not unchanged hints/dirtiness. + Capacity pressure retains plans/wires/batches/receipts; wait without blocking control/release. + Permanent bound mismatch fails. Withdrawals cancel/join only their interest; no adapter policy loops. *) +val sync_policy + : policy_config + -> initial_network:network_gate + -> (cold, ready, unit) program + +val error_message : error -> string diff --git a/logseq_sync/spec/sync_dsl/dune b/logseq_sync/spec/sync_dsl/dune new file mode 100644 index 00000000..3d80fb23 --- /dev/null +++ b/logseq_sync/spec/sync_dsl/dune @@ -0,0 +1,7 @@ +(library + (name sync_dsl) + (public_name logseq_sync.sync_dsl) + (modules core) + (virtual_modules core) + (default_implementation logseq_sync_sync_dsl_impl) + (libraries logseq_overlay_db logseq_db_types rrbvec)) diff --git a/logseq_sync/spec/sync_service/core.mli b/logseq_sync/spec/sync_service/core.mli new file mode 100644 index 00000000..17546ab0 --- /dev/null +++ b/logseq_sync/spec/sync_service/core.mli @@ -0,0 +1,70 @@ +(** Persistent service; contexts remain encapsulated. *) +module Dsl = Sync_dsl.Core +module Execute = Execute.Core + +type t +type config = + { identity : Dsl.identity + ; limits : Dsl.limits + ; network : Dsl.network_gate + ; policy : Dsl.policy_config } + +type ingress = + { outbox_committed : owner:Execute.database_owner -> identity:Dsl.identity + -> (unit, Dsl.error) result + ; submit_unlock : identity:Dsl.identity -> valid_for:float -> Dsl.unlock_input + -> (Execute.ingress_ack, Dsl.error) result + ; update_demand : Dsl.demand_change -> (Execute.ingress_ack, Dsl.error) result + ; receive_demand : unit -> (Dsl.demand_output, Dsl.error) result + ; demand_status : Dsl.demand_delivery -> (Execute.delivery_status, Dsl.error) result + ; release_demand : Dsl.demand_delivery -> (unit, Dsl.error) result } +(** Exact-runtime ingress follows Execute rules; rejection transfers nothing. Closing preserves delivery status/release, including failed cleanup. *) + +type 'clock dependencies = + { clock : 'clock Eio.Time.Mono.t + ; database_owner : Execute.database_owner + ; capabilities : Execute.capabilities + ; bind_ingress : ingress -> unit } +(** Caller keeps database/Worker owners alive through retirement and owns account coordination. *) + +type error = Dsl.error + +exception Reentrant_close +exception Close_already_called +exception Close_failed of + { error : error option + ; cause : (exn * Printexc.raw_backtrace) option } +(** Preserve at least one diagnostic and all original causes/backtraces. *) + +type lifecycle = Running | Closing | Closed +type status = + { lifecycle : lifecycle + ; observation : Execute.runtime_observation } +(** Online/idle does not mean synced. *) + +(** Runs Dsl.sync_policy; Ok means started, not synced. Startup failure retains failed cleanup under sw. + After Ok returns, bind_ingress runs once before on_change and policy inspection; + neither callback runs on Error. Binding must install the durable postcommit invalidator; + initial dirtiness covers earlier commits. Closing before binding suppresses policy start. + Callbacks must be nonblocking; snapshots coalesce, with no history or handle-storage guarantee. + Callback failure cleans up and propagates its original exception/backtrace to sw. *) +val start : + sw:Eio.Switch.t -> + config:config -> + dependencies:'clock dependencies -> + on_change:(status -> unit) -> + (t, error) result + +(** Closed immediately fences and schedules the exact-generation join; ACK is not join. + Open fails fast until that join succeeds; failed joins retain obligations, without retry. + Controls work during binding/Restore; Execute admission and initial-install rules apply. + Rejected Open is never deferred or replayed; no sync flush. *) +val set_network : t -> Sync_dsl.Core.network_gate -> (unit, error) result + +(** Own-callback calls raise Reentrant_close before effects, preserving the close right. + First accepted call consumes it; all later calls immediately raise Close_already_called. + Waits for retirement and callback quiescence, without upload/flush or requiring Open. + Consumers must stop using revoked bytes and release deliveries/imported-file leases; + keep independent release fibers and original owners alive. No forced lease timeout. + Failure raises Close_failed and retains Closing/cleanup obligations; no automatic retry. *) +val close : t -> unit diff --git a/logseq_sync/spec/sync_service/dune b/logseq_sync/spec/sync_service/dune new file mode 100644 index 00000000..725ef963 --- /dev/null +++ b/logseq_sync/spec/sync_service/dune @@ -0,0 +1,7 @@ +(library + (name logseq_sync_sync_service) + (public_name logseq_sync.sync_service) + (modules core) + (virtual_modules core) + (default_implementation logseq_sync_sync_service_impl) + (libraries eio sync_dsl execute)) diff --git a/logseq_sync/test/asset_adapter_memory.ml b/logseq_sync/test/asset_adapter_memory.ml index b06727c9..6ceaa951 100644 --- a/logseq_sync/test/asset_adapter_memory.ml +++ b/logseq_sync/test/asset_adapter_memory.ml @@ -1,250 +1,162 @@ -(* Exercise the public runner with its production Apple adapter and synthetic keys. *) -module R = Logseq_sync_effect_runner.Effect_runner -module C = Logseq_sync_pure_reducer.Core -module A = Logseq_db_types.Asset_descriptor -module Codec = Logseq_sync_effect_runner.Asset_codec +module D = Sync_dsl.Core +module Codec = Logseq_sync_native_crypto.Asset_codec +module Native = Logseq_sync_native_crypto.Platform_crypto +module Owner = Logseq_sync_assets.Core +module Intent = Logseq_db_types.Asset_upload_intent let get = Result.get_ok -let uuid = Logseq_db_types.Graph_types.Uuid.of_string -let graph_id = get (uuid "11111111-1111-4111-8111-111111111111") +let ok = get -let key_request () = - let limits = - get - (C.limits - ~maximum_response_bytes:Logseq_db_types.Limits.maximum_response_bytes - ~maximum_artifact_bytes:1073741824 - ~submission_batch_size:32) - in - let config = - get - (C.config ~managed_sync_origin:(Uri.of_string "https://asset-test.invalid") ~limits) - in - let initial = get (C.initial config) in - let initial = (C.step initial (C.Sign_in_requested {account_generation=0; presentation_generation=0})).next in - let authenticated = - C.step initial (C.Account_authenticated { user_id = Some "fixture" }) - in - let graph : C.graph = - { graph_id - ; name = "Fixture" - ; encrypted = true - ; schema = { major = 1; minor = 0; exact = true } - } - in - let catalog = - List.find_map - (function - | C.Run (C.Request (ticket, C.Fetch_catalog _)) -> - Some - (C.step - authenticated.next - (C.Runner_completed (C.Completion (ticket, Ok [ graph ])))) - | _ -> None) - authenticated.effects - |> Option.get - in - let selected = C.step catalog.next (C.Graph_selected graph_id) in - let scope = - List.find_map - (function - | C.Delegate (C.Inspect_mirror request) -> Some request.scope - | _ -> None) - selected.effects - |> Option.get - in - let missing = C.step selected.next (C.Mirror_inspected (C.Mirror_absent scope)) in - let instruction = - List.find_map - (function - | C.Run (C.Request (_, C.Load_and_unlock_graph_key _) as runnable) -> - Some runnable - | _ -> None) - missing.effects - |> Option.get - in - missing.next, instruction +let identity = + get + (D.identity + ~origin:"https://asset-test.invalid" + ~account:"fixture" + ~graph:"11111111-1111-4111-8111-111111111111" + ~account_generation:0L + ~graph_generation:0L) +;; + +let limits = + ok + (D.limits + ~maximum_items:8 + ~maximum_bytes:1024 + ~maximum_pending:8 + ~maximum_resources:8 + ~steps_per_turn:8 + ~maximum_tasks:8 + ~maximum_messages:8 + ~maximum_message_bytes:1024 + ~maximum_demands:8 + ~maximum_active_assets:2 + ~foreground_reserve:1 + ~foreground_demand_reserve:1 + ~foreground_descriptor_reserve_bytes:128 + ~asset_cache_budget_bytes:16777216L + ~asset_maximum_file_bytes:(8 * 1024 * 1024) + ~asset_pending_budget_bytes:16777216L + ~snapshot_maximum_artifact_bytes:4096L + ~snapshot_staging_budget_bytes:4096L + ~snapshot_scratch_budget_bytes:4096L) ;; let () = - let support = Sys.argv.(1) in - Eio_main.run (fun env -> - Eio.Switch.run (fun sw -> - let key = String.init 32 Char.chr in - let secrets = - get - (R.secrets - ~unlock_private_key: - (fun - ~managed_sync_origin:_ ~user_id:_ ~password:_ ~private_key_package:_ -> - Error "unused") - ~unlock_graph_key: - (fun - ~managed_sync_origin:_ ~user_id:_ ~encrypted_graph_key:_ -> Ok key) - ~load_wrapped_graph_key:(fun ~managed_sync_origin:_ ~user_id:_ ~graph_id:_ -> - Ok "synthetic") - ~verify_and_save_wrapped_graph_key: - (fun - ~managed_sync_origin:_ ~user_id:_ ~graph_id:_ ~encrypted_graph_key:_ -> - Error "unused") - ~delete_account_secrets:(fun ~managed_sync_origin:_ ~user_id:_ -> Ok ())) - in - let auth_calls = ref 0 in - let dependencies = - get - (R.dependencies - ~runtime:(get (R.runtime ~fork:(fun ~sw:_ f -> f ()) ~sleep:(fun _ -> ()))) - ~transport: - (get - (R.transport - ~tls_authenticator:(get (R.system_tls_authenticator ())) - ~network:(Eio.Stdenv.net env) - ~clock:(Eio.Stdenv.clock env) - ~websocket_liveness:R.Disabled)) - ~local_store:(get (R.local_store ~application_support_directory:support ())) - ~artifact_store: - (get - (R.artifact_store - ~staging_directory:(Filename.concat support "staging"))) - ~secrets - ~crypto:(get (R.apple_crypto ())) - ~id_token_provider: - (R.id_token_provider - ~acquire:(fun _ -> - incr auth_calls; - Error "fixture stops before network") - ~invalidate:(fun _ ~token:_ -> ()))) - in - let posted = ref [] in - let runner = - get (R.create ~sw dependencies ~post:(fun e -> posted := e :: !posted)) - in - let before_key, instruction = key_request () in - R.submit runner instruction; - let unlocked = - match !posted with - | [ event ] -> (C.step before_key event).next - | _ -> failwith "expected one key completion" - in - let context = C.asset_context unlocked |> Option.get in - let handle = context.key |> Option.get in - let owner = ref unlocked in - let protected operation action = - posted := []; - let requested = - C.step - !owner - (C.Protected_requested - { protected_operation = operation - ; protected_scope = context.scope - ; protected_action = action - }) - in - List.iter - (function - | C.Run runnable -> R.submit runner runnable - | _ -> ()) - requested.effects; - let completed = C.step requested.next (List.hd !posted) in - owner := completed.next; - List.find_map - (function - | C.Publish (C.Protected_finished output) -> Some output.protected_result - | _ -> None) - completed.effects - |> Option.get - |> get - in - List.iter - (fun size -> - let value = String.init size (fun i -> Char.chr (i mod 128)) in - let plaintext = - Transit_native.Transit.Json.to_string (Transit_core.Json.String value) - in - let iv, ciphertext = - match - protected - ("encrypt-" ^ string_of_int size) - (C.Encrypt_values (handle, [ plaintext ])) - with - | C.Encrypted_values [ pair ] -> pair - | _ -> failwith "missing encrypted value" - in - let wire = - Transit_native.Transit.Json.to_string - (Transit_core.Json.Array [ Binary iv; Binary ciphertext ]) - in - if - protected ("decrypt-" ^ string_of_int size) (C.Decrypt_value (handle, wire)) - <> C.Decrypted_value value - then failwith "production adapter roundtrip changed bytes") - [ 0; 1; 256; 4097; 131057 ]; - let size = 8 * 1024 * 1024 in - let plaintext = String.init size (fun i -> Char.chr (i mod 256)) in - let source_file = Filename.concat support "source.bin" in - Out_channel.with_open_bin source_file (fun out -> output_string out plaintext); - let version = - get (A.version ~checksum:(Codec.checksum plaintext) ~file_type:"bin") - in - Gc.full_major (); - let staged = - C.step - !owner - (C.Asset_requested - { scope = context.scope - ; operation = "stage" - ; action = - C.Stage_asset_file - { operation = graph_id; file_type = "bin"; source_file } - }) - in - posted := []; - List.iter - (function - | C.Run runnable -> R.submit runner runnable - | _ -> ()) - staged.effects; - let staged_done = C.step staged.next (List.hd !posted) in - let file = - List.find_map - (function - | C.Publish (C.Asset_finished { result = Ok (C.Asset_staged { file; _ }); _ }) - -> Some file - | _ -> None) - staged_done.effects - |> Option.get - in - let before = Gc.allocated_bytes () in - let upload = - C.step - staged_done.next - (C.Asset_requested - { scope = context.scope - ; operation = "upload" - ; action = - C.Put_asset_file - { asset = graph_id; version; file; maximum_plaintext_bytes = size } - }) - in - posted := []; - List.iter - (function - | C.Run runnable -> R.submit runner runnable - | _ -> ()) - upload.effects; - let finished = C.step upload.next (List.hd !posted) in - let outcome = - List.find_map - (function - | C.Publish (C.Asset_finished output) -> Some output.result - | _ -> None) - finished.effects - |> Option.get - in - let allocated = Gc.allocated_bytes () -. before in - if Result.map_error C.asset_failure_kind outcome <> Error C.Asset_authentication || !auth_calls <> 1 - then failwith "production adapter did not encode the maximum-size asset"; - Printf.printf "Production upload OCaml allocation bytes: %.0f\n%!" allocated; - if allocated > 128. *. 1024. *. 1024. - then failwith "asset adapter allocation exceeds 128 MiB")) + Eio_main.run (fun _ -> + let support = Sys.argv.(1) in + let key = String.init 32 Char.chr in + let crypto = Native.crypto in + List.iter + (fun size -> + let bytes = String.init size (fun index -> Char.chr (index mod 256)) in + let iv, ciphertext = get (crypto.encrypt_aes_gcm ~key ~plaintext:bytes) in + if get (crypto.decrypt_aes_gcm ~key ~iv ~ciphertext) <> bytes + then failwith "Production adapter roundtrip changed bytes") + [ 0; 1; 256; 4097; 131057 ]; + let crypto_owner = + Logseq_sync_crypto.Core.create + ~identity + ~limits + ~encrypted:true + ~backend:{ crypto; unlock = (fun ~identity:_ ~wrapped_graph_key:_ -> Ok key) } + in + ignore + (get + (Logseq_sync_crypto.Core.run + crypto_owner + ~check_current:(fun () -> Ok ()) + ~identity + (D.Unlock (get (D.unlock_input ~wrapped_graph_key:"synthetic"))))); + let auth_calls = ref 0 in + let network : Owner.network = + { get = + (fun ~identity:_ ~asset:_ ~version:_ ~maximum_wire_bytes:_ -> + Error (D.Invalid_input "Unexpected GET")) + ; put = + (fun ~identity:_ ~asset:_ ~version:_ ~wire:_ -> + incr auth_calls; + Error (D.Authentication_failure "synthetic authorization failure")) + } + in + let owner = + get + (Owner.create + ~root:support + ~identity + ~limits + ~crypto:crypto_owner + ~key:(Owner.native_key ~identity ~crypto:crypto_owner) + ~network) + in + Fun.protect + ~finally:(fun () -> + ignore (get (Owner.close owner)); + Logseq_sync_crypto.Core.retire crypto_owner) + (fun () -> + let size = 8 * 1024 * 1024 in + let bytes = String.init size (fun index -> Char.chr (index mod 256)) in + let source_file = Filename.concat support "source.bin" in + Out_channel.with_open_bin source_file (fun output -> output_string output bytes); + let uuid = + get + (Logseq_db_types.Graph_types.Uuid.of_string + "11111111-1111-4111-8111-111111111111") + in + let input : Logseq_db_types.Asset_import.t = + { operation = uuid + ; asset = uuid + ; replace_reference = None + ; target = uuid + ; local_mutation = uuid + ; metadata_mutation = uuid + ; source_file + ; title = "fixture" + ; file_type = "bin" + } + in + let staged = get (Owner.stage owner ~check_current:(fun () -> Ok ()) input) in + let version = + get + (Logseq_db_types.Asset_descriptor.version + ~checksum:(Codec.checksum bytes) + ~file_type:"bin") + in + let intent = + get + (Intent.prepare + ~operation_id:uuid + ~origin:"https://asset-test.invalid" + ~account:"fixture" + ~graph:uuid + ~asset:uuid + ~version + ~title:"fixture" + ~size:staged.size + ~staged_file:staged.file + ~replace_reference:None + ~target:uuid + ~local_mutation:uuid + ~metadata_mutation: + (get + (Logseq_db_types.Graph_types.Uuid.of_string + "11111111-1111-4111-8111-111111111112"))) + in + Gc.full_major (); + let before = Gc.allocated_bytes () in + let network : Execute.Core.worker_network = + { with_admission = (fun run -> run ()) } + in + let result = + Owner.put_staged owner ~check_current:(fun () -> Ok ()) ~network intent + in + let allocated = Gc.allocated_bytes () -. before in + if + result <> Error (D.Authentication_failure "synthetic authorization failure") + || !auth_calls <> 1 + then failwith "Production adapter did not encode the maximum-size asset"; + Printf.printf "Production upload OCaml allocation bytes: %.0f\n%!" allocated; + if allocated > 128. *. 1024. *. 1024. + then failwith "Asset adapter allocation exceeds 128 MiB"; + get + (Logseq_sync_assets.Cache.release_staged (Owner.cache owner) ~file:staged.file))) ;; diff --git a/logseq_sync/test/asset_native_interop.ml b/logseq_sync/test/asset_native_interop.ml index a9bcf621..2ba103e1 100644 --- a/logseq_sync/test/asset_native_interop.ml +++ b/logseq_sync/test/asset_native_interop.ml @@ -1,5 +1,5 @@ (* Run through tool/test_asset_interop.py with the production Apple crypto library. *) -module C = Logseq_sync_effect_runner.Asset_codec +module C = Logseq_sync_native_crypto.Asset_codec module T = Transit_native.Transit.Json external native_crypto diff --git a/logseq_sync/test/asset_owner_lane/asset_owner_lane_contract.ml b/logseq_sync/test/asset_owner_lane/asset_owner_lane_contract.ml new file mode 100644 index 00000000..3d3737fd --- /dev/null +++ b/logseq_sync/test/asset_owner_lane/asset_owner_lane_contract.ml @@ -0,0 +1,1279 @@ +module A = Logseq_db_types.Asset_descriptor +module S = Logseq_sync_assets.Selection + +let get = Result.get_ok +let ( let* ) = Result.bind + +let uuid = + get (Logseq_db_types.Graph_types.Uuid.of_string "11111111-1111-4111-8111-111111111111") +;; + +let version checksum = + get (A.version ~checksum:(String.make 64 checksum) ~file_type:"png") +;; + +let descriptor source current_checksum = + get (A.create ~uuid ~source ~current_checksum ~size:None ~dimensions:None) +;; + +let checksums versions = List.map (fun (v : A.version) -> v.checksum) versions + +let lookup_order () = + let remote = version 'b' in + let asset = descriptor (A.Managed (Some remote)) (Some (version 'a').checksum) in + Alcotest.(check (list string)) + "original before remote" + [ (version 'a').checksum; remote.checksum ] + (checksums (S.lookup_versions asset)); + Alcotest.(check (list string)) + "deduplicated" + [ remote.checksum ] + (checksums + (S.lookup_versions (descriptor (A.Managed (Some remote)) (Some remote.checksum)))) +;; + +let no_remote () = + let asset = descriptor (A.Managed None) (Some (version 'a').checksum) in + Alcotest.(check bool) + "missing type cannot invent local version" + true + (S.lookup_versions asset = []); + Alcotest.(check bool) "never GET original" true (S.download_version asset = None); + let external_asset = descriptor (A.External "https://example.invalid/a.png") None in + Alcotest.(check bool) + "external_asset is not managed" + true + (S.download_version external_asset = None && S.lookup_versions external_asset = []) +;; + +module D = Sync_dsl.Core +module Owner = Logseq_sync_assets.Core + +let ok = function + | Ok value -> value + | Error error -> Alcotest.fail (D.error_message error) +;; + +let identity = + ok + (D.identity + ~origin:"https://example.com" + ~account:"owner" + ~graph:"11111111-1111-4111-8111-111111111111" + ~account_generation:0L + ~graph_generation:0L) +;; + +let limits = + ok + (D.limits + ~maximum_items:8 + ~maximum_bytes:1024 + ~maximum_pending:8 + ~maximum_resources:1 + ~steps_per_turn:8 + ~maximum_tasks:8 + ~maximum_messages:8 + ~maximum_message_bytes:1024 + ~maximum_demands:8 + ~maximum_active_assets:2 + ~foreground_reserve:1 + ~foreground_demand_reserve:1 + ~foreground_descriptor_reserve_bytes:128 + ~asset_cache_budget_bytes:4096L + ~asset_maximum_file_bytes:1024 + ~asset_pending_budget_bytes:4096L + ~snapshot_maximum_artifact_bytes:4096L + ~snapshot_staging_budget_bytes:4096L + ~snapshot_scratch_budget_bytes:4096L) +;; + +let current () = Ok () +let original _ = Ok None +let upload _ = Error (D.Invalid_input "No import") + +let run owner operation = + Owner.run + owner + ~check_current:current + ~admit_network:current + ~original + ~upload + ~identity + operation +;; + +let rejected = function + | Error _ -> () + | Ok _ -> Alcotest.fail "Expected rejection" +;; + +let with_assets_env get callback = + Eio_main.run (fun env -> + let root = Filename.temp_file "asset-owner-contract" "" in + Sys.remove root; + Unix.mkdir root 0o700; + let crypto = + Logseq_sync_crypto.Core.create + ~identity + ~limits + ~encrypted:false + ~backend:Logseq_sync_crypto.Core.apple_backend + in + let network : Owner.network = + { get; put = (fun ~identity:_ ~asset:_ ~version:_ ~wire:_ -> Ok ()) } + in + let owner = + ok + (Owner.create + ~root + ~identity + ~limits + ~crypto + ~key:(fun ~check_current -> + let open Result in + bind (check_current ()) (fun () -> Ok None)) + ~network) + in + let rec remove path = + if Sys.is_directory path + then ( + Array.iter (fun name -> remove (Filename.concat path name)) (Sys.readdir path); + Unix.rmdir path) + else Sys.remove path + in + Fun.protect + ~finally:(fun () -> + ignore (Owner.close owner); + remove root) + (fun () -> callback owner env root)) +;; + +let with_assets get callback = with_assets_env get (fun owner _ _ -> callback owner) +let wire = "owner bytes" + +let wire_version = + get + (A.version + ~checksum:(Logseq_sync_native_crypto.Asset_codec.checksum wire) + ~file_type:"png") +;; + +let remote_asset = descriptor (A.Managed (Some wire_version)) (Some wire_version.checksum) + +let available = function + | D.Available file -> file + | _ -> Alcotest.fail "Expected file" +;; + +let ownership () = + with_assets + (fun ~identity:_ ~asset:_ ~version:_ ~maximum_wire_bytes:_ -> Ok (Some wire)) + (fun owner -> + let file = available (ok (run owner (D.Download remote_asset))) in + let forged_asset = descriptor (A.Managed None) None in + let forged = ok (D.asset_file file.file forged_asset) in + rejected (Owner.file_path owner forged); + rejected (Owner.release_file owner ~identity forged); + ignore (ok (Owner.file_path owner file)); + ok (Owner.release_file owner ~identity file); + ok (Owner.release_file owner ~identity file); + rejected (Owner.file_path owner file)) +;; + +let reserve_before_io () = + let requests = ref 0 in + let owner_ref = ref None in + with_assets + (fun ~identity:_ ~asset:_ ~version:_ ~maximum_wire_bytes:_ -> + incr requests; + if !requests = 1 + then rejected (run (Option.get !owner_ref) (D.Download remote_asset)); + Ok (Some wire)) + (fun owner -> + owner_ref := Some owner; + let file = available (ok (run owner (D.Download remote_asset))) in + Alcotest.(check int) "second request never started" 1 !requests; + ok (Owner.release_file owner ~identity file)) +;; + +let failed_transfer_releases_budget () = + let requests = ref 0 in + with_assets + (fun ~identity:_ ~asset:_ ~version:_ ~maximum_wire_bytes:_ -> + incr requests; + if !requests = 1 then Error D.Cancelled else Ok (Some wire)) + (fun owner -> + rejected (run owner (D.Download remote_asset)); + let file = available (ok (run owner (D.Download remote_asset))) in + ok (Owner.release_file owner ~identity file); + let file = available (ok (run owner (D.Download remote_asset))) in + ok (Owner.release_file owner ~identity file)) +;; + +module Intent = Logseq_db_types.Asset_upload_intent +module Store = Logseq_db_storage.Asset_upload_store + +let sql_ok = function + | Ok value -> value + | Error message -> Alcotest.fail message +;; + +let id ordinal = + get + (Logseq_db_types.Graph_types.Uuid.of_string + (Printf.sprintf "22222222-2222-4222-8222-%012d" ordinal)) +;; + +let checkpoint ordinal = + sql_ok + (Intent.prepare + ~operation_id:(id ordinal) + ~origin:identity.origin + ~account:identity.account + ~graph:uuid + ~asset:uuid + ~version:wire_version + ~title:"Import" + ~size:(Int64.of_int (String.length wire)) + ~staged_file:"original.png" + ~replace_reference:None + ~target:(id 100) + ~local_mutation:(id (ordinal + 200)) + ~metadata_mutation:(id (ordinal + 400))) +;; + +let original_lookup () = + let db = Sqlite3.db_open ":memory:" in + Fun.protect + ~finally:(fun () -> ignore (Sqlite3.db_close db)) + (fun () -> + sql_ok (Store.initialize_database db); + let first = checkpoint 1 in + sql_ok (Store.save db ~expected:None first); + let cancelled = sql_ok (Intent.advance first Cancelled) in + sql_ok (Store.save db ~expected:(Some 0) cancelled); + let second = checkpoint 2 in + sql_ok (Store.save db ~expected:None second); + Alcotest.(check bool) + "skips cancelled exact original" + true + (sql_ok + (Store.find_original + db + ~origin:identity.origin + ~account:identity.account + ~graph:uuid + ~asset:uuid + ~checksum:wire_version.checksum) + = Some second); + Alcotest.(check bool) + "foreign owner has no original" + true + (sql_ok + (Store.find_original + db + ~origin:identity.origin + ~account:"foreign" + ~graph:uuid + ~asset:uuid + ~checksum:wire_version.checksum) + = None)) +;; + +let original_index_upgrade () = + let db = Sqlite3.db_open ":memory:" in + Fun.protect + ~finally:(fun () -> ignore (Sqlite3.db_close db)) + (fun () -> + sql_ok (Store.initialize_database db); + sql_ok (Store.save db ~expected:None (checkpoint 1)); + Sqlite3.Rc.check + (Sqlite3.exec + db + "DROP INDEX IF EXISTS asset_upload_original; CREATE INDEX \ + asset_upload_original ON \ + asset_upload_intents(origin,account,graph,json_extract(payload,'$.asset'),json_extract(payload,'$.version.checksum'),operation) \ + WHERE json_extract(payload,'$.phase[0]') <> 'Cancelled'"); + sql_ok (Store.initialize_database db); + Sqlite3.Rc.check + (Sqlite3.exec db "UPDATE asset_upload_intents SET payload='invalid checkpoint'"); + rejected (Store.read db ~operation:(id 1)); + sql_ok (Store.save db ~expected:None (checkpoint 2)); + Alcotest.(check bool) + "guarded index preserves bounded exact lookup" + true + (sql_ok + (Store.find_original + db + ~origin:identity.origin + ~account:identity.account + ~graph:uuid + ~asset:uuid + ~checksum:wire_version.checksum) + = Some (checkpoint 2))) +;; + +module Worker = Logseq_sync_worker_owner.Core +module X = Execute.Core + +let seed_database root = + let module Storage = Logseq_db_storage.Logseq_sqlite_storage in + let module Checkpoint = Logseq_db_types.Sync_checkpoint in + let module Graph = Logseq_db_types.Graph_types in + let attribute : Datascript.schema_attr = + { cardinality = One + ; unique = None + ; indexed = true + ; is_component = false + ; no_history = false + ; doc = None + ; value_type = None + ; tuple_attrs = None + ; tuple_types = None + } + in + let schema = + List.map + (fun name -> + ( name + , if name = "block/uuid" || name = "db/ident" + then { attribute with unique = Some Identity } + else if name = "block/parent" || name = "block/page" + then { attribute with value_type = Some RefType } + else if name = "block/tags" + then { attribute with cardinality = Many; value_type = Some RefType } + else attribute )) + [ "db/ident" + ; "block/uuid" + ; "block/title" + ; "block/name" + ; "block/journal-day" + ; "block/link" + ; "block/refs" + ; "block/parent" + ; "block/page" + ; "block/order" + ; "block/tags" + ; "kv/value" + ; "logseq.property.asset/type" + ; "logseq.property.asset/checksum" + ; "logseq.property.asset/size" + ] + in + let fact e a v : Datascript.datom = { e; a; v; tx = 536870913; added = true } in + let kv e name value = [ fact e "db/ident" (Keyword name); fact e "kv/value" value ] in + let facts = + List.concat + [ kv 1 "logseq.kv/graph-remote?" (Bool true) + ; kv 2 "logseq.kv/graph-uuid" (Uuid (Graph.Uuid.to_string uuid)) + ; kv 3 "logseq.kv/db-type" (String "db") + ; kv + 4 + "logseq.kv/schema-version" + (Map [ Keyword "major", Int64 65L; Keyword "minor", Int64 33L ]) + ; [ fact 5 "block/uuid" (Uuid (Graph.Uuid.to_string (id 100))) + ; fact 5 "block/name" (String "Imports") + ; fact 5 "block/title" (String "Imports") + ] + ; [ fact 6 "db/ident" (Keyword "logseq.class/Asset") ] + ] + in + let rec mkdir directory = + if not (Sys.file_exists directory) + then ( + mkdir (Filename.dirname directory); + Unix.mkdir directory 0o700) + in + let directory = + Filename.concat root ("logseq-db-worker/synced-graphs/" ^ Graph.Uuid.to_string uuid) + in + mkdir directory; + let path = Filename.concat directory "db.sqlite" in + let sqlite = Sqlite3.db_open path in + Fun.protect + ~finally:(fun () -> ignore (Sqlite3.db_close sqlite)) + (fun () -> + Sqlite3.Rc.check + (Sqlite3.exec + sqlite + "CREATE TABLE kvs (addr INTEGER PRIMARY KEY, content TEXT, addresses JSON)"); + sql_ok + (Logseq_db_storage.Sync_checkpoint_store.initialize_database + sqlite + (sql_ok + (Checkpoint.create + ~graph_id:uuid + ~schema:Graph.{ major = 65; minor = 33 } + ~applied_server_t:(get (Logseq_db_types.Server_cursor.of_int64 0L)) + ~checksum:"0000000000000000"))); + sql_ok (Logseq_db_storage.Sync_outbox_store.initialize_database sqlite)); + let connection = get (Storage.open_database path) in + let callbacks = Storage.connection_callbacks connection in + Fun.protect + ~finally:(fun () -> ignore (Storage.close callbacks)) + (fun () -> + sql_ok (callbacks.begin_staging ()); + let stored = + Datascript.store ~storage:callbacks.storage (Datascript.init_db ~schema facts) + in + ignore stored; + let batch = sql_ok (callbacks.finish_staging None []) in + get (Storage.commit_batch callbacks batch)) +;; + +let with_worker + ?(live_database = false) + ?(notify = fun ~owner:_ ~identity:_ -> Ok ()) + ?(on_store = fun _ -> ()) + ?(wire_batch_max_bytes = 1024) + callback + = + with_assets_env + (fun ~identity:_ ~asset:_ ~version:_ ~maximum_wire_bytes:_ -> Ok None) + (fun assets env root -> + Eio.Switch.run (fun sw -> + let module O = D.Overlay in + let deps = + get + (O.Database.dependencies + ~epoch_ms:(fun () -> 0L) + ~monotonic_ns:(fun () -> 0L) + ~limits: + { O.Types.response_budget_bytes = 1024 + ; outbox_max_records = 8 + ; outbox_max_bytes = max 4096 wire_batch_max_bytes + ; change_max_items = 8 + ; change_max_bytes = 1024 + ; dispatcher_capacity = 8 + ; wire_batch_max_bytes + }) + in + if live_database then seed_database root; + let inspection = + get + (O.Database.inspect_mirror + ~application_support_directory:root + ~graph_id:uuid) + in + let database_owner = + if live_database + then + X.database_owner + (get (O.Database.open_ ~sw deps inspection ~graph_name:"Owner fixture")) + else + ok + (X.activation_owner + ~sw + ~dependencies:deps + ~inspection + ~graph_name:"Owner fixture" + ~snapshot_source:(fun _ -> Error (D.Invalid_input "No snapshot"))) + in + let db = Sqlite3.db_open ":memory:" in + Fun.protect + ~finally:(fun () -> ignore (Sqlite3.db_close db)) + (fun () -> + sql_ok (Store.initialize_database db); + on_store db; + let worker = + Worker.create + ~clock:(Eio.Stdenv.mono_clock env) + ~namespace:(String.make 64 'a') + ~assets + ~store:(Worker.store db) + ~notify_outbox_changed:notify + in + let module W = (val Worker.worker_owner worker : X.WORKER_OWNER) in + ok + (W.bind_runtime + ~identity + ~database_owner + ~limits + ~cleanup_seconds:0.01 + ~network:{ with_admission = (fun callback -> callback ()) }); + callback worker (module W : X.WORKER_OWNER) database_owner root))) +;; + +let import_request_guard () = + List.iter (fun retire_after -> + with_worker (fun _ packed _ root -> + let module W = (val packed : X.WORKER_OWNER) in + let source_file = Filename.concat root "guard.png" in + let channel = open_out_bin source_file in + output_string channel wire; + close_out channel; + let request : Logseq_db_types.Asset_import.t = + { operation = id 2; asset = uuid; replace_reference = None; target = id 100 + ; local_mutation = id 202; metadata_mutation = id 402; source_file + ; title = "Guarded"; file_type = "png" } + in + ok (W.with_scope ~identity ~operation:request.operation (fun scope -> + let checks = ref 0 in + let current () = incr checks; !checks < retire_after in + rejected (W.prepare_import scope request ~current); + let input = get (Logseq_db_types.Asset_import.Pure_intent.create + ~namespace:(String.make 64 'a') request) + |> Logseq_db_types.Asset_import.Pure_intent.encode in + Alcotest.(check bool) "stale input never becomes Prepared" true + (match ok (W.confirm_import_input scope input) with + | None -> true | Some item -> item.prepared = None); + Ok ())))) [1; 2; 3; 4; 5; 6; 7; 8] +;; + +let staging_reconciliation_serializes_preparation () = + with_worker (fun worker packed _ root -> + let module W = (val packed : X.WORKER_OWNER) in + Eio.Switch.run (fun sw -> + let source_file = Filename.concat root "serialized.png" in + let channel = open_out_bin source_file in + output_string channel wire; + close_out channel; + let request : Logseq_db_types.Asset_import.t = + { operation = id 3; asset = uuid; replace_reference = None; target = id 100 + ; local_mutation = id 203; metadata_mutation = id 403; source_file + ; title = "Serialized"; file_type = "png" } in + let finished, resolve = Eio.Promise.create () in + let started = ref false in + let current () = + if not !started then ( + started := true; + Eio.Fiber.fork ~sw (fun () -> + Eio.Promise.resolve resolve + (Worker.run_assets worker ~identity ~operation:(id 903) D.Reconcile_staging))); + for _ = 1 to 4 do Eio.Fiber.yield () done; + Alcotest.(check bool) "reconciliation waits throughout stage and durable Prepared" false + (Eio.Promise.is_resolved finished); + true in + let intent = ok (W.with_scope ~identity ~operation:request.operation (fun scope -> + W.prepare_import scope request ~current)) in + Alcotest.(check int) "serialized reconciliation keeps the committed source" 0 + (ok (Eio.Promise.await finished)); + Sys.remove source_file; + ok (W.with_scope ~identity ~operation:request.operation (fun scope -> + let file = ok (W.retain_imported_file scope intent) |> Option.get in + Alcotest.(check bool) "original source remains readable without picker" true + (Sys.file_exists (W.imported_file_path file)); + W.release_imported_file file)); + let other = ok (D.identity ~origin:identity.origin ~account:"other" ~graph:identity.graph + ~account_generation:0L ~graph_generation:0L) in + rejected (Worker.run_assets worker ~identity:other ~operation:(id 904) D.Reconcile_staging); + ok (Worker.fence worker ~identity); + rejected (Worker.run_assets worker ~identity ~operation:(id 905) D.Reconcile_staging); + ok (Worker.await_quiescent worker ~identity))) +;; + +let import_ownership () = + with_worker (fun worker packed database_owner root -> + let module W = (val packed : X.WORKER_OWNER) in + let source_file = Filename.concat root "picker.png" in + let channel = open_out_bin source_file in + output_string channel wire; + close_out channel; + let request : Logseq_db_types.Asset_import.t = + { operation = id 1 + ; asset = uuid + ; replace_reference = None + ; target = id 100 + ; local_mutation = id 201 + ; metadata_mutation = id 401 + ; source_file + ; title = "Import" + ; file_type = "png" + } + in + let held = ref None in + let cancelled = + ok + (W.with_scope ~identity ~operation:request.operation (fun scope -> + let intent = ok (W.prepare_import scope request ~current:(fun () -> true)) in + let input = + get + (Logseq_db_types.Asset_import.Pure_intent.create + ~namespace:(String.make 64 'a') + request) + |> Logseq_db_types.Asset_import.Pure_intent.encode + in + let confirmed = Option.get (ok (W.confirm_import_input scope input)) in + Alcotest.(check bool) + "recovery retains exact original input" + true + (confirmed.input = input && confirmed.prepared <> None); + rejected + (W.recover_import_inputs + scope + ~after:None + ~maximum_items:9 + ~maximum_bytes:1024); + ok + (W.acknowledge_import_input + scope + ~input + ~prepared:(Option.get confirmed.prepared)); + Alcotest.(check bool) + "exact recovery acknowledgement is durable" + true + (Option.get (ok (W.confirm_import_input scope input))).acknowledged; + Sys.remove source_file; + Alcotest.(check bool) + "retry uses original staging without picker" + true + (ok (W.prepare_import scope request ~current:(fun () -> true)) = intent); + rejected (W.prepare_import scope { request with title = "Changed" } ~current:(fun () -> true)); + rejected (W.publish_import_preview scope (checkpoint 1)); + rejected (W.release_import_staging scope intent); + held := ok (W.retain_imported_file scope intent); + Alcotest.(check bool) + "lease pins existing path" + true + (Sys.file_exists (W.imported_file_path (Option.get !held))); + ok (W.cancel_import scope intent); + let cancelled = sql_ok (Intent.advance intent Cancelled) in + ok (W.release_import_staging scope cancelled); + Alcotest.(check bool) + "terminal retirement preserves held preview" + true + (Sys.file_exists (W.imported_file_path (Option.get !held))); + Ok cancelled)) + in + ok (Worker.database_changed worker ~owner:database_owner ~identity); + rejected + (Worker.database_changed + worker + ~owner:database_owner + ~identity: + (ok + (D.identity + ~origin:identity.origin + ~account:"other" + ~graph:identity.graph + ~account_generation:0L + ~graph_generation:0L))); + let file = Option.get !held in + ok (W.release_imported_file file); + ok (W.release_imported_file file); + Alcotest.(check bool) + "last consumer release removes terminal source" + false + (Sys.file_exists (W.imported_file_path file)); + ok + (W.with_scope ~identity ~operation:request.operation (fun scope -> + W.release_import_staging scope cancelled)); + ok (Worker.fence worker ~identity); + rejected (W.with_scope ~identity ~operation:(id 2) (fun _ -> Ok ())); + ok (Worker.await_quiescent worker ~identity)) +;; + +let committed_import_hooks () = + let notifications = ref 0 in + let notified_owner = ref None in + let fail_notification = ref true in + let notify ~owner ~identity:requested = + incr notifications; + notified_owner := Some owner; + Alcotest.(check bool) "exact notification identity" true (requested = identity); + if !fail_notification + then Error (D.Storage_failure "Synthetic postcommit notification failure") + else Ok () + in + let upload_store = ref None in + with_worker + ~live_database:true + ~notify + ~wire_batch_max_bytes:65536 + ~on_store:(fun db -> upload_store := Some db) + (fun worker packed database_owner root -> + let module W = (val packed : X.WORKER_OWNER) in + let held = ref None in + let source_file = Filename.concat root "picker.png" in + let channel = open_out_bin source_file in + output_string channel wire; + close_out channel; + let request : Logseq_db_types.Asset_import.t = + { operation = id 1 + ; asset = uuid + ; replace_reference = None + ; target = id 100 + ; local_mutation = id 201 + ; metadata_mutation = id 401 + ; source_file + ; title = "Import" + ; file_type = "png" + } + in + ok + (W.with_scope ~identity ~operation:request.operation (fun scope -> + let intent = ok (W.prepare_import scope request ~current:(fun () -> true)) in + ok (W.publish_import_preview scope intent); + rejected (W.apply_local_import scope intent); + Alcotest.(check bool) + "notification failure preserves local commit" + true + (ok (W.inspect_import scope intent) = `Local_present); + fail_notification := false; + ok (W.apply_local_import scope intent); + ok (W.put_import scope intent); + ok (W.apply_import_metadata scope intent); + Alcotest.(check bool) + "PUT and metadata are not publication ACK" + true + (ok (W.inspect_import scope intent) = `Metadata_present); + ok (W.apply_import_metadata scope intent); + let database = ok (X.database database_owner) in + let sync = get (D.Overlay.Database.inspect_sync database) in + Alcotest.(check int) + "exact original mutations, no replay" + 2 + (List.length (D.Overlay.Types.sync_view_submissions sync)); + Alcotest.(check bool) + "exact notification binding" + true + (Option.get !notified_owner == database_owner); + Alcotest.(check int) + "every successful or recovered commit notifies" + 4 + !notifications; + Sys.remove source_file; + Alcotest.(check int) "owner reconciliation retains Metadata_pending source" 0 + (ok (Worker.run_assets worker ~identity ~operation:(id 902) D.Reconcile_staging)); + held := ok (W.retain_imported_file scope intent); + Ok ())); + let module O = D.Overlay in + let module T = O.Types in + let database = ok (X.database database_owner) in + let read_intent () = + Option.get + (sql_ok (Store.read (Option.get !upload_store) ~operation:request.operation)) + in + let file = Option.get !held in + let path = W.imported_file_path file in + let sqlite_proof ~pending = + let graph = Logseq_db_types.Graph_types.Uuid.to_string uuid in + let path = + Filename.concat root ("logseq-db-worker/synced-graphs/" ^ graph ^ "/db.sqlite") + in + let sqlite = Sqlite3.db_open ~mode:`READONLY path in + Fun.protect + ~finally:(fun () -> ignore (Sqlite3.db_close sqlite)) + (fun () -> + let count query = + let result = ref (-1) in + Sqlite3.Rc.check + (Sqlite3.exec sqlite query ~cb:(fun row _ -> + result := int_of_string (Option.get row.(0)))); + !result + in + Alcotest.(check int) + "SQLite exact metadata terminal Applied receipt before/after ACK" + (if pending then 0 else 1) + (count + (Printf.sprintf + "SELECT count(*) FROM overlay_mutation_receipts WHERE \ + json_extract(receipt,'$[1].mutation_id')='%s' AND \ + json_extract(receipt,'$[1].status[0]')='Applied'" + (Logseq_db_types.Graph_types.Uuid.to_string + request.metadata_mutation))); + Alcotest.(check int) + "SQLite original outbox before/after authoritative echo" + (if pending then 2 else 0) + (count "SELECT count(*) FROM sync_outbox")) + in + Eio.Switch.run (fun sw -> + let finished, resolver = Eio.Promise.create () in + Eio.Fiber.fork ~sw (fun () -> + Eio.Promise.resolve + resolver + (Worker.run_assets + worker + ~identity + ~operation:(id 900) + (D.Upload + (descriptor + (A.Managed (Some wire_version)) + (Some wire_version.checksum))))); + for _ = 1 to 16 do + Eio.Fiber.yield () + done; + if Eio.Promise.is_resolved finished then ignore (ok (Eio.Promise.await finished)); + Alcotest.(check bool) + "upload waits for actual metadata ACK" + true + ((read_intent ()).phase = Intent.Metadata_pending + && not (Eio.Promise.is_resolved finished)); + let logical_notifications = ref 0 in + let subscription, snapshot = get (O.Database.listen database) in + O.Database.release_snapshot snapshot; + get + (O.Database.activate_subscription subscription ~notify:(fun _ -> + incr logical_notifications)); + Fun.protect + ~finally:(fun () -> O.Database.unlisten subscription) + (fun () -> + let transition action = + let view = get (O.Database.inspect_sync database) in + let prepared, protection = + match + O.Database.begin_outbox_transition + database + ~expected:(T.sync_view_token view) + action + with + | Ok prepared -> prepared + | Error (T.Outbox_transition_invalid message) -> Alcotest.fail message + | Error T.Outbox_limit_exceeded -> + Alcotest.fail "submission exceeds fixture wire budget" + | Error (T.Outbox_dependency_ineligible _) -> + Alcotest.fail "submission dependency ineligible" + | Error _ -> Alcotest.fail "submission preparation rejected" + in + let encrypted = + Option.map + (fun request -> request, O.Database.protection_plaintexts request) + protection + in + get (O.Database.apply_outbox_transition database prepared ~encrypted) + in + let submitted = + Option.get + (transition + (T.Submit_group [ request.local_mutation; request.metadata_mutation ])) + .submission_batch + in + let wires = T.submission_batch_wires submitted in + Alcotest.(check bool) + "submission contains exact metadata mutation" + true + (List.exists + (fun wire -> + T.submission_wire_mutation_id wire = request.metadata_mutation) + wires); + let through = + get (T.Server_cursor.of_int64 (Int64.of_int (List.length wires))) + in + ignore + (transition + (T.Accept_group + { batch_id = T.submission_batch_id submitted + ; barrier = + { through + ; checksum = + get (T.Checksum.of_string "checksum:v1:0000000000000000") + } + })); + Alcotest.(check bool) + "transport acceptance alone retains metadata outbox" + true + (List.exists + (fun (item : T.submission_descriptor) -> + item.mutation_id = request.metadata_mutation) + (T.sync_view_submissions (get (O.Database.inspect_sync database)))); + let normalize transaction = + let module Transit = Transit_core.Json in + let module Codec = Transit_native.Transit.Json in + let entity = function + | Transit.String value + when String.starts_with ~prefix:"overlay-insert:" value + || String.starts_with ~prefix:"overlay-page:" value -> + let start = String.index value ':' + 1 in + Transit.String (String.sub value start (String.length value - start)) + | value -> value + in + let datom = function + | Transit.Array + [ (Transit.Keyword ("db/add" | "db/retract") as operation) + ; target + ; (Transit.Keyword attribute as encoded_attribute) + ; value + ] -> + let value = + if attribute = "block/parent" || attribute = "block/page" + then entity value + else value + in + Transit.Array + [ operation + ; entity target + ; encoded_attribute + ; value + ; Transit.Int 1 + ] + | operation -> operation + in + match Codec.of_string transaction with + | Transit.Array operations -> + Codec.to_string + ~mode:Codec.Verbose + (Transit.Array (List.map datom operations)) + | _ -> Alcotest.fail "submission is not a transaction" + in + let transactions = + List.mapi + (fun ordinal wire -> + let cursor = + get (T.Server_cursor.of_int64 (Int64.of_int (ordinal + 1))) + in + let normalized = + normalize (T.submission_wire_protected_transaction wire) + in + let transaction = + get + (T.encoded_transaction_of_string ~maximum_bytes:65536 normalized) + in + T.authoritative_transaction ~cursor ~transaction) + wires + in + let batch = + get + (T.authoritative_batch + ~maximum_count:8 + ~maximum_bytes:65536 + ~transactions + ~through + ~checksum:None) + in + sqlite_proof ~pending:true; + Alcotest.(check bool) + "original remains pinned through ACK" + true + (Sys.file_exists path); + let before = get (O.Database.inspect_sync database) in + let prepared, protection = + get + (O.Database.begin_authoritative + database + ~expected:(T.sync_view_token before) + batch) + in + let decrypted = + Option.map + (fun request -> + ( request + , List.map + (fun (item, encoded) -> + match Transit_native.Transit.Json.of_string encoded with + | Transit_core.Json.String plaintext -> item, plaintext + | _ -> + Alcotest.fail "synthetic protection did not encode a string") + (O.Database.unprotection_ciphertexts request) )) + protection + in + let commit = + match O.Database.apply_authoritative database prepared ~decrypted with + | Ok (O.Database.Authoritative_applied commit) -> commit + | Ok (Authoritative_deferred _) -> Alcotest.fail "own echo deferred" + | Error + ( T.Authoritative_decode_failed message + | Authoritative_integrity_failure message + | Authoritative_commit_persistence_failed message ) -> + Alcotest.fail message + | Error _ -> Alcotest.fail "authoritative own echo rejected" + in + Alcotest.(check bool) + "authoritative own echo has no logical change" + true + (commit.logical_change_summary = T.No_logical_change); + Alcotest.(check bool) + "exact metadata Applied receipt" + true + (List.exists + (fun (receipt : T.terminal_receipt) -> + match receipt.receipt with + | T.Applied_receipt { mutation_id; _ } -> + mutation_id = request.metadata_mutation + | _ -> false) + commit.terminal_receipts); + Alcotest.(check int) + "actual own echo clears outbox" + 0 + (List.length + (T.sync_view_submissions (get (O.Database.inspect_sync database)))); + sqlite_proof ~pending:false; + for _ = 1 to 16 do + Eio.Fiber.yield () + done; + Alcotest.(check int) + "ACK never notifies logical subscribers" + 0 + !logical_notifications; + Alcotest.(check bool) + "ACK wakes public upload fiber" + true + (Eio.Promise.is_resolved finished); + ignore (ok (Eio.Promise.await finished)); + let complete = read_intent () in + Alcotest.(check bool) + "upload persists Complete to SQLite" + true + (complete.phase = Intent.Complete); + Alcotest.(check bool) + "Complete retirement preserves pinned original" + true + (Sys.file_exists path); + ok (W.release_imported_file file); + Alcotest.(check bool) + "final pin release unlinks original" + false + (Sys.file_exists path); + ok (W.release_imported_file file); + ok + (W.with_scope ~identity ~operation:request.operation (fun scope -> + W.release_import_staging scope complete)); + ignore + (ok + (Worker.run_assets + worker + ~identity + ~operation:(id 901) + (D.Upload + (descriptor + (A.Managed (Some wire_version)) + (Some wire_version.checksum))))); + Alcotest.(check bool) + "public Complete replay tolerates already absent original" + true + (read_intent () = complete && not (Sys.file_exists path)))); + ok (Worker.fence worker ~identity); + ok (Worker.await_quiescent worker ~identity)) +;; + +let shared_crypto_key () = + Eio_main.run (fun _ -> + let key = String.make 32 'k' in + let backend : Logseq_sync_crypto.Core.backend = + { crypto = Logseq_sync_native_crypto.Platform_crypto.crypto + ; unlock = (fun ~identity:_ ~wrapped_graph_key:_ -> Ok key) + } + in + let crypto = + Logseq_sync_crypto.Core.create ~identity ~limits ~encrypted:true ~backend + in + let read_key () = Owner.native_key ~identity ~crypto ~check_current:current in + rejected (read_key ()); + ignore + (ok + (Logseq_sync_crypto.Core.run + crypto + ~check_current:current + ~identity + (D.Unlock (ok (D.unlock_input ~wrapped_graph_key:"synthetic-wrapped"))))); + Alcotest.(check (option string)) + "assets use the installed worker key" + (Some key) + (ok (read_key ())); + Logseq_sync_crypto.Core.retire crypto; + rejected (read_key ())) +;; + +let scope_cancel () = + with_worker (fun worker packed _ _ -> + let module W = (val packed : X.WORKER_OWNER) in + rejected + (W.with_scope ~identity ~operation:(id 1) (fun scope -> + rejected (W.with_scope ~identity ~operation:(id 1) (fun _ -> Ok ())); + let* () = W.cancel scope in + W.crypto scope (D.Decrypt_value "secret"))); + ok (W.with_scope ~identity ~operation:(id 1) (fun _ -> Ok ())); + ok (Worker.fence worker ~identity); + ok (Worker.await_quiescent worker ~identity)) +;; + +let absent_native_coordinator ~owned ~available () = + with_assets_env + (fun ~identity:_ ~asset:_ ~version:_ ~maximum_wire_bytes:_ -> Ok None) + (fun _ env root -> + Eio.Switch.run (fun sw -> + let module C = Journal_sync_lifecycle.Core in + if available then seed_database root; + let deps = get (D.Overlay.Database.dependencies + ~epoch_ms:(fun () -> 0L) ~monotonic_ns:(fun () -> 0L) + ~limits:{ D.Overlay.Types.response_budget_bytes = 1024 + ; outbox_max_records = 8; outbox_max_bytes = 4096 + ; change_max_items = 8; change_max_bytes = 1024 + ; dispatcher_capacity = 8; wire_batch_max_bytes = 1024 }) in + let inspection = get (D.Overlay.Database.inspect_mirror + ~application_support_directory:root ~graph_id:uuid) in + let database_owner = ok (X.activation_owner ~sw ~dependencies:deps + ~inspection ~graph_name:"Absent native fixture" + ~snapshot_source:(fun _ -> Error (D.Invalid_input "No offline snapshot"))) in + let sqlite = Sqlite3.db_open ":memory:" in + Fun.protect ~finally:(fun () -> + Alcotest.(check bool) "borrowed upload store closes" true (Sqlite3.db_close sqlite)) + (fun () -> + sql_ok (Store.initialize_database sqlite); + let retry = ok (D.retry_policy ~initial_seconds:0.01 ~maximum_seconds:0.01 + ~multiplier:1. ~maximum_attempts:0) in + let config : Logseq_sync_sync_service.Core.config = + { identity; limits; network = D.Closed + ; policy = ok (D.policy_config ~sync_retry:retry ~asset_retry:retry + ~no_progress_seconds:0.01) } in + let changes = ref [] and errors = ref [] and tokens = ref 0 in + let start = if owned then + C.start_owned_native ~dependencies:deps ~graph_name:"Owned native fixture" + else C.start_native ~database_owner in + let service = ok (start ~sw ~clock:(Eio.Stdenv.mono_clock env) + ~network:(Eio.Stdenv.net env) + ~authenticator:(X509.Authenticator.chain_of_trust + ~time:(fun () -> Some (Ptime_clock.now ())) []) + ~config ~deadlines:{ X.setup_seconds = 0.1; tls_handshake_seconds = 0.1 + ; http_response_seconds = 0.1; download_seconds = 0.1 } + ~liveness:X.Peer_close_only ~cleanup_seconds:1. ~root + ~upload_store:(Worker.store sqlite) ~namespace:(String.make 64 'a') ~encrypted:false + ~token:(fun ~epoch:_ ~refresh:_ -> incr tokens; + Error (D.Authentication_failure "Offline fixture")) + ~on_change:(fun status -> changes := status :: !changes) + ~on_error:(fun error -> errors := error :: !errors)) in + for _ = 1 to 16 do Eio.Fiber.yield () done; + ignore (ok (C.ingress service)); + Alcotest.(check bool) "mirror checkpoint matches presence" true + (List.exists (fun (status : Logseq_sync_sync_service.Core.status) -> + Option.is_some status.observation.state.checkpoint = available) !changes); + C.close service; + if owned && available then ( + let inspection = get (D.Overlay.Database.inspect_mirror + ~application_support_directory:root ~graph_id:uuid) in + let reopened = get (D.Overlay.Database.open_ ~sw deps inspection + ~graph_name:"Reopened owned fixture") in + get (D.Overlay.Database.close reopened)); + Alcotest.(check int) "offline absent mirror never authenticates" 0 !tokens; + Alcotest.(check int) "no subscription error for absent mirror" 0 (List.length !errors)))) +;; + +let native_coordinator () = + with_assets_env + (fun ~identity:_ ~asset:_ ~version:_ ~maximum_wire_bytes:_ -> Ok None) + (fun _ env root -> + Eio.Switch.run (fun sw -> + seed_database root; + let deps = get (D.Overlay.Database.dependencies + ~epoch_ms:(fun () -> 0L) ~monotonic_ns:(fun () -> 0L) + ~limits:{ D.Overlay.Types.response_budget_bytes = 1024 + ; outbox_max_records = 8; outbox_max_bytes = 4096 + ; change_max_items = 8; change_max_bytes = 1024 + ; dispatcher_capacity = 8; wire_batch_max_bytes = 1024 }) in + let inspection = get (D.Overlay.Database.inspect_mirror + ~application_support_directory:root ~graph_id:uuid) in + let database = get (D.Overlay.Database.open_ ~sw deps inspection + ~graph_name:"Native coordinator fixture") in + let database_owner = X.database_owner database in + let sqlite = Sqlite3.db_open ":memory:" in + Fun.protect ~finally:(fun () -> + Alcotest.(check bool) "borrowed upload store closes" true (Sqlite3.db_close sqlite)) + (fun () -> + sql_ok (Store.initialize_database sqlite); + let module C = Journal_sync_lifecycle.Core in + let retry = ok (D.retry_policy ~initial_seconds:0.01 ~maximum_seconds:0.01 + ~multiplier:1. ~maximum_attempts:0) in + let config : Logseq_sync_sync_service.Core.config = + { identity; limits; network = D.Closed + ; policy = ok (D.policy_config ~sync_retry:retry ~asset_retry:retry + ~no_progress_seconds:0.01) } in + let calls = ref 0 and changes = ref 0 and errors = ref [] in + let start namespace = C.start_native ~sw ~clock:(Eio.Stdenv.mono_clock env) + ~network:(Eio.Stdenv.net env) + ~authenticator:(X509.Authenticator.chain_of_trust + ~time:(fun () -> Some (Ptime_clock.now ())) []) + ~config ~deadlines:{ X.setup_seconds = 0.1; tls_handshake_seconds = 0.1 + ; http_response_seconds = 0.1; download_seconds = 0.1 } + ~liveness:X.Peer_close_only ~cleanup_seconds:1. ~root ~database_owner + ~upload_store:(Worker.store sqlite) ~namespace ~encrypted:false + ~token:(fun ~epoch:_ ~refresh:_ -> incr calls; + Error (D.Authentication_failure "Closed fixture must not request a token")) + ~on_change:(fun _ -> incr changes) ~on_error:(fun error -> errors := error :: !errors) + in + rejected (start ""); + let coordinator = ok (start (String.make 64 'a')) in + (match start (String.make 64 'a') with + | Error (D.Identity_mismatch _) -> () + | Error error -> Alcotest.fail (D.error_message error) + | Ok _ -> Alcotest.fail "A second service borrowed the same database owner"); + for _ = 1 to 16 do Eio.Fiber.yield () done; + ignore (ok (C.ingress coordinator)); + Alcotest.(check bool) "status delivered after start" true (!changes > 0); + let worker = C.worker_owner coordinator in + let module W = (val Worker.worker_owner worker : X.WORKER_OWNER) in + let source_file = Filename.concat root "native-picker.png" in + let channel = open_out_bin source_file in + output_string channel wire; + close_out channel; + let request : Logseq_db_types.Asset_import.t = + { operation = id 1; asset = uuid; replace_reference = None; target = id 100 + ; local_mutation = id 201; metadata_mutation = id 401; source_file + ; title = "Native import"; file_type = "png" } in + ok (W.with_scope ~identity ~operation:request.operation (fun scope -> + let intent = ok (W.prepare_import scope request ~current:(fun () -> true)) in + ok (W.publish_import_preview scope intent); + ok (W.apply_local_import scope intent); + (match W.put_import scope intent with + | Error D.Network_closed -> () + | Error error -> Alcotest.fail (D.error_message error) + | Ok () -> Alcotest.fail "Closed native owner admitted PUT"); + ok (W.cancel_import scope intent); + let cancelled = sql_ok (Intent.advance intent Cancelled) in + W.release_import_staging scope cancelled)); + ok (C.projection_changed coordinator); + (try C.close coordinator with + | Logseq_sync_sync_service.Core.Close_failed {error; cause} -> + let rec describe = function + | D.Cleanup_failure {primary; failures} -> + String.concat "; " (Option.to_list (Option.map describe primary) + @ List.map describe (List.of_seq (Rrbvec.to_seq failures))) + | error -> D.error_message error in + let message = Option.fold ~none:"No typed close error" ~some:describe error in + let cause = Option.fold ~none:"" ~some:(fun (exn, _) -> Printexc.to_string exn) cause in + Alcotest.fail (message ^ " " ^ cause)); + rejected (C.projection_changed coordinator); + Alcotest.(check int) "offline native service never authenticates" 0 !calls; + Alcotest.(check int) "postcommit notifications succeed" 0 (List.length !errors); + let sync = get (D.Overlay.Database.inspect_sync database) in + Alcotest.(check int) "borrowed database survives sync close" 1 + (List.length (D.Overlay.Types.sync_view_submissions sync)); + Alcotest.check_raises "close right is consumed" + Logseq_sync_sync_service.Core.Close_already_called + (fun () -> C.close coordinator)))) +;; + +let () = + Alcotest.run + "asset owner lane" + [ ( "worker" + , [ Alcotest.test_case "absent mirror starts offline" `Quick + (absent_native_coordinator ~owned:false ~available:false) + ; Alcotest.test_case "App owns absent mirror" `Quick + (absent_native_coordinator ~owned:true ~available:false) + ; Alcotest.test_case "App closes available mirror after Sync" `Quick + (absent_native_coordinator ~owned:true ~available:true) + ; Alcotest.test_case "native coordinator owns real offline imports" `Quick native_coordinator + ; Alcotest.test_case "request guard fences Prepared" `Quick import_request_guard + ; Alcotest.test_case "staging reconciliation serializes preparation" `Quick staging_reconciliation_serializes_preparation + ; Alcotest.test_case "assets share worker crypto key" `Quick shared_crypto_key + ; Alcotest.test_case + "real commits and postcommit hooks" + `Quick + committed_import_hooks + ; Alcotest.test_case + "original proof cancellation and lease release" + `Quick + import_ownership + ; Alcotest.test_case "scope cancellation physically joins" `Quick scope_cancel + ] ) + ; ( "imports" + , [ Alcotest.test_case + "legacy index upgrade preserves checkpoint validation" + `Quick + original_index_upgrade + ; Alcotest.test_case "bounded exact original lookup" `Quick original_lookup + ] ) + ; ( "ownership" + , [ Alcotest.test_case "exact file metadata" `Quick ownership + ; Alcotest.test_case "reserve before I/O" `Quick reserve_before_io + ; Alcotest.test_case + "cancelled transfer returns budget" + `Quick + failed_transfer_releases_budget + ] ) + ; ( "selection" + , [ Alcotest.test_case "offline current first" `Quick lookup_order + ; Alcotest.test_case "no implicit transfer" `Quick no_remote + ] ) + ] +;; diff --git a/logseq_sync/test/asset_owner_lane/dune b/logseq_sync/test/asset_owner_lane/dune new file mode 100644 index 00000000..b09dc367 --- /dev/null +++ b/logseq_sync/test/asset_owner_lane/dune @@ -0,0 +1,4 @@ +(test + (name asset_owner_lane_contract) + (modules asset_owner_lane_contract) + (libraries melange-transit-core melange-transit-native datascript_ocaml datascript_ocaml.types alcotest logseq_sync.assets logseq_sync.worker_owner logseq_sync.crypto logseq_journal.sync_lifecycle logseq_sync_sync_service_impl logseq_sync_execute_impl logseq_sync_sync_dsl_impl x509 ptime.clock.os logseq_overlay_db.impl datascript-ocaml-native logseq_db_storage logseq_overlay_db sqlite3 eio eio_main logseq_db_types sync_dsl logseq_sync.native_crypto execute rrbvec unix)) diff --git a/logseq_sync/test/core_contract.ml b/logseq_sync/test/core_contract.ml deleted file mode 100644 index 885643b3..00000000 --- a/logseq_sync/test/core_contract.ml +++ /dev/null @@ -1,6918 +0,0 @@ -let server_cursor_of_int value = - Logseq_db_types.Server_cursor.of_int64 (Int64.of_int value) |> Result.get_ok -;; - -let cursor_testable = - Alcotest.testable - (fun fmt cursor -> - Format.fprintf fmt "%Ld" (Logseq_db_types.Server_cursor.to_int64 cursor)) - Logseq_db_types.Server_cursor.equal -;; - -module Core = Logseq_sync_pure_reducer.Core - -type observed = - { state : Core.state - ; admitted_graph_scope : Core.graph_scope option - } - -let observe core = - { state = Core.state core; admitted_graph_scope = Core.admitted_graph_scope core } -;; - -let check_observed test_id message expected actual = - Alcotest.(check bool) (test_id ^ " " ^ message) true (expected = actual) -;; - -let check_instructions test_id expected actual = - Alcotest.(check bool) - (test_id ^ " ordered instructions") - true - (Core.equal_instructions expected actual) -;; - -let preview_step test_id origin event = - let origin_observed = observe origin in - let transition = Core.step origin event in - check_observed test_id "preview preserves origin" origin_observed (observe origin); - transition -;; - -let check_replay test_id origin event (first : Core.transition) = - let origin_observed = observe origin in - check_observed test_id "origin remains immutable" origin_observed (observe origin); - let replay = Core.step origin event in - check_observed test_id "replay observation" (observe first.next) (observe replay.next); - check_instructions test_id first.effects replay.effects; - first -;; - -let check_step test_id origin event expected effects = - let origin_observed = observe origin in - let first = Core.step origin event in - check_observed test_id "public observation" expected (observe first.next); - check_instructions test_id effects first.effects; - check_observed test_id "origin remains immutable" origin_observed (observe origin); - check_replay test_id origin event first -;; - -let limits () = - Core.limits - ~maximum_response_bytes:Logseq_db_types.Limits.maximum_response_bytes - ~maximum_artifact_bytes:(1024 * 1024) - ~submission_batch_size:32 - |> Result.get_ok -;; - -let raw_initial () = - Core.config - ~managed_sync_origin:(Uri.of_string "https://api.logseq.io") - ~limits:(limits ()) - |> Result.get_ok - |> Core.initial - |> Result.get_ok -;; - -let startup_scope core : Core.startup_scope = - let f = (Core.state core).snapshot.startup in - { account_generation=f.account_generation;presentation_generation=f.presentation_generation } -;; -let timeline_scope core : Core.timeline_scope = - let s=(Core.state core).snapshot in - {startup=startup_scope core;graph_id=Option.get s.selected_graph;graph_generation=s.startup.graph_generation} -;; -let initial () = (Core.step (raw_initial ()) (Core.Sign_in_requested (startup_scope (raw_initial ())))).next -;; - -let graph_id = - Logseq_db_types.Graph_types.Uuid.of_string "11111111-1111-4111-8111-111111111111" - |> Result.get_ok -;; - -let graph : Core.graph = - { graph_id - ; name = "Journal" - ; schema = { major = 1; minor = 0; exact = true } - ; encrypted = false - } -;; - -let encrypted_graph = { graph with encrypted = true } - -let selected_graph selected_graph = - let authenticated = - Core.step (initial ()) (Core.Account_authenticated { user_id = Some "user" }) - in - let catalog = - List.find_map - (function - | Core.Run (Core.Request (ticket, Core.Fetch_catalog _)) -> - Some - (Core.step - authenticated.next - (Core.Runner_completed (Core.Completion (ticket, Ok [ selected_graph ])))) - | Run _ | Delegate _ | Publish _ -> None) - authenticated.effects - |> Option.get - in - let selected = Core.step catalog.next (Core.Graph_selected selected_graph.graph_id) in - selected, Core.admitted_graph_scope selected.next |> Option.get -;; - -(* Invalid/stale submission scenarios still carry a genuine challenge, obtained - through production events rather than a test constructor. *) -let fixture_challenge = lazy ( - let selected, scope = selected_graph encrypted_graph in - let absent = Core.step selected.next (Core.Mirror_inspected (Core.Mirror_absent scope)) in - let prepare = List.find_map (function - | Core.Run (Core.Request (ticket, Core.Load_and_unlock_graph_key _)) -> - Some (Core.step absent.next (Core.Runner_completed - (Core.Completion (ticket, Error (Core.Effect_failed "synthetic missing key"))))) - | _ -> None) absent.effects |> Option.get in - let graph_key = List.find_map (function - | Core.Run (Core.Request (ticket, Core.Fetch_e2ee_graph_key _)) -> - Some (Core.step prepare.next (Core.Runner_completed - (Core.Completion (ticket, Ok "encrypted-graph-key")))) - | _ -> None) prepare.effects |> Option.get in - let prompt = List.find_map (function - | Core.Run (Core.Request (ticket, Core.Fetch_e2ee_user_keys _)) -> - Some (Core.step graph_key.next (Core.Runner_completed - (Core.Completion (ticket, Ok "private-key-package")))) - | _ -> None) graph_key.effects |> Option.get in - Core.unlock_challenge (Core.state prompt.next).snapshot.unlock |> Option.get) -;; - -let password_event core password = - let challenge = Core.unlock_challenge (Core.state core).snapshot.unlock - |> Option.value ~default:(Lazy.force fixture_challenge) in - Core.E2ee_password_submitted (challenge, password) -;; - -let other_graph_id = - Logseq_db_types.Graph_types.Uuid.of_string "99999999-9999-4999-8999-999999999999" - |> Result.get_ok -;; - -let other_graph : Core.graph = - { graph_id = other_graph_id - ; name = "Other Journal" - ; schema = { major = 1; minor = 0; exact = true } - ; encrypted = false - } -;; - -let save_catalog_cache effects = - List.find_map - (function - | Core.Run (Core.Request (_, Core.Save_catalog { cache; _ })) -> Some cache - | Run _ | Delegate _ | Publish _ -> None) - effects - |> Option.get -;; - -let load_catalog_completion effects (cache : Core.catalog_cache option) = - List.find_map - (function - | Core.Run (Core.Request (ticket, Core.Load_catalog _)) -> - Some (Core.Runner_completed (Core.Completion (ticket, Ok cache))) - | Run _ | Delegate _ | Publish _ -> None) - effects - |> Option.get -;; - -let fetch_catalog_completion effects (graphs : Core.graph list) = - List.find_map - (function - | Core.Run (Core.Request (ticket, Core.Fetch_catalog _)) -> - Some (Core.Runner_completed (Core.Completion (ticket, Ok graphs))) - | Run _ | Delegate _ | Publish _ -> None) - effects - |> Option.get -;; - -let inspect_mirror_request effects = - List.find_map - (function - | Core.Delegate (Core.Inspect_mirror request) -> Some request - | Run _ | Delegate _ | Publish _ -> None) - effects - |> Option.get -;; - -let has_graph_open_work effects = - List.exists - (function - | Core.Delegate (Core.Inspect_mirror _) - | Core.Delegate (Core.Attach_graph _) - | Core.Delegate (Core.Activate_snapshot _) -> true - | Run _ | Delegate _ | Publish _ -> false) - effects -;; - -let has_catalog_request effects = - List.exists - (function - | Core.Run (Core.Request (_, Core.Fetch_catalog _)) -> true - | Run _ | Delegate _ | Publish _ -> false) - effects -;; - -let refresh_catalog core graphs = - let requested = Core.step core (Core.Catalog_refresh_requested (startup_scope core)) in - Core.step requested.next (fetch_catalog_completion requested.effects graphs) -;; - -let selected_graph_survives_picker_and_codec_restart () = - let selected, _ = selected_graph graph in - let persisted = - save_catalog_cache selected.effects - |> Core.encode_catalog_cache - |> Core.decode_catalog_cache - |> Result.get_ok - in - let picker = Core.step selected.next Core.Graph_picker_requested in - let picker_snapshot = (Core.state picker.next).snapshot in - Alcotest.(check bool) - "picker clears only the current process selection" - true - (picker_snapshot.selected_graph = None && picker_snapshot.startup.awaiting_selection); - Alcotest.(check bool) - "picker does not overwrite the durable selection" - false - (List.exists - (function - | Core.Run (Core.Request (_, Core.Save_catalog _)) -> true - | Run _ | Delegate _ | Publish _ -> false) - picker.effects); - let restoring = - Core.step (initial ()) (Core.Restore_local_account { user_id = "user" }) - in - let generation_before = (Core.state restoring.next).snapshot.startup.graph_generation in - let restored = - Core.step restoring.next (load_catalog_completion restoring.effects (Some persisted)) - in - let snapshot = (Core.state restored.next).snapshot in - Alcotest.(check bool) - "warm launch restores the cached selected graph" - true - (Option.equal - Logseq_db_types.Graph_types.Uuid.equal - snapshot.selected_graph - (Some graph_id)); - Alcotest.(check bool) - "warm launch bypasses graph selection with a fresh generation" - true - ((not snapshot.startup.awaiting_selection) - && snapshot.startup.graph_generation > generation_before); - let request = inspect_mirror_request restored.effects in - Alcotest.(check bool) - "warm launch inspects the admitted cached graph" - true - (Logseq_db_types.Graph_types.Uuid.equal request.graph.graph_id graph_id - && request.scope.graph_generation = snapshot.startup.graph_generation); - Alcotest.(check bool) - "cache loading does not rewrite the cache" - false - (List.exists - (function - | Core.Run (Core.Request (_, Core.Save_catalog _)) -> true - | Run _ | Delegate _ | Publish _ -> false) - restored.effects) -;; - -let invalid_cached_selections_fail_closed () = - let check label cache = - let restoring = - Core.step (initial ()) (Core.Restore_local_account { user_id = "user" }) - in - let restored = - Core.step restoring.next (load_catalog_completion restoring.effects (Some cache)) - in - let snapshot = (Core.state restored.next).snapshot in - Alcotest.(check bool) - (label ^ " awaits graph selection") - true - (snapshot.startup.awaiting_selection && snapshot.selected_graph = None); - Alcotest.(check bool) - (label ^ " starts no graph work") - false - (has_graph_open_work restored.effects) - in - check - "absent cached selection" - (Core.catalog_cache ~user_id:"user" ~graphs:[ graph ] ~selected_graph:None); - check - "stale cached selection" - (Core.catalog_cache - ~user_id:"user" - ~graphs:[ graph ] - ~selected_graph:(Some other_graph_id)); - check - "different-account cached selection" - (Core.catalog_cache - ~user_id:"other-user" - ~graphs:[ graph ] - ~selected_graph:(Some graph_id)) -;; - -let same_account_authentication_preserves_pending_warm_restore () = - let selected, _ = selected_graph graph in - let cache = save_catalog_cache selected.effects in - let restoring = - Core.step (initial ()) (Core.Restore_local_account { user_id = "user" }) - in - let reconciled = - Core.step restoring.next (Core.Account_authenticated { user_id = Some "user" }) - in - Alcotest.(check bool) - "same-account authentication waits for local cache admission" - false - (has_catalog_request reconciled.effects); - let restored = - Core.step reconciled.next (load_catalog_completion restoring.effects (Some cache)) - in - let snapshot = (Core.state restored.next).snapshot in - Alcotest.(check bool) - "pending warm restore survives same-account authentication" - true - ((not snapshot.startup.authenticated) - && (not snapshot.startup.awaiting_selection) - && Option.equal - Logseq_db_types.Graph_types.Uuid.equal - snapshot.selected_graph - (Some graph_id)); - ignore (inspect_mirror_request restored.effects); - Alcotest.(check bool) - "local admission alone does not authorize remote reconciliation" - false - (has_catalog_request restored.effects) -;; - -let catalog_refresh_preserves_or_revokes_selection () = - let selected, _ = selected_graph graph in - let before = (Core.state selected.next).snapshot in - let preserved = refresh_catalog selected.next [ graph; other_graph ] in - let preserved_snapshot = (Core.state preserved.next).snapshot in - Alcotest.(check bool) - "refresh preserves an admitted selected graph" - true - (Option.equal - Logseq_db_types.Graph_types.Uuid.equal - preserved_snapshot.selected_graph - (Some graph_id) - && (not preserved_snapshot.startup.awaiting_selection) - && preserved_snapshot.startup.graph_generation = before.startup.graph_generation); - Alcotest.(check bool) - "refresh persists the admitted selection" - true - (Option.equal - Logseq_db_types.Graph_types.Uuid.equal - (Core.catalog_cache_selected_graph (save_catalog_cache preserved.effects)) - (Some graph_id)); - let selected, _ = selected_graph graph in - let before = (Core.state selected.next).snapshot in - let revoked = refresh_catalog selected.next [ other_graph ] in - let revoked_snapshot = (Core.state revoked.next).snapshot in - Alcotest.(check bool) - "refresh clears and fences a removed selection" - true - (revoked_snapshot.selected_graph = None - && revoked_snapshot.startup.awaiting_selection - && revoked_snapshot.startup.graph_generation > before.startup.graph_generation); - Alcotest.(check bool) - "refresh detaches the removed graph" - true - (List.exists - (function - | Core.Delegate (Core.Detach_graph _) -> true - | Run _ | Delegate _ | Publish _ -> false) - revoked.effects); - Alcotest.(check bool) - "refresh starts no work for the removed graph" - false - (has_graph_open_work revoked.effects); - Alcotest.(check (option string)) - "refresh persists no removed selection" - None - (Core.catalog_cache_selected_graph (save_catalog_cache revoked.effects) - |> Option.map Logseq_db_types.Graph_types.Uuid.to_string) -;; - -let catalog_save_failure_keeps_selected_graph_usable () = - let selected, scope = selected_graph graph in - let failed = - List.find_map - (function - | Core.Run (Core.Request (ticket, Core.Save_catalog _)) -> - Some - (Core.step - selected.next - (Core.Runner_completed - (Core.Completion (ticket, Error (Core.Effect_failed "disk unavailable"))))) - | Run _ | Delegate _ | Publish _ -> None) - selected.effects - |> Option.get - in - Alcotest.(check bool) - "advisory cache save failure preserves selection" - true - ((Core.state failed.next).snapshot = (Core.state selected.next).snapshot); - let groups = (Core.state failed.next).diagnostics.groups in - Alcotest.(check bool) "ignored catalog save is visible in Diagnostics" true - (List.exists (fun (g : Core.diagnostic_group) -> - List.exists (fun (_, value) -> String.starts_with ~prefix:"saveCatalog" value) g.entries) groups); - Alcotest.(check bool) "diagnostic update is published" true - (List.exists (function Core.Publish (Core.State_changed _) -> true | _ -> false) failed.effects); - let duplicate = List.find_map (function - | Core.Run (Core.Request (ticket, Core.Save_catalog _)) -> - Some (Core.step failed.next (Core.Runner_completed - (Core.Completion (ticket, Error (Core.Effect_failed "sensitive ignored duplicate"))))) - | _ -> None) selected.effects |> Option.get in - Alcotest.(check bool) "consumed duplicate cannot add or replace a diagnostic" true - (Core.state duplicate.next = Core.state failed.next && duplicate.effects = []); - let changed = Core.step failed.next (Core.Account_authenticated {user_id=Some "other-user"}) in - Alcotest.(check bool) "new account cannot inherit advisory errors" true - ((Core.state changed.next).diagnostics.groups = []); - let inspected = - Core.step failed.next (Core.Mirror_inspected (Core.Mirror_available { graph; scope })) - in - Alcotest.(check bool) - "mirror opening continues after cache save failure" - true - (List.exists - (function - | Core.Delegate (Core.Attach_graph request) -> request.scope = scope - | Run _ | Delegate _ | Publish _ -> false) - inspected.effects) -;; - -let warm_encrypted_graph_loads_key_before_attach_and_protects_outbox () = - let selected, scope = selected_graph encrypted_graph in - let mirror = Core.{ graph = encrypted_graph; scope } in - let inspected = - Core.step selected.next (Core.Mirror_inspected (Mirror_available mirror)) - in - let key = Core.graph_key_handle ~id:"warm-graph-key" ~scope in - let key_request = - List.find_map - (function - | Core.Run (Core.Request (ticket, Core.Load_and_unlock_graph_key requested_scope)) - -> - Some - ( requested_scope - , Core.step - inspected.next - (Core.Runner_completed (Core.Completion (ticket, Ok key))) ) - | Run _ | Delegate _ | Publish _ -> None) - inspected.effects - in - Alcotest.(check bool) - "encrypted warm mirror requests its cached key" - true - (Option.is_some key_request); - Alcotest.(check bool) - "encrypted warm mirror is not attached before key recovery" - false - (List.exists - (function - | Core.Delegate (Core.Attach_graph _) -> true - | Run _ | Delegate _ | Publish _ -> false) - inspected.effects); - match key_request with - | None -> () - | Some (requested_scope, unlocked) -> - Alcotest.(check bool) - "cached-key request retains the selected scope" - true - (requested_scope = scope); - let attached_request = - List.find_map - (function - | Core.Delegate (Core.Attach_graph request) -> Some request - | Run _ | Delegate _ | Publish _ -> None) - unlocked.effects - in - Alcotest.(check bool) - "warm mirror attaches after cached-key recovery" - true - (Option.is_some attached_request); - (match attached_request with - | None -> () - | Some request -> - Alcotest.(check bool) - "attachment retains selected scope" - true - (request.scope = scope)); - let sync_token = - Logseq_overlay_db.Types.sync_token_of_string "sync-token:v1:warm" |> Result.get_ok - in - let checkpoint = - Logseq_overlay_db.Types.Server_cursor.of_int64 0L - |> Result.map_error (fun `Negative_cursor -> "negative cursor") - |> Result.get_ok - in - let sync = - Logseq_overlay_db.Types.sync_view ~token:sync_token ~checkpoint ~submissions:[] - in - let attached = Core.step unlocked.next (Core.Graph_attached { scope; sync }) in - let connection = - List.find_map - (function - | Core.Run (Core.Start_websocket request) -> Some request.scope - | Run _ | Delegate _ | Publish _ -> None) - attached.effects - |> Option.get - in - let opened = Core.step attached.next (Core.Websocket_opened connection) in - let mutation_id = - Logseq_db_types.Graph_types.Uuid.of_string "22222222-2222-4222-8222-222222222220" - |> Result.get_ok - in - let fingerprint = - Logseq_overlay_db.Types.Mutation_fingerprint.of_string - "mutation-fingerprint:v1:warm" - |> Result.get_ok - in - let pending : Logseq_overlay_db.Types.submission_descriptor = - { mutation_id - ; fingerprint - ; state = Logseq_overlay_db.Types.Queued - ; operation = Logseq_overlay_db.Types.Save_block_operation - ; dependency_eligible = true - ; attempt_count = 0 - ; plaintext_bytes = 16 - ; protected_bytes = None - } - in - let pending_sync = - Logseq_overlay_db.Types.sync_view - ~token:sync_token - ~checkpoint - ~submissions:[ pending ] - in - let changed = Core.step opened.next Core.Local_outbox_changed in - let inspection_scope = - List.find_map - (function - | Core.Delegate (Core.Inspect_sync requested_scope) -> Some requested_scope - | Run _ | Delegate _ | Publish _ -> None) - changed.effects - |> Option.get - in - let planned = - Core.step - changed.next - (Core.Sync_inspected { scope = inspection_scope; sync = pending_sync }) - in - let transition_key = - List.find_map - (function - | Core.Delegate (Core.Apply_outbox_transition request) -> request.key - | Run _ | Delegate _ | Publish _ -> None) - planned.effects - in - Alcotest.(check (option string)) - "outbox protection retains the recovered key" - (Some "warm-graph-key") - (Option.map Core.graph_key_handle_id transition_key) -;; - -let graph_picker_clears_terminal_graph_failure () = - let selected, scope = selected_graph encrypted_graph in - let missing = - Core.step selected.next (Core.Mirror_inspected (Core.Mirror_absent scope)) - in - let failed = - List.find_map - (function - | Core.Run (Core.Request (ticket, Core.Load_and_unlock_graph_key _)) -> - Some - (Core.step - missing.next - (Core.Runner_completed - (Core.Completion - (ticket, Error (Core.Effect_failed "cached key unavailable"))))) - | Run _ | Delegate _ | Publish _ -> None) - missing.effects - |> Option.get - in - let picker = Core.step failed.next Core.Graph_picker_requested in - let snapshot = (Core.state picker.next).snapshot in - Alcotest.(check bool) - "graph picker clears the terminal failure" - true - (snapshot.startup.awaiting_selection - && snapshot.startup.failure = None - && snapshot.last_error = None - && snapshot.sync_phase = Core.Offline - && snapshot.selected_graph = None) -;; - -let graph_picker_fences_late_warm_key_completion () = - let selected, scope = selected_graph encrypted_graph in - let mirror = Core.{ graph = encrypted_graph; scope } in - let inspected = - Core.step selected.next (Core.Mirror_inspected (Mirror_available mirror)) - in - let key_completion = - List.find_map - (function - | Core.Run (Core.Request (ticket, Core.Load_and_unlock_graph_key _)) -> - let picker = Core.step inspected.next Core.Graph_picker_requested in - let key = Core.graph_key_handle ~id:"stale-warm-key" ~scope in - Some - (Core.step - picker.next - (Core.Runner_completed (Core.Completion (ticket, Ok key)))) - | Run _ | Delegate _ | Publish _ -> None) - inspected.effects - in - Alcotest.(check bool) - "warm encrypted mirror has a pending key request" - true - (Option.is_some key_completion); - match key_completion with - | None -> () - | Some completed -> - Alcotest.(check int) - "late key completion emits no graph or token work" - 0 - (List.length completed.effects) -;; - -let warm_encrypted_graph_online_recovery_attaches_existing_mirror () = - let selected, scope = selected_graph encrypted_graph in - let mirror = Core.{ graph = encrypted_graph; scope } in - let inspected = - Core.step selected.next (Core.Mirror_inspected (Mirror_available mirror)) - in - let failed = - List.find_map - (function - | Core.Run (Core.Request (ticket, Core.Load_and_unlock_graph_key _)) -> - Some - (Core.step - inspected.next - (Core.Runner_completed - (Core.Completion - (ticket, Error (Core.Effect_failed "cached key unavailable"))))) - | Run _ | Delegate _ | Publish _ -> None) - inspected.effects - |> Option.get - in - let recovery = failed in - let user_key_fetch = - List.find_map - (function - | Core.Run (Core.Request (ticket, Core.Fetch_e2ee_graph_key _)) -> - Some - (Core.step - recovery.next - (Core.Runner_completed (Core.Completion (ticket, Ok "encrypted-graph-key")))) - | Run _ | Delegate _ | Publish _ -> None) - recovery.effects - |> Option.get - in - let password_prompt = - List.find_map - (function - | Core.Run (Core.Request (ticket, Core.Fetch_e2ee_user_keys _)) -> - Some - (Core.step - user_key_fetch.next - (Core.Runner_completed (Core.Completion (ticket, Ok "private-key-package")))) - | Run _ | Delegate _ | Publish _ -> None) - user_key_fetch.effects - |> Option.get - in - let private_key_unlock = - Core.step password_prompt.next (password_event password_prompt.next "password") - in - let graph_key_unlock = - List.find_map - (function - | Core.Run (Core.Request (ticket, Core.Unlock_private_key _)) -> - Some - (Core.step - private_key_unlock.next - (Core.Runner_completed (Core.Completion (ticket, Ok ())))) - | Run _ | Delegate _ | Publish _ -> None) - private_key_unlock.effects - |> Option.get - in - let recovered = - List.find_map - (function - | Core.Run (Core.Request (ticket, Core.Fetch_and_unlock_graph_key request)) -> - let key = Core.graph_key_handle ~id:"online-warm-key" ~scope:request.scope in - Some - (Core.step - graph_key_unlock.next - (Core.Runner_completed (Core.Completion (ticket, Ok key)))) - | Run _ | Delegate _ | Publish _ -> None) - graph_key_unlock.effects - |> Option.get - in - Alcotest.(check bool) - "online key recovery attaches the existing mirror" - true - (List.exists - (function - | Core.Delegate (Core.Attach_graph request) -> request.scope = scope - | Run _ | Delegate _ | Publish _ -> false) - recovered.effects); - Alcotest.(check bool) - "online warm recovery does not replace the existing mirror" - false - (List.exists - (function - | Core.Run (Core.Request (_, Core.Fetch_snapshot_metadata _)) -> true - | Run _ | Delegate _ | Publish _ -> false) - recovered.effects) -;; - -let e2ee_password_retry_recovers_after_failed_unlock () = - let selected, scope = selected_graph encrypted_graph in - let missing = Core.step selected.next (Core.Mirror_inspected (Mirror_absent scope)) in - let failed_cache = - List.find_map - (function - | Core.Run (Core.Request (ticket, Core.Load_and_unlock_graph_key _)) -> - Some - (Core.step - missing.next - (Core.Runner_completed - (Core.Completion (ticket, Error (Core.Effect_failed "missing key"))))) - | Run _ | Delegate _ | Publish _ -> None) - missing.effects - |> Option.get - in - let recovery = failed_cache in - let user_key_fetch = - List.find_map - (function - | Core.Run (Core.Request (ticket, Core.Fetch_e2ee_graph_key _)) -> - Some - (Core.step - recovery.next - (Core.Runner_completed (Core.Completion (ticket, Ok "encrypted-graph-key")))) - | Run _ | Delegate _ | Publish _ -> None) - recovery.effects - |> Option.get - in - let prompt = - List.find_map - (function - | Core.Run (Core.Request (ticket, Core.Fetch_e2ee_user_keys _)) -> - Some - (Core.step - user_key_fetch.next - (Core.Runner_completed (Core.Completion (ticket, Ok "private-key-package")))) - | Run _ | Delegate _ | Publish _ -> None) - user_key_fetch.effects - |> Option.get - in - let first = Core.step prompt.next (password_event prompt.next "wrong-synthetic") in - let rejected, repeat_rejected = - List.find_map - (function - | Core.Run (Core.Request (ticket, Core.Unlock_private_key _)) -> - let event = - Core.Runner_completed - (Core.Completion - ( ticket - , Error (Core.Crypto_failed (Invalid_key_material, "wrong password")) )) - in - Some (Core.step first.next event, event) - | Run _ | Delegate _ | Publish _ -> None) - first.effects - |> Option.get - in - Alcotest.(check bool) - "failure stays visible" - true - ((Core.state rejected.next).snapshot.startup.failure = Some During_e2ee); - let corrected = - Core.step rejected.next (password_event rejected.next " corrected-synthetic ") - in - let requests = - List.filter_map - (function - | Core.Run (Core.Request (ticket, Core.Unlock_private_key request)) -> - Some - ( request - , fun (result : (unit, Core.effect_error) result) -> - Core.Runner_completed (Core.Completion (ticket, result)) ) - | Run _ | Delegate _ | Publish _ -> None) - corrected.effects - in - Alcotest.(check int) - "corrected password emits one unlock request" - 1 - (List.length requests); - let request, complete = List.hd requests in - Alcotest.(check string) - "retry preserves password bytes" - " corrected-synthetic " - request.password; - Alcotest.(check string) - "retry preserves encrypted package" - "private-key-package" - request.private_key_package; - Alcotest.(check bool) - "retry remains in account scope" - true - (request.scope = scope.account); - Alcotest.(check bool) - "failure awaits another password" - true - (Core.state rejected.next).snapshot.startup.awaiting_e2ee_password; - Alcotest.(check bool) - "retry clears previous failure" - true - ((Core.state corrected.next).snapshot.startup.failure = None); - let duplicate = - Core.step corrected.next (password_event corrected.next "duplicate-synthetic") - in - Alcotest.(check int) "pending duplicate is inert" 0 (List.length duplicate.effects); - let stale = Core.step duplicate.next repeat_rejected in - Alcotest.(check int) "old rejection completion is inert" 0 (List.length stale.effects); - let again = - Core.step stale.next (complete (Error (Core.Effect_failed "still wrong"))) - in - Alcotest.(check bool) - "repeated failure remains retryable" - true - (Core.state again.next).snapshot.startup.awaiting_e2ee_password; - let final = Core.step again.next (password_event again.next "final-synthetic") in - let unlocked = - List.find_map - (function - | Core.Run (Core.Request (ticket, Core.Unlock_private_key _)) -> - Some - (Core.step - final.next - (Core.Runner_completed (Core.Completion (ticket, Ok ())))) - | Run _ | Delegate _ | Publish _ -> None) - final.effects - |> Option.get - in - let resumed = - List.find_map - (function - | Core.Run (Core.Request (ticket, Core.Fetch_and_unlock_graph_key request)) -> - Alcotest.(check string) - "retry retains encrypted graph key" - "encrypted-graph-key" - request.encrypted_graph_key; - let handle = Core.graph_key_handle ~id:"retry-graph-key" ~scope:request.scope in - Some - (Core.step - unlocked.next - (Core.Runner_completed (Core.Completion (ticket, Ok handle)))) - | Run _ | Delegate _ | Publish _ -> None) - unlocked.effects - |> Option.get - in - Alcotest.(check bool) - "successful retry resumes snapshot bootstrap" - true - (List.exists - (function - | Core.Run (Core.Request (_, Core.Fetch_snapshot_metadata _)) -> true - | Run _ | Delegate _ | Publish _ -> false) - resumed.effects); - let picker = Core.step rejected.next Core.Graph_picker_requested in - let abandoned = - Core.step picker.next (password_event picker.next "abandoned-synthetic") - in - Alcotest.(check int) - "picker discards password challenge" - 0 - (List.length abandoned.effects); - let account = - Core.step rejected.next (Core.Account_authenticated { user_id = Some "other-user" }) - in - let abandoned = - Core.step account.next (password_event account.next "other-account-synthetic") - in - Alcotest.(check int) - "account switch discards password challenge" - 0 - (List.length abandoned.effects) -;; - -let encrypted_graph_recovery_fetches_and_unlocks_key_before_bootstrap () = - let authenticated = - Core.step (initial ()) (Core.Account_authenticated { user_id = Some "user" }) - in - let catalog = - List.find_map - (function - | Core.Run (Core.Request (ticket, Core.Fetch_catalog _)) -> - Some - (Core.step - authenticated.next - (Core.Runner_completed (Core.Completion (ticket, Ok [ encrypted_graph ])))) - | Run _ | Delegate _ | Publish _ -> None) - authenticated.effects - |> Option.get - in - let selected = Core.step catalog.next (Core.Graph_selected graph_id) in - let scope = Core.admitted_graph_scope selected.next |> Option.get in - let missing = - Core.step selected.next (Core.Mirror_inspected (Core.Mirror_absent scope)) - in - let failed = - List.find_map - (function - | Core.Run (Core.Request (ticket, Core.Load_and_unlock_graph_key _)) -> - Some - (Core.step - missing.next - (Core.Runner_completed - (Core.Completion (ticket, Error (Core.Effect_failed "missing key"))))) - | Run _ | Delegate _ | Publish _ -> None) - missing.effects - |> Option.get - in - let recovery = failed in - let user_key_fetch = - List.find_map - (function - | Core.Run (Core.Request (ticket, Core.Fetch_e2ee_graph_key _)) -> - Some - (Core.step - recovery.next - (Core.Runner_completed (Core.Completion (ticket, Ok "encrypted-graph-key")))) - | Run _ | Delegate _ | Publish _ -> None) - recovery.effects - |> Option.get - in - let password_prompt = - List.find_map - (function - | Core.Run (Core.Request (ticket, Core.Fetch_e2ee_user_keys _)) -> - Some - (Core.step - user_key_fetch.next - (Core.Runner_completed (Core.Completion (ticket, Ok "private-key-package")))) - | Run _ | Delegate _ | Publish _ -> None) - user_key_fetch.effects - |> Option.get - in - Alcotest.(check bool) - "private key package prompts for the E2EE password" - true - (Core.state password_prompt.next).snapshot.startup.awaiting_e2ee_password; - let private_key_unlock = - Core.step password_prompt.next (password_event password_prompt.next "password") - in - let graph_key_unlock = - List.find_map - (function - | Core.Run (Core.Request (ticket, Core.Unlock_private_key request)) -> - Alcotest.(check string) "password is forwarded once" "password" request.password; - Alcotest.(check string) - "private key package is forwarded" - "private-key-package" - request.private_key_package; - Some - (Core.step - private_key_unlock.next - (Core.Runner_completed (Core.Completion (ticket, Ok ())))) - | Run _ | Delegate _ | Publish _ -> None) - private_key_unlock.effects - |> Option.get - in - let bootstrapping = - List.find_map - (function - | Core.Run (Core.Request (ticket, Core.Fetch_and_unlock_graph_key request)) -> - Alcotest.(check string) - "encrypted graph key is forwarded" - "encrypted-graph-key" - request.encrypted_graph_key; - let handle = Core.graph_key_handle ~id:"graph-key" ~scope:request.scope in - Some - (Core.step - graph_key_unlock.next - (Core.Runner_completed (Core.Completion (ticket, Ok handle)))) - | Run _ | Delegate _ | Publish _ -> None) - graph_key_unlock.effects - |> Option.get - in - Alcotest.(check bool) - "bootstrap phase is published before waiting for metadata" - true - (List.exists - (function - | Core.Publish (Core.State_changed state) -> - (match state.snapshot.unlock with Core.Unlock_bootstrapping _ -> true | _ -> false) - | _ -> false) - bootstrapping.effects); - Alcotest.(check bool) - "unlocked graph key resumes snapshot bootstrap" - true - (List.exists - (function - | Core.Run (Core.Request (_, Core.Fetch_snapshot_metadata _)) -> true - | Run _ | Delegate _ | Publish _ -> false) - bootstrapping.effects) -;; - -let begin_snapshot_bootstrap ?(graphs = [ graph ]) () = - let authenticated = - Core.step (initial ()) (Core.Account_authenticated { user_id = Some "user" }) - in - let catalog = - List.find_map - (function - | Core.Run (Core.Request (ticket, Core.Fetch_catalog _)) -> - Some - (Core.step - authenticated.next - (Core.Runner_completed (Core.Completion (ticket, Ok graphs)))) - | Run _ | Delegate _ | Publish _ -> None) - authenticated.effects - |> Option.get - in - let selected = Core.step catalog.next (Core.Graph_selected graph_id) in - let scope = - List.find_map - (function - | Core.Delegate (Core.Inspect_mirror request) -> Some request.scope - | Run _ | Delegate _ | Publish _ -> None) - selected.effects - |> Option.get - in - let missing = - Core.step selected.next (Core.Mirror_inspected (Core.Mirror_absent scope)) - in - missing, scope -;; - -let snapshot_activation_request effects = - List.find_map (function Core.Delegate (Core.Activate_snapshot request) -> Some request - | _ -> None) effects |> Option.get -;; - -let snapshot_activation_fixture () = - let requested, _ = begin_snapshot_bootstrap () in - let downloading = List.find_map (function - | Core.Run (Core.Request (ticket, Core.Fetch_snapshot_metadata _)) -> - Some (Core.step requested.next (Core.Runner_completed (Core.Completion - (ticket, Ok {|{"ok":true,"t":42,"url":"https://example.com/synthetic"}|})))) - | _ -> None) requested.effects |> Option.get in - let activating = List.find_map (function - | Core.Run (Core.Request (ticket, Core.Download_snapshot request)) -> - let artifact = Core.stage_snapshot_download request ~path:"/tmp/synthetic-activation" - ~expected_rows:1 |> Result.get_ok in - Some (Core.step downloading.next (Core.Runner_completed (Core.Completion (ticket, Ok artifact)))) - | _ -> None) downloading.effects |> Option.get in - activating -;; - -let fixture_snapshot_activation = lazy (snapshot_activation_request (snapshot_activation_fixture ()).effects) -;; - -let snapshot_catchup_fixture () = - let requested, _scope = begin_snapshot_bootstrap () in - Alcotest.(check bool) - "bootstrap starts with exactly one metadata request and no preliminary pull" - true - (match List.filter (function Core.Publish _ -> false | _ -> true) requested.effects with - | [ Core.Run (Core.Request (_, Core.Fetch_snapshot_metadata _)) ] -> true - | _ -> false); - let download = - List.find_map - (function - | Core.Run (Core.Request (ticket, Core.Fetch_snapshot_metadata _)) -> - Some - (Core.step - requested.next - (Core.Runner_completed - (Core.Completion - ( ticket - , Ok {|{"ok":true,"t":42,"url":"https://example.com/snapshot.sqlite"}|} )))) - | Run _ | Delegate _ | Publish _ -> None) - requested.effects - |> Option.get - in - let activated = - List.find_map - (function - | Core.Run (Core.Request (ticket, Core.Download_snapshot request)) -> - Alcotest.(check int) - "download retains configured bound" - (1024 * 1024) - (Core.snapshot_download_maximum_bytes request); - let artifact = - Core.stage_snapshot_download request ~path:"/tmp/snapshot.sqlite" ~expected_rows:10 |> Result.get_ok - in - Some - (Core.step - download.next - (Core.Runner_completed (Core.Completion (ticket, Ok artifact)))) - | Run _ | Delegate _ | Publish _ -> None) - download.effects - |> Option.get - in - let activation = - List.find_map - (function - | Core.Delegate (Core.Activate_snapshot request) -> Some request - | Run _ | Delegate _ | Publish _ -> None) - activated.effects - |> Option.get - in - Alcotest.(check cursor_testable) - "activation uses metadata cursor" - (server_cursor_of_int 42) - (Core.snapshot_activation_cursor activation); - Alcotest.(check bool) - "plain snapshot has no graph key" - true - (Option.is_none (Core.snapshot_activation_key activation)); - Alcotest.(check (option cursor_testable)) - "download completion cannot publish an uncommitted cursor" - None (Core.state activated.next).snapshot.applied_server_t; - let activated_snapshot = Core.step activated.next (Core.Snapshot_activated (Core.snapshot_activation_completed activation)) in - let mirror = inspect_mirror_request activated_snapshot.effects in - let attaching = - Core.step - activated_snapshot.next - (Core.Mirror_inspected (Core.Mirror_available mirror)) - in - let attachment = - List.find_map - (function - | Core.Delegate (Core.Attach_graph request) -> Some request - | Run _ | Delegate _ | Publish _ -> None) - attaching.effects - |> Option.get - in - let sync_token = - Logseq_overlay_db.Types.sync_token_of_string "sync-token:v1:cold-start" - |> Result.get_ok - in - let checkpoint = - Logseq_overlay_db.Types.Server_cursor.of_int64 42L - |> Result.map_error (fun `Negative_cursor -> "negative cursor") - |> Result.get_ok - in - let sync = - Logseq_overlay_db.Types.sync_view ~token:sync_token ~checkpoint ~submissions:[] - in - let attached = - Core.step attaching.next (Core.Graph_attached { scope = attachment.scope; sync }) - in - let connection = - List.find_map - (function - | Core.Run (Core.Start_websocket request) -> Some request.scope - | Run _ | Delegate _ | Publish _ -> None) - attached.effects - |> Option.get - in - let pulling = Core.step attached.next (Core.Websocket_opened connection) in - Alcotest.(check bool) - "cold start opens a pull without another lifecycle event" - true - (List.exists - (function - | Core.Run - (Core.Send_websocket - { scope; message = Logseq_sync_pure_reducer.Sync_protocol.Client.Pull _ }) - -> scope = connection - | Run _ | Delegate _ | Publish _ -> false) - pulling.effects); - pulling, connection, sync -;; - -let cold_start_absent_mirror_reaches_pull_after_snapshot_activation () = - ignore (snapshot_catchup_fixture ()) -;; - -let snapshot_baseline_decoder_retains_http_budget () = - (* Ordinary cursor values and small valid Transit transactions form a valid - HTTP pull response larger than a WebSocket frame, within the endpoint budget. *) - let transaction = - let module Transit = Transit_core.Json in - Transit_native.Transit.Json.to_string - (Transit.Array - [ Transit.Array - [ Transit.Keyword "db/add" - ; Transit.Int 1 - ; Transit.Keyword "block/title" - ; Transit.String (String.make 16_384 'x') - ] - ]) - in - let body = - Yojson.Safe.to_string - (`Assoc - [ "type", `String "pull/ok" - ; "t", `Int 32 - ; ( "txs" - , `List - (List.init 32 (fun index -> - `Assoc [ "t", `Int (index + 1); "tx", `String transaction ])) ) - ]) - in - Alcotest.(check bool) - (Printf.sprintf - "ordinary %d-byte HTTP fixture exceeds only the WebSocket budget" - (String.length body)) - true - (String.length body > Logseq_db_types.Limits.maximum_response_bytes - && String.length body <= 64 * 1024 * 1024); - Alcotest.(check bool) - "legacy HTTP baseline decoder retains its endpoint budget" - true - (Result.is_ok (Logseq_sync_pure_reducer.Sync_protocol.decode_snapshot_baseline body)) -;; - -let invalid_snapshot_metadata_fails_bootstrap () = - let requested, _scope = begin_snapshot_bootstrap () in - let failed = - List.find_map - (function - | Core.Run (Core.Request (ticket, Core.Fetch_snapshot_metadata _)) -> - Some - (Core.step - requested.next - (Core.Runner_completed - (Core.Completion (ticket, Ok {|{"ok":true,"t":-1,"url":"https://example.com/snapshot"}|})))) - | Run _ | Delegate _ | Publish _ -> None) - requested.effects - |> Option.get - in - let snapshot = (Core.state failed.next).snapshot in - Alcotest.(check bool) - "invalid metadata cursor enters failed state" - true - (snapshot.sync_phase = Core.Failed - && snapshot.startup.failure = Some Core.During_bootstrap) -;; - -module Overlay = Logseq_overlay_db.Types -module Protocol = Logseq_sync_pure_reducer.Sync_protocol - -let token module_of_string value = module_of_string value |> Result.get_ok - -let mutation_ids = - [ "22222222-2222-4222-8222-222222222221" - ; "22222222-2222-4222-8222-222222222222" - ; "22222222-2222-4222-8222-222222222223" - ] - |> List.map (fun value -> - Logseq_db_types.Graph_types.Uuid.of_string value |> Result.get_ok) -;; - -let submitted_core () = - let authenticated = - Core.step (initial ()) (Core.Account_authenticated { user_id = Some "user" }) - in - let catalog = - List.find_map - (function - | Core.Run (Core.Request (ticket, Core.Fetch_catalog _)) -> - Some - (Core.step - authenticated.next - (Core.Runner_completed (Core.Completion (ticket, Ok [ graph ])))) - | Run _ | Delegate _ | Publish _ -> None) - authenticated.effects - |> Option.get - in - let selected = Core.step catalog.next (Core.Graph_selected graph_id) in - let mirror_request = - List.find_map - (function - | Core.Delegate (Core.Inspect_mirror request) -> Some request - | Run _ | Delegate _ | Publish _ -> None) - selected.effects - |> Option.get - in - let inspected = - Core.step selected.next (Core.Mirror_inspected (Core.Mirror_available mirror_request)) - in - let attach_request = - List.find_map - (function - | Core.Delegate (Core.Attach_graph request) -> Some request - | Run _ | Delegate _ | Publish _ -> None) - inspected.effects - |> Option.get - in - let scope = attach_request.scope in - let sync_token = Overlay.sync_token_of_string "sync-token:v1:1" |> Result.get_ok in - let checkpoint = - Overlay.Server_cursor.of_int64 0L - |> Result.map_error (fun `Negative_cursor -> "negative cursor") - |> Result.get_ok - in - let sync = Overlay.sync_view ~token:sync_token ~checkpoint ~submissions:[] in - let attached = Core.step inspected.next (Core.Graph_attached { scope; sync }) in - let connection = - List.find_map - (function - | Core.Run (Core.Start_websocket request) -> Some request.scope - | Run _ | Delegate _ | Publish _ -> None) - attached.effects - |> Option.get - in - let opened = Core.step attached.next (Core.Websocket_opened connection) in - let batch_id = - Overlay.Submission_batch_id.of_string "submission-batch:v1:test" |> Result.get_ok - in - let wires = - List.map - (fun mutation_id -> - Overlay.submission_wire - ~maximum_bytes:1024 - ~mutation_id - ~operation:Overlay.Save_block_operation - ~protected_transaction:"protected" - |> Result.get_ok) - mutation_ids - in - let batch = - Overlay.submission_batch - ~maximum_wires:3 - ~maximum_bytes:4096 - ~batch_id - ~t_before:checkpoint - ~wires - |> Result.get_ok - in - let queued_submissions = - List.map2 - (fun mutation_id wire -> - { Overlay.mutation_id - ; fingerprint = - Overlay.Mutation_fingerprint.of_string - ("mutation-fingerprint:v1:" - ^ Logseq_db_types.Graph_types.Uuid.to_string mutation_id) - |> Result.get_ok - ; state = Overlay.Queued - ; operation = Overlay.Save_block_operation - ; dependency_eligible = true - ; attempt_count = 0 - ; plaintext_bytes = 16 - ; protected_bytes = Some (Overlay.submission_wire_byte_length wire) - }) - mutation_ids - wires - in - let queued_sync = - Overlay.sync_view ~token:sync_token ~checkpoint ~submissions:queued_submissions - in - let inspect = Core.step opened.next Core.Local_outbox_changed in - let inspected_scope = - List.find_map - (function - | Core.Delegate (Core.Inspect_sync requested_scope) -> Some requested_scope - | Run _ | Delegate _ | Publish _ -> None) - inspect.effects - |> Option.get - in - let planned = - Core.step - inspect.next - (Core.Sync_inspected { scope = inspected_scope; sync = queued_sync }) - in - let transition_request = - List.find_map - (function - | Core.Delegate (Core.Apply_outbox_transition request) -> Some request - | Run _ | Delegate _ | Publish _ -> None) - planned.effects - |> Option.get - in - let generation = Overlay.Generation.of_string "generation:v1:1" |> Result.get_ok in - let projection = - Overlay.Projection_revision.of_string "projection:v1:1" |> Result.get_ok - in - let commit : Overlay.outbox_commit = - { generation - ; before_projection_revision = projection - ; after_projection_revision = projection - ; sync_token - ; transition = transition_request.transition - ; activity = Overlay.Logically_active - ; logical_change_summary = Overlay.No_logical_change - ; submission_batch = Some batch - } - in - let submitted = - Core.step - planned.next - (Core.Outbox_transition_applied { scope = transition_request.scope; commit; sync }) - in - submitted.next, connection -;; - -let rejection_transition effects = - List.find_map - (function - | Core.Delegate - (Core.Apply_outbox_transition - { transition = Overlay.Reject_group { resolution; _ }; _ }) -> - Some resolution - | Run _ | Delegate _ | Publish _ -> None) - effects - |> Option.get -;; - -let partial_rejection_is_normalized_exactly () = - let core, connection = submitted_core () in - let first, second, third = - match mutation_ids with - | [ first; second; third ] -> first, second, third - | _ -> assert false - in - let rejection : Logseq_sync_pure_reducer.Sync_protocol.rejection = - { reason = Db_transact_failed - ; t = Some (server_cursor_of_int 1) - ; checksum = Some "0123456789abcdef" - ; success_tx_ids = [ first ] - ; failed_tx_id = Some second - ; missing_block_uuids = [ graph_id ] - ; error_detail = Some "cannot store value as expected UUID type" - ; data = None - } - in - let rejected = - Core.step - core - (Core.Websocket_message - (connection, Logseq_sync_pure_reducer.Sync_protocol.Server.Tx_reject rejection)) - in - match rejection_transition rejected.effects with - | Overlay.Definitive { reason = Missing_dependencies; partition } -> - Alcotest.(check bool) - "accepted prefix retained" - true - (partition.accepted_prefix = [ first ]); - Alcotest.(check bool) - "failed member retained" - true - (partition.failed_member = Some second); - Alcotest.(check bool) "suffix inferred" true (partition.unexecuted_suffix = [ third ]); - Alcotest.(check bool) - "prefix barrier required" - true - (Option.is_some partition.acceptance_barrier) - | _ -> Alcotest.fail "partial rejection was not normalized as missing dependencies" -;; - -let local_outbox_change_requests_sync_inspection () = - let core, _connection = submitted_core () in - let transition = Core.step core Core.Local_outbox_changed in - Alcotest.(check bool) - "local outbox change delegates one fresh sync inspection" - true - (List.exists - (function - | Core.Delegate (Core.Inspect_sync _) -> true - | Run _ | Delegate _ | Publish _ -> false) - transition.effects) -;; - -let accepted_submission_pulls_its_authoritative_transaction () = - let core, connection = submitted_core () in - (* Complete the opening pull before verifying a fresh accepted-submission pull. *) - let core = - (Core.step - core - (Core.Websocket_message - ( connection - , Protocol.Server.Pull_ok - { t = server_cursor_of_int 0; checksum = None; txs = [] } ))) - .next - in - let acknowledged = - Core.step - core - (Core.Websocket_message - ( connection - , Logseq_sync_pure_reducer.Sync_protocol.Server.Tx_batch_ok - { t = server_cursor_of_int 1; checksum = Some "0123456789abcdef" } )) - in - let request = - List.find_map - (function - | Core.Delegate - (Core.Apply_outbox_transition - ({ transition = Overlay.Accept_group _; _ } as request)) -> Some request - | Run _ | Delegate _ | Publish _ -> None) - acknowledged.effects - |> Option.get - in - let sync_token = Overlay.sync_token_of_string "sync-token:v1:2" |> Result.get_ok in - let checkpoint = - Overlay.Server_cursor.of_int64 0L - |> Result.map_error (fun `Negative_cursor -> "negative cursor") - |> Result.get_ok - in - let sync = Overlay.sync_view ~token:sync_token ~checkpoint ~submissions:[] in - let generation = Overlay.Generation.of_string "generation:v1:2" |> Result.get_ok in - let projection = - Overlay.Projection_revision.of_string "projection:v1:1" |> Result.get_ok - in - let commit : Overlay.outbox_commit = - { generation - ; before_projection_revision = projection - ; after_projection_revision = projection - ; sync_token - ; transition = request.transition - ; activity = Overlay.Logically_inactive - ; logical_change_summary = Overlay.No_logical_change - ; submission_batch = None - } - in - let completed = - Core.step - acknowledged.next - (Core.Outbox_transition_applied { scope = request.scope; commit; sync }) - in - Alcotest.(check bool) - "accepted submission pulls from the durable cursor" - true - (List.exists - (function - | Core.Run - (Core.Send_websocket - { message = Logseq_sync_pure_reducer.Sync_protocol.Client.Pull { since } - ; _ - }) -> since = Some (server_cursor_of_int 0) - | Run _ | Delegate _ | Publish _ -> false) - completed.effects) -;; - -let whole_batch_invalid_tx_partitions_every_member () = - let core, connection = submitted_core () in - let rejection : Logseq_sync_pure_reducer.Sync_protocol.rejection = - { reason = Invalid_tx - ; t = None - ; checksum = None - ; success_tx_ids = [] - ; failed_tx_id = None - ; missing_block_uuids = [] - ; error_detail = None - ; data = None - } - in - let rejected = - Core.step - core - (Core.Websocket_message - (connection, Logseq_sync_pure_reducer.Sync_protocol.Server.Tx_reject rejection)) - in - match rejection_transition rejected.effects, mutation_ids with - | ( Overlay.Definitive - { reason = Invalid_request - ; partition = - { accepted_prefix = []; failed_member = Some failed; unexecuted_suffix; _ } - } - , expected_failed :: expected_suffix ) -> - Alcotest.(check bool) - "first member failed" - true - (Logseq_db_types.Graph_types.Uuid.equal failed expected_failed); - Alcotest.(check bool) - "remaining members are unexecuted" - true - (unexecuted_suffix = expected_suffix) - | _ -> Alcotest.fail "invalid tx did not produce a complete batch partition" -;; - -let canonical_overlay_happy_path () = - let cursor_diagnostics local remote : Core.diagnostics = - { groups = [ { Core.title = "Sync transactions" - ; entries = [ "Local tx", local; "Remote tx", remote ] } ] } - in - let origin = initial () in - let initial_state = Core.state origin in - let authenticated_startup = - { initial_state.snapshot.startup with - authenticated = true - ; local_access = Core.Local_pending - ; network_phase = Core.Online_by_user - ; catalog_loading = true - ; account_generation = 1 - ; presentation_generation = 1 - } - in - let authenticated_state = - { initial_state with - snapshot = { initial_state.snapshot with startup = authenticated_startup } - } - in - let authenticated_observed = - { state = authenticated_state; admitted_graph_scope = None } - in - let hp01_event = Core.Account_authenticated { user_id = Some "user" } in - let hp01_preview = preview_step "HP01" origin hp01_event in - let (catalog_ticket, catalog_scope) - : Core.graph list Core.effect_ticket * Core.account_scope - = - match hp01_preview.effects with - | [ Core.Publish (Core.State_changed state) - ; Core.Run (Core.Request (ticket, Core.Fetch_catalog scope)) - ] - when state = authenticated_state && scope.user_id = "user" -> ticket, scope - | _ -> Alcotest.fail "HP01 unexpected instruction shape" - in - let account_scope : Core.account_scope = - { managed_sync_origin = Uri.of_string "https://api.logseq.io" - ; user_id = "user" - ; account_generation = 1 - ; presentation_generation = 1 - ; lifecycle_generation = 0L - } - in - Alcotest.(check bool) "HP01 full account scope" true (catalog_scope = account_scope); - let hp01 = - check_step "HP01" origin hp01_event authenticated_observed hp01_preview.effects - in - let hp02 = hp01 in - let catalogued_startup = - { authenticated_startup with catalog_loading = false; awaiting_selection = true } - in - let catalogued_state = - { authenticated_state with - snapshot = - { authenticated_state.snapshot with - catalog = [ graph ] - ; startup = catalogued_startup - } - } - in - let catalogued_observed = { state = catalogued_state; admitted_graph_scope = None } in - let hp03_event = - Core.Runner_completed (Core.Completion (catalog_ticket, Ok [ graph ])) - in - let hp03_preview = preview_step "HP03" hp02.next hp03_event in - let save_catalog_effect = - match hp03_preview.effects with - | [ Core.Publish (Core.State_changed state) - ; (Core.Run (Core.Request (_, Core.Save_catalog { account; cache })) as save_effect) - ] - when state = catalogued_state - && account = account_scope - && Core.catalog_cache_user_id cache = "user" - && Core.catalog_cache_graphs cache = [ graph ] - && Core.catalog_cache_selected_graph cache = None -> save_effect - | _ -> Alcotest.fail "HP03 unexpected instruction shape" - in - let hp03 = - check_step - "HP03" - hp02.next - hp03_event - catalogued_observed - [ Core.Publish (Core.State_changed catalogued_state); save_catalog_effect ] - in - let graph_scope : Core.graph_scope = - { account = {account_scope with presentation_generation=2}; graph_id; graph_generation = 1 } - in - let selected_startup = - { catalogued_startup with - awaiting_selection = false - ; restoring_local = true - ; graph_generation = 1 - ; presentation_generation = 2 - } - in - let selected_state = - { Core.diagnostics = cursor_diagnostics "Not available" "Not available" - ; snapshot = - { catalogued_state.snapshot with - selected_graph = Some graph_id - ; startup = selected_startup - } - } - in - let selected_observed = - { state = selected_state; admitted_graph_scope = Some graph_scope } - in - let hp04_event = Core.Graph_selected graph_id in - let hp04_preview = preview_step "HP04" hp03.next hp04_event in - let mirror_request, selected_save_effect = - match hp04_preview.effects with - | [ Core.Delegate (Core.Inspect_mirror request) - ; Core.Publish (Core.State_changed state) - ; (Core.Run (Core.Request (_, Core.Save_catalog { account; cache })) as save_effect) - ] - when (request = Core.{ graph; scope = graph_scope }) - && state = selected_state - && account = graph_scope.account - && Core.catalog_cache_user_id cache = "user" - && Core.catalog_cache_graphs cache = [ graph ] - && Core.catalog_cache_selected_graph cache = Some graph_id -> - request, save_effect - | _ -> Alcotest.fail "HP04 unexpected instruction shape" - in - let hp04 = - check_step - "HP04" - hp03.next - hp04_event - selected_observed - [ Core.Delegate (Core.Inspect_mirror mirror_request) - ; Core.Publish (Core.State_changed selected_state) - ; selected_save_effect - ] - in - let hp05_event = Core.Mirror_inspected (Core.Mirror_available mirror_request) in - let hp05 = - check_step - "HP05" - hp04.next - hp05_event - selected_observed - [ Core.Delegate (Core.Attach_graph mirror_request) ] - in - let cursor0 = token Overlay.Server_cursor.of_int64 0L in - let sync_token0 = token Overlay.sync_token_of_string "sync-token:v1:hp-0" in - let empty_sync0 = - Overlay.sync_view ~token:sync_token0 ~checkpoint:cursor0 ~submissions:[] - in - let attached_startup = - { selected_startup with restoring_local = false; bootstrapping = false; local_access=Core.Local_available } - in - let attached_state = - { Core.diagnostics = cursor_diagnostics "0" "Not available" - ; snapshot = - { selected_state.snapshot with - sync_phase = Core.Current - ; applied_server_t = Some (server_cursor_of_int 0) - ; timeline_presentation_pending = true - ; startup = attached_startup - } - } - in - let hp06_event = - Core.Graph_attached { scope = mirror_request.scope; sync = empty_sync0 } - in - let connecting_state = - { attached_state with - snapshot = { attached_state.snapshot with sync_phase = Core.Connecting } - } - in - let connecting_observed = - { state = connecting_state; admitted_graph_scope = Some graph_scope } - in - let connection : Core.connection_scope = - { graph = graph_scope; connection_generation = 1 } - in - let websocket_request : Core.websocket_request = - { scope = connection - ; uri = Uri.of_string "wss://api.logseq.io/sync/11111111-1111-4111-8111-111111111111" - } - in - let hp06_preview = preview_step "HP06" hp05.next hp06_event in - (match hp06_preview.effects with - | [ Core.Publish (Core.State_changed state); Core.Run (Core.Start_websocket request) ] - when state = connecting_state && request = websocket_request -> () - | _ -> Alcotest.fail "HP06 unexpected instruction shape"); - let hp06 = - check_step "HP06" hp05.next hp06_event connecting_observed hp06_preview.effects - in - let hp07 = hp06 in - let pulling0_state = - { connecting_state with - snapshot = { connecting_state.snapshot with sync_phase = Core.Pulling } - } - in - let pulling0_observed = - { state = pulling0_state; admitted_graph_scope = Some graph_scope } - in - let hp08_event = Core.Websocket_opened connection in - let hp08_preview = preview_step "HP08" hp07.next hp08_event in - (match hp08_preview.effects with - | [ Core.Publish (Core.State_changed state) - ; Core.Run (Core.Send_websocket { scope; message }) - ; Core.Run (Core.Schedule_timer timer) - ] - when timer.delay_seconds = 30. - && timer.scope.connection_generation = Some connection.connection_generation - && state = pulling0_state - && scope = connection - && message = Protocol.Client.Pull { since = Some (server_cursor_of_int 0) } -> - () - | _ -> Alcotest.fail "HP08 unexpected instruction shape"); - let hp08 = - check_step "HP08" hp07.next hp08_event pulling0_observed hp08_preview.effects - in - let remote_tx : Protocol.Server.pull_transaction = - { t = server_cursor_of_int 1; tx = "[]"; outliner_op = Some "save-block" } - in - let opening_message = - Protocol.Server.Pull_ok - { t = server_cursor_of_int 1 - ; checksum = Some "0123456789abcdef" - ; txs = [ remote_tx ] - } - in - let hp09_event = Core.Websocket_message (connection, opening_message) in - let hp09_preview = preview_step "HP09" hp08.next hp09_event in - let pulling_remote1_state = - { pulling0_state with diagnostics = cursor_diagnostics "0" "1" } - in - let opening_batch = - match hp09_preview.effects with - | [ Core.Delegate (Core.Apply_authoritative_batch request) - ; Core.Publish (Core.State_changed state) ] - when request.scope = connection - && state = pulling_remote1_state - && request.presentation_generation = 2 - && request.lifecycle_generation = 0L - && request.key = None - && Overlay.Server_cursor.to_int64 - (Overlay.authoritative_batch_through request.input) - = 1L - && List.length (Overlay.authoritative_batch_transactions request.input) = 1 -> - request - | _ -> Alcotest.fail "HP09 unexpected instruction shape" - in - let hp09 = - check_step - "HP09" - hp08.next - hp09_event - { pulling0_observed with state = pulling_remote1_state } - [ Core.Delegate (Core.Apply_authoritative_batch opening_batch) - ; Core.Publish (Core.State_changed pulling_remote1_state) ] - in - let generation1 = token Overlay.Generation.of_string "generation:v1:hp-1" in - let projection1 = token Overlay.Projection_revision.of_string "projection:v1:1" in - let cursor1 = token Overlay.Server_cursor.of_int64 1L in - let sync_token1 = token Overlay.sync_token_of_string "sync-token:v1:hp-1" in - let sync1 = Overlay.sync_view ~token:sync_token1 ~checkpoint:cursor1 ~submissions:[] in - let authoritative_commit1 : Overlay.authoritative_commit = - { generation = generation1 - ; before_projection_revision = projection1 - ; after_projection_revision = projection1 - ; checkpoint = cursor1 - ; sync_token = sync_token1 - ; terminal_receipts = [] - ; replanned_queued_ids = [] - ; blocked_ids = [] - ; logical_change_summary = Overlay.No_logical_change - } - in - let current1_state = - { Core.diagnostics = cursor_diagnostics "1" "1" - ; snapshot = - { pulling0_state.snapshot with - sync_phase = Core.Current - ; applied_server_t = Some (server_cursor_of_int 1) - } - } - in - let current1_observed = - { state = current1_state; admitted_graph_scope = Some graph_scope } - in - let hp10_event = - Core.Authoritative_batch_applied - { scope = opening_batch.scope.graph; commit = authoritative_commit1; sync = sync1 } - in - let hp10 = - check_step - "HP10" - hp09.next - hp10_event - current1_observed - [ Core.Publish (Core.State_changed current1_state) ] - in - let hp11_event = Core.Local_outbox_changed in - let hp11 = - check_step - "HP11" - hp10.next - hp11_event - current1_observed - [ Core.Delegate (Core.Inspect_sync graph_scope) ] - in - let mutation_id = List.hd mutation_ids in - let fingerprint = - token Overlay.Mutation_fingerprint.of_string "mutation-fingerprint:v1:hp" - in - let queued : Overlay.submission_descriptor = - { mutation_id - ; fingerprint - ; state = Overlay.Queued - ; operation = Overlay.Save_block_operation - ; dependency_eligible = true - ; attempt_count = 0 - ; plaintext_bytes = 16 - ; protected_bytes = Some 16 - } - in - let queued_sync = - Overlay.sync_view ~token:sync_token1 ~checkpoint:cursor1 ~submissions:[ queued ] - in - let submitting_state = - { current1_state with - snapshot = { current1_state.snapshot with sync_phase = Core.Submitting } - } - in - let submitting_observed = - { state = submitting_state; admitted_graph_scope = Some graph_scope } - in - let hp12_event = Core.Sync_inspected { scope = graph_scope; sync = queued_sync } in - let hp12_preview = preview_step "HP12" hp11.next hp12_event in - let submit_request = - match hp12_preview.effects with - | [ Core.Publish (Core.State_changed state) - ; Core.Delegate (Core.Apply_outbox_transition request) - ] - when state = submitting_state - && request.scope = graph_scope - && request.key = None - && Overlay.sync_token_equal request.expected sync_token1 - && request.transition = Overlay.Submit_group [ mutation_id ] -> request - | _ -> Alcotest.fail "HP12 unexpected instruction shape" - in - let hp12 = - check_step - "HP12" - hp11.next - hp12_event - submitting_observed - [ Core.Publish (Core.State_changed submitting_state) - ; Core.Delegate (Core.Apply_outbox_transition submit_request) - ] - in - let batch_id = token Overlay.Submission_batch_id.of_string "submission-batch:v1:hp" in - let wire = - Overlay.submission_wire - ~maximum_bytes:1024 - ~mutation_id - ~operation:Overlay.Save_block_operation - ~protected_transaction:"protected-hp-transaction" - |> Result.get_ok - in - let submission_batch = - Overlay.submission_batch - ~maximum_wires:1 - ~maximum_bytes:4096 - ~batch_id - ~t_before:cursor1 - ~wires:[ wire ] - |> Result.get_ok - in - let sync_token2 = token Overlay.sync_token_of_string "sync-token:v1:hp-2" in - let submitted_descriptor = { queued with state = Overlay.Submitted batch_id } in - let submitted_sync = - Overlay.sync_view - ~token:sync_token2 - ~checkpoint:cursor1 - ~submissions:[ submitted_descriptor ] - in - let submit_commit : Overlay.outbox_commit = - { generation = generation1 - ; before_projection_revision = projection1 - ; after_projection_revision = projection1 - ; sync_token = sync_token2 - ; transition = submit_request.transition - ; activity = Overlay.Logically_active - ; logical_change_summary = Overlay.No_logical_change - ; submission_batch = Some submission_batch - } - in - let tx_message = - Protocol.Client.Tx_batch - { client_revision = Some (Overlay.Submission_batch_id.to_string batch_id) - ; t_before = server_cursor_of_int 1 - ; txs = - [ { Protocol.Client.tx = "protected-hp-transaction" - ; tx_id = Some mutation_id - ; outliner_op = Some "save-block" - } - ] - } - in - let hp13_event = - Core.Outbox_transition_applied - { scope = submit_request.scope; commit = submit_commit; sync = submitted_sync } - in - let hp13_preview = preview_step "HP13" hp12.next hp13_event in - let hp13 = - (match hp13_preview.effects with - | [ Core.Run (Core.Send_websocket { scope; message }) - ; Core.Run (Core.Schedule_timer timer) - ] - when scope = connection - && message = tx_message - && timer.delay_seconds = 30. - && timer.scope.connection_generation = Some connection.connection_generation - -> () - | _ -> Alcotest.fail "HP13 submission omitted its scoped response timeout"); - check_step "HP13" hp12.next hp13_event submitting_observed hp13_preview.effects - in - let hp14_event = - Core.Websocket_message - ( connection - , Protocol.Server.Tx_batch_ok - { t = server_cursor_of_int 2; checksum = Some "fedcba9876543210" } ) - in - let hp14_preview = preview_step "HP14" hp13.next hp14_event in - let acknowledged_state = - { submitting_state with diagnostics = cursor_diagnostics "1" "2" } - in - let accept_request = - match hp14_preview.effects with - | [ Core.Delegate (Core.Apply_outbox_transition request) - ; Core.Publish (Core.State_changed state) ] - when request.scope = graph_scope - && state = acknowledged_state - && request.key = None - && Overlay.sync_token_equal request.expected sync_token2 - && request.transition - = Overlay.Accept_group - { batch_id - ; barrier = - { through = token Overlay.Server_cursor.of_int64 2L - ; checksum = - token Overlay.Checksum.of_string "checksum:v1:fedcba9876543210" - } - } -> request - | _ -> Alcotest.fail "HP14 unexpected instruction shape" - in - let hp14 = - check_step - "HP14" - hp13.next - hp14_event - { submitting_observed with state = acknowledged_state } - [ Core.Delegate (Core.Apply_outbox_transition accept_request) - ; Core.Publish (Core.State_changed acknowledged_state) ] - in - let sync_token3 = token Overlay.sync_token_of_string "sync-token:v1:hp-3" in - let accepted_descriptor = - { queued with state = Overlay.Accepted_pending_authoritative batch_id } - in - let accepted_sync = - Overlay.sync_view - ~token:sync_token3 - ~checkpoint:cursor1 - ~submissions:[ accepted_descriptor ] - in - let accept_commit : Overlay.outbox_commit = - { generation = generation1 - ; before_projection_revision = projection1 - ; after_projection_revision = projection1 - ; sync_token = sync_token3 - ; transition = accept_request.transition - ; activity = Overlay.Logically_active - ; logical_change_summary = Overlay.No_logical_change - ; submission_batch = None - } - in - let pulling1_state = - { Core.diagnostics = cursor_diagnostics "1" "2" - ; snapshot = { submitting_state.snapshot with sync_phase = Core.Pulling } - } - in - let pulling1_observed = - { state = pulling1_state; admitted_graph_scope = Some graph_scope } - in - let hp15_event = - Core.Outbox_transition_applied - { scope = accept_request.scope; commit = accept_commit; sync = accepted_sync } - in - let hp15_preview = preview_step "HP15" hp14.next hp15_event in - (match hp15_preview.effects with - | [ Core.Publish (Core.State_changed state) - ; Core.Run (Core.Send_websocket { scope; message }) - ; Core.Run (Core.Schedule_timer timer) - ] - when timer.delay_seconds = 30. - && timer.scope.connection_generation = Some connection.connection_generation - && state = pulling1_state - && scope = connection - && message = Protocol.Client.Pull { since = Some (server_cursor_of_int 1) } -> - () - | _ -> Alcotest.fail "HP15 unexpected instruction shape"); - let hp15 = - check_step "HP15" hp14.next hp15_event pulling1_observed hp15_preview.effects - in - let incorporated_tx : Protocol.Server.pull_transaction = - { t = server_cursor_of_int 2; tx = "[]"; outliner_op = Some "save-block" } - in - let confirmation_message = - Protocol.Server.Pull_ok - { t = server_cursor_of_int 2 - ; checksum = Some "fedcba9876543210" - ; txs = [ incorporated_tx ] - } - in - let hp16_event = Core.Websocket_message (connection, confirmation_message) in - let hp16_preview = preview_step "HP16" hp15.next hp16_event in - let confirmation_batch = - match hp16_preview.effects with - | [ Core.Delegate (Core.Apply_authoritative_batch request) ] - when request.scope = connection - && request.presentation_generation = 2 - && request.lifecycle_generation = 0L - && request.key = None - && Overlay.Server_cursor.to_int64 - (Overlay.authoritative_batch_through request.input) - = 2L - && List.length (Overlay.authoritative_batch_transactions request.input) = 1 -> - request - | _ -> Alcotest.fail "HP16 unexpected instruction shape" - in - let hp16 = - check_step - "HP16" - hp15.next - hp16_event - pulling1_observed - [ Core.Delegate (Core.Apply_authoritative_batch confirmation_batch) ] - in - let cursor2 = token Overlay.Server_cursor.of_int64 2L in - let sync_token4 = token Overlay.sync_token_of_string "sync-token:v1:hp-4" in - let sync2 = Overlay.sync_view ~token:sync_token4 ~checkpoint:cursor2 ~submissions:[] in - let authoritative_commit2 : Overlay.authoritative_commit = - { generation = generation1 - ; before_projection_revision = projection1 - ; after_projection_revision = projection1 - ; checkpoint = cursor2 - ; sync_token = sync_token4 - ; terminal_receipts = [] - ; replanned_queued_ids = [] - ; blocked_ids = [] - ; logical_change_summary = Overlay.No_logical_change - } - in - let current2_state = - { Core.diagnostics = cursor_diagnostics "2" "2" - ; snapshot = - { pulling1_state.snapshot with - sync_phase = Core.Current - ; applied_server_t = Some (server_cursor_of_int 2) - } - } - in - let current2_observed = - { state = current2_state; admitted_graph_scope = Some graph_scope } - in - let hp17_event = - Core.Authoritative_batch_applied - { scope = confirmation_batch.scope.graph - ; commit = authoritative_commit2 - ; sync = sync2 - } - in - ignore - (check_step - "HP17" - hp16.next - hp17_event - current2_observed - [ Core.Publish (Core.State_changed current2_state) ]) -;; - -let config_is_bounded () = - Alcotest.check - Alcotest.bool - "zero response bound is rejected" - true - (Result.is_error - (Core.limits - ~maximum_response_bytes:0 - ~maximum_artifact_bytes:1 - ~submission_batch_size:1)) -;; - -let authentication_delegates_catalog_transport () = - let authenticated = - Core.step (initial ()) (Core.Account_authenticated { user_id = Some "user" }) - in - let requested = - List.exists - (function - | Core.Run (Core.Request (_, Core.Fetch_catalog _)) -> true - | Run _ | Delegate _ | Publish _ -> false) - authenticated.effects - in - Alcotest.check Alcotest.bool "catalog request is transport-owned" true requested -;; - -let websocket_connect_uses_deployed_graph_path () = - let authenticated = - Core.step (initial ()) (Core.Account_authenticated { user_id = Some "user" }) - in - let catalog = - List.find_map - (function - | Core.Run (Core.Request (ticket, Core.Fetch_catalog _)) -> - Some - (Core.step - authenticated.next - (Core.Runner_completed (Core.Completion (ticket, Ok [ graph ])))) - | Run _ | Delegate _ | Publish _ -> None) - authenticated.effects - |> Option.get - in - let selected = Core.step catalog.next (Core.Graph_selected graph_id) in - let mirror_request = - List.find_map - (function - | Core.Delegate (Core.Inspect_mirror request) -> Some request - | Run _ | Delegate _ | Publish _ -> None) - selected.effects - |> Option.get - in - let inspected = - Core.step selected.next (Core.Mirror_inspected (Core.Mirror_available mirror_request)) - in - let attach_request = - List.find_map - (function - | Core.Delegate (Core.Attach_graph request) -> Some request - | Run _ | Delegate _ | Publish _ -> None) - inspected.effects - |> Option.get - in - let scope = attach_request.scope in - let sync_token = Overlay.sync_token_of_string "sync-token:v1:1" |> Result.get_ok in - let checkpoint = - Overlay.Server_cursor.of_int64 0L - |> Result.map_error (fun `Negative_cursor -> "negative cursor") - |> Result.get_ok - in - let sync = Overlay.sync_view ~token:sync_token ~checkpoint ~submissions:[] in - let attached = Core.step inspected.next (Core.Graph_attached { scope; sync }) in - let uri = - List.find_map - (function - | Core.Run (Core.Start_websocket request) -> Some request.uri - | Run _ | Delegate _ | Publish _ -> None) - attached.effects - |> Option.get - in - Alcotest.(check string) - "deployed WebSocket endpoint" - ("wss://api.logseq.io/sync/" ^ Logseq_db_types.Graph_types.Uuid.to_string graph_id) - (Uri.to_string uri) -;; - -let authenticated_fetch user_id - : Core.transition * Core.graph list Core.effect_ticket * Core.account_scope - = - let authenticated = - Core.step (initial ()) (Core.Account_authenticated { user_id = Some user_id }) - in - match authenticated.effects with - | [ Core.Run (Core.Request (ticket, Core.Fetch_catalog account)) ] -> - authenticated, ticket, account - | [ Core.Publish _; Core.Run (Core.Request (ticket, Core.Fetch_catalog account)) ] -> - authenticated, ticket, account - | _ -> Alcotest.fail "authentication did not issue one catalog request" -;; - -let account_deletion_completion signed_out (result : (unit, Core.effect_error) result) = - List.find_map - (function - | Core.Run (Core.Request (ticket, Core.Delete_account_secrets account)) -> - Some (Core.Runner_completed (Core.Completion (ticket, result)), account) - | Run _ | Delegate _ | Publish _ -> None) - signed_out.Core.effects - |> Option.get -;; - -let sign_out_captures_identity_advances_generations_and_fences_old_work () = - let authorized, catalog_ticket, old_account = authenticated_fetch "old-user" in - let before = Core.state authorized.next in - let signed_out = - Core.step authorized.next (Core.Account_authenticated { user_id = None }) - in - let after = Core.state signed_out.next in - Alcotest.(check bool) - "account generation advances" - true - (after.snapshot.startup.account_generation - = before.snapshot.startup.account_generation + 1); - Alcotest.(check bool) - "presentation generation advances" - true - (after.snapshot.startup.presentation_generation - = before.snapshot.startup.presentation_generation + 1); - Alcotest.(check bool) - "authentication is cleared" - false - after.snapshot.startup.authenticated; - (match signed_out.effects with - | [ Core.Run (Core.Cancel_effects cancelled) - ; Core.Delegate (Core.Reset_managed_account reset) - ; Core.Run (Core.Request (_, Core.Delete_account_secrets deleted)) - ; Core.Publish (Core.State_changed published) - ] -> - Alcotest.(check bool) - "old account work is cancelled before cleanup" - true - (cancelled = Core.effect_scope_of_account old_account); - Alcotest.(check bool) "worker reset captures old account" true (reset = old_account); - Alcotest.(check bool) "cleanup captures old account" true (deleted = old_account); - Alcotest.(check bool) "signed-out state is published" true (published = after) - | _ -> Alcotest.fail "sign-out effects were missing or incorrectly ordered"); - let late_catalog = - Core.step - signed_out.next - (Core.Runner_completed (Core.Completion (catalog_ticket, Ok [ graph ]))) - in - Alcotest.(check int) - "old catalog completion is inert" - 0 - (List.length late_catalog.effects); - Alcotest.(check bool) - "old catalog completion cannot restore state" - true - (Core.state late_catalog.next = after) -;; - -let sign_out_cleanup_completion_is_inert_and_sanitized () = - let authorized, _, _ = authenticated_fetch "old-user" in - let signed_out = - Core.step authorized.next (Core.Account_authenticated { user_id = None }) - in - let success, _ = account_deletion_completion signed_out (Ok ()) in - let succeeded = Core.step signed_out.next success in - Alcotest.(check int) - "successful cleanup publishes nothing" - 0 - (List.length succeeded.effects); - Alcotest.(check bool) - "successful cleanup leaves public state unchanged" - true - (Core.state succeeded.next = Core.state signed_out.next); - let signed_out_again = - Core.step authorized.next (Core.Account_authenticated { user_id = None }) - in - let failure, _ = - account_deletion_completion - signed_out_again - (Error (Core.Effect_failed "service=secret key=private")) - in - let failed = Core.step signed_out_again.next failure in - let failed_state = Core.state failed.next in - Alcotest.(check (option string)) - "cleanup failure is sanitized" - (Some "account secret cleanup failed") - failed_state.snapshot.last_error; - Alcotest.(check bool) - "cleanup failure does not restore authentication" - false - failed_state.snapshot.startup.authenticated; - let login=Core.step signed_out_again.next (Core.Sign_in_requested (startup_scope signed_out_again.next)) in - let replacement = - Core.step login.next (Core.Account_authenticated { user_id = Some "new-user" }) - in - let late_failure = Core.step replacement.next failure in - Alcotest.(check int) - "old cleanup failure publishes nothing for replacement account" - 0 - (List.length late_failure.effects); - Alcotest.(check bool) - "old cleanup failure cannot fail replacement account" - true - (Core.state late_failure.next = Core.state replacement.next) -;; - -let local_cache_deletion_rejects_unopened_graphs () = - let authenticated = - Core.step (initial ()) (Core.Account_authenticated { user_id = Some "user" }) - in - let selected, scope = selected_graph graph in - let absent = - Core.step selected.next (Core.Mirror_inspected (Core.Mirror_absent scope)) - in - let downloading = - List.find_map - (function - | Core.Run (Core.Request (ticket, Core.Fetch_snapshot_metadata _)) -> - Some - (Core.step - absent.next - (Core.Runner_completed - (Core.Completion - ( ticket - , Ok {|{"ok":true,"t":42,"url":"https://example.com/snapshot.sqlite"}|} )))) - | _ -> None) - absent.effects - |> Option.get - in - let activating = - List.find_map - (function - | Core.Run (Core.Request (ticket, Core.Download_snapshot download_request)) -> - let artifact = - Core.stage_snapshot_download download_request ~path:"/tmp/snapshot.sqlite" ~expected_rows:10 |> Result.get_ok - in - Some - (Core.step - downloading.next - (Core.Runner_completed (Core.Completion (ticket, Ok artifact)))) - | _ -> None) - downloading.effects - |> Option.get - in - let activated = Core.step activating.next (Core.Snapshot_activated - (Core.snapshot_activation_completed (snapshot_activation_request activating.effects))) in - let attaching = - Core.step - activated.next - (Core.Mirror_inspected - (Core.Mirror_available (inspect_mirror_request activated.effects))) - in - let encrypted, _ = selected_graph encrypted_graph in - let loading_key = - Core.step - encrypted.next - (Core.Mirror_inspected - (Core.Mirror_available (inspect_mirror_request encrypted.effects))) - in - List.iter - (fun core -> - let deletion = Core.step core (Core.Local_cache_deletion_requested graph_id) in - Alcotest.(check bool) - "unsupported deletion is inert" - true - (observe core = observe deletion.next && deletion.effects = [])) - [ initial () - ; authenticated.next - ; selected.next - ; absent.next - ; downloading.next - ; activating.next - ; activated.next - ; attaching.next - ; loading_key.next - ] -;; - -let deletion_ready_graph encrypted = - let graph = { graph with encrypted } in - let selected, scope = selected_graph graph in - let inspected = - Core.step - selected.next - (Core.Mirror_inspected - (Core.Mirror_available (inspect_mirror_request selected.effects))) - in - let attaching = - if encrypted - then ( - let completion = - List.find_map - (function - | Core.Run (Core.Request (ticket, Core.Load_and_unlock_graph_key _)) -> - Some - (Core.Runner_completed - (Core.Completion - (ticket, Ok (Core.graph_key_handle ~id:"retained-key" ~scope)))) - | _ -> None) - inspected.effects - |> Option.get - in - Core.step inspected.next completion) - else inspected - in - let token = Overlay.sync_token_of_string "sync-token:v1:delete" |> Result.get_ok in - let checkpoint = - Overlay.Server_cursor.of_int64 0L - |> Result.map_error (fun `Negative_cursor -> "negative cursor") - |> Result.get_ok - in - let sync = Overlay.sync_view ~token ~checkpoint ~submissions:[] in - let attached = Core.step attaching.next (Core.Graph_attached { scope; sync }) in - let ready = Core.step attached.next (Core.Timeline_presented (timeline_scope attached.next)) in - ready.next, scope, sync, selected.effects -;; - -let deletion_save_completion effects (result : (unit, Core.effect_error) result) = - List.find_map - (function - | Core.Run (Core.Request (ticket, Core.Save_catalog _)) -> - Some (Core.Runner_completed (Core.Completion (ticket, result))) - | _ -> None) - effects - |> Option.get -;; - -let deletion_mirror effects = - let request = List.find_map - (function - | Core.Delegate (Core.Delete_mirror request) -> Some request - | _ -> None) - effects in - match request with - | Some request -> request - | None -> Alcotest.fail "non-current deletion must delegate mirror removal" -;; - -let check_deletion_stage stage core = - Alcotest.(check bool) - "visible deletion stage" - true - ((Core.state core).snapshot.local_deletion = Some (Core.Deletion_in_progress stage)); - Alcotest.(check bool) - "picker waits for persistence" - false - (Core.state core).snapshot.startup.awaiting_selection -;; - -let check_no_work core events = - List.iter - (fun event -> - let step = Core.step core event in - Alcotest.(check bool) - "event cannot restart or change deletion" - true - (observe core = observe step.next && step.effects = [])) - events -;; - -let deletion_blocked_events (scope : Core.graph_scope) sync = - [ Core.Local_cache_deletion_requested graph_id - ; Core.Local_cache_deletion_requested other_graph_id - ; Core.Graph_selected graph_id - ; Core.Graph_selected other_graph_id - ; Core.Graph_picker_requested - ; Core.Online_recovery_requested {startup={account_generation=scope.account.account_generation;presentation_generation=scope.account.presentation_generation};graph_id=scope.graph_id;graph_generation=scope.graph_generation} - ; Core.Local_outbox_changed - ; Core.Local_feed_acknowledged - ; Core.Timeline_presented {startup={account_generation=scope.account.account_generation;presentation_generation=scope.account.presentation_generation};graph_id=scope.graph_id;graph_generation=scope.graph_generation} - ; Core.Catalog_refresh_requested {account_generation=scope.account.account_generation;presentation_generation=scope.account.presentation_generation} - ; Core.Account_authenticated { user_id = Some "user" } - ; Core.E2ee_password_submitted (Lazy.force fixture_challenge, "ignored") - ; Core.Foreground_changed { foreground = true; lifecycle_generation = 98L } - ; Core.Foreground_changed { foreground = false; lifecycle_generation = 99L } - ; Core.Graph_attached { scope; sync } - ; Core.Sync_inspected { scope; sync } - ; Core.Snapshot_activated (Core.snapshot_activation_completed (Lazy.force fixture_snapshot_activation)) - ; Core.Mirror_inspected (Core.Mirror_available { graph; scope }) - ; Core.Websocket_opened { graph = scope; connection_generation = 1 } - ; Core.Websocket_closed - ({ graph = scope; connection_generation = 1 }, Core.Recoverable_close { message = Some "late"; attention = Requires_attention }) - ] -;; - -let deletion_noncurrent core = - let target = List.find (fun (candidate : Core.graph) -> candidate.graph_id = graph_id) - (Core.state core).snapshot.catalog in - let catalog = refresh_catalog core [ target; other_graph ] in - let selected = Core.step catalog.next (Core.Graph_selected other_graph_id) in - let request = inspect_mirror_request selected.effects in - let inspected = Core.step selected.next (Core.Mirror_inspected (Core.Mirror_available request)) in - let token = Overlay.sync_token_of_string "sync-token:v1:noncurrent" |> Result.get_ok in - let sync = Overlay.sync_view ~token ~checkpoint:(server_cursor_of_int 0) ~submissions:[] in - let attached = Core.step inspected.next (Core.Graph_attached { scope = request.scope; sync }) in - let ready = Core.step attached.next (Core.Timeline_presented (timeline_scope attached.next)) in - ready.next, request.scope, sync, selected.effects -;; - -let local_deletion_orders_cleanup encrypted () = - let original, old_scope, _, _ = deletion_ready_graph encrypted in - let ready, scope, sync, selection_effects = deletion_noncurrent original in - check_no_work ready [ Core.Local_cache_deletion_requested other_graph_id ]; - let deleting = Core.step ready (Core.Local_cache_deletion_requested graph_id) in - check_deletion_stage Deleting_mirror deleting.next; - let request = deletion_mirror deleting.effects in - Alcotest.(check bool) "target differs from current" true - (request.graph_id = graph_id && scope.graph_id = other_graph_id); - Alcotest.(check bool) "selected graph and admitted session remain unchanged" true - ((Core.state deleting.next).snapshot.selected_graph = Some other_graph_id - && Core.admitted_graph_scope deleting.next = Some scope); - Alcotest.(check bool) "no active graph close or remote request" true - (List.for_all (function Core.Delegate (Core.Delete_mirror _) | Core.Publish _ -> true | _ -> false) deleting.effects); - check_no_work deleting.next - [ Core.Local_cache_deletion_requested graph_id; Core.Local_cache_deletion_requested other_graph_id ]; - let stale = { request with scope = { request.scope with lifecycle_generation = Some 99999L } } in - check_no_work deleting.next [ Core.Mirror_deleted (stale, Ok ()); Core.Graph_detached (old_scope, Ok ()) ]; - let removed = Core.step deleting.next (Core.Mirror_deleted (request, Ok ())) in - let snapshot = (Core.state removed.next).snapshot in - Alcotest.(check bool) "success retains current and catalog" true - (snapshot.selected_graph = Some other_graph_id && snapshot.local_deletion = None - && snapshot.catalog = (Core.state ready).snapshot.catalog - && Core.admitted_graph_scope removed.next = Some scope && snapshot.last_error = None); - check_no_work removed.next [ Core.Mirror_deleted (request, Ok ()); Core.Graph_detached (old_scope, Ok ()) ]; - let cache = save_catalog_cache selection_effects in - Alcotest.(check bool) "saved selection remains the other graph" true - (Core.catalog_cache_selected_graph cache = Some other_graph_id); - let restarting = Core.step (initial ()) (Core.Restore_local_account { user_id = "user" }) in - let restored = Core.step restarting.next (load_catalog_completion restarting.effects (Some cache)) in - Alcotest.(check bool) "restart restores unchanged selection" true - ((Core.state restored.next).snapshot.selected_graph = Some other_graph_id); - let selected = Core.step removed.next (Core.Graph_selected graph_id) in - let mirror = inspect_mirror_request selected.effects in - let requested=Core.step selected.next (Core.Online_recovery_requested (timeline_scope selected.next)) in - let absent = Core.step requested.next (Core.Mirror_inspected (Core.Mirror_absent mirror.scope)) in - Alcotest.(check bool) "explicit download of deleted graph uses normal absent-mirror flow" true - (List.exists (function - | Core.Run (Core.Request (_, Core.Fetch_snapshot_metadata _)) -> not encrypted - | Core.Run (Core.Request (_, Core.Load_and_unlock_graph_key _)) -> encrypted - | _ -> false) absent.effects); - let _ = sync in - () -;; - -let local_deletion_discards_unsynchronized_work () = - let submitted, connection = submitted_core () in - Alcotest.(check bool) "fixture has unacknowledged work" true - ((Core.state submitted).snapshot.sync_phase = Core.Submitting); - let ready, scope, _, _ = deletion_noncurrent submitted in - let deleting = Core.step ready (Core.Local_cache_deletion_requested graph_id) in - let request = deletion_mirror deleting.effects in - Alcotest.(check bool) "confirmed deletion does not wait for upload" true - (request.graph_id = graph_id - && Core.admitted_graph_scope deleting.next = Some scope - && List.for_all (function Core.Publish _ | Core.Delegate (Core.Delete_mirror _) -> true | _ -> false) deleting.effects); - check_no_work deleting.next - [ Core.Websocket_message (connection, Protocol.Server.Tx_batch_ok - { t = server_cursor_of_int 1; checksum = Some "0123456789abcdef" }) ]; - let removed = Core.step deleting.next (Core.Mirror_deleted (request, Ok ())) in - Alcotest.(check bool) "unsynced discard completes locally" true - ((Core.state removed.next).snapshot.local_deletion = None) -;; - -let local_deletion_failures_stop encrypted () = - let original, _, _, _ = deletion_ready_graph encrypted in - let ready, scope, _, _ = deletion_noncurrent original in - let deleting = Core.step ready (Core.Local_cache_deletion_requested graph_id) in - let request = deletion_mirror deleting.effects in - let failed = Core.step deleting.next (Core.Mirror_deleted (request, Error "secret path")) in - let snapshot = (Core.state failed.next).snapshot in - Alcotest.(check bool) "failure releases the flight and retains current session" true - (snapshot.local_deletion = None && snapshot.selected_graph = Some other_graph_id - && Core.admitted_graph_scope failed.next = Some scope); - Alcotest.(check (option string)) "failure is sanitized" - (Some "Local graph deletion failed. Try again.") snapshot.last_error; - let retry = Core.step failed.next (Core.Local_cache_deletion_requested graph_id) in - let retry_request = deletion_mirror retry.effects in - Alcotest.(check bool) "retry has a distinct identity" true (retry_request <> request); - check_no_work retry.next [ Core.Mirror_deleted (request, Ok ()); Core.Mirror_deleted (request, Error "late") ]; - let recovered = Core.step retry.next (Core.Mirror_deleted (retry_request, Ok ())) in - Alcotest.(check bool) "retry recovers" true ((Core.state recovered.next).snapshot.last_error = None); - let reopened = Core.step failed.next (Core.Graph_selected graph_id) in - Alcotest.(check bool) "failed deletion can reopen intact graph" true - ((inspect_mirror_request reopened.effects).graph.graph_id = graph_id) -;; - -let local_deletion_account_change_fences_completion () = - let original, _, _, _ = deletion_ready_graph false in - let ready, _, _, _ = deletion_noncurrent original in - let deleting = Core.step ready (Core.Local_cache_deletion_requested graph_id) in - let request = deletion_mirror deleting.effects in - let changed = Core.step deleting.next (Core.Account_authenticated { user_id = Some "another-user" }) in - Alcotest.(check bool) "account change retires deletion" true - ((Core.state changed.next).snapshot.startup.account_generation - > (Core.state deleting.next).snapshot.startup.account_generation - && (Core.state changed.next).snapshot.local_deletion = None); - check_no_work changed.next [ Core.Mirror_deleted (request, Ok ()); Core.Mirror_deleted (request, Error "late") ] -;; - -let recovery_connection effects = - List.find_map - (function - | Core.Run (Core.Start_websocket request) -> Some request.scope - | Run _ | Delegate _ | Publish _ -> None) - effects - |> Option.get -;; - -let restore_recovery_core sync = - let restoring = - Core.step (initial ()) (Core.Restore_local_account { user_id = "user" }) - in - let cache = - Core.catalog_cache ~user_id:"user" ~graphs:[ graph ] ~selected_graph:(Some graph_id) - in - let restored = - Core.step restoring.next (load_catalog_completion restoring.effects (Some cache)) - in - let mirror = inspect_mirror_request restored.effects in - let inspected = - Core.step restored.next (Core.Mirror_inspected (Core.Mirror_available mirror)) - in - let attached = - Core.step inspected.next (Core.Graph_attached { scope = mirror.scope; sync }) - in - let shown=Core.step attached.next (Core.Timeline_presented (timeline_scope attached.next)) in - let authenticated=Core.step shown.next (Core.Account_authenticated {user_id=Some "user"}) in - let connection = recovery_connection authenticated.effects in - let opened = Core.step authenticated.next (Core.Websocket_opened connection) in - opened.next, connection -;; - -let recovery_delete_transaction = - "[[\"~:db/retractEntity\",[\"~:block/uuid\",\"~u11111111-1111-4111-8111-111111111111\"]]]" -;; - -let submitted_delete_recovery_fixture () = - let mutation_id = List.hd mutation_ids in - let cursor = - Overlay.Server_cursor.of_int64 0L - |> Result.map_error (fun `Negative_cursor -> "negative cursor") - |> Result.get_ok - in - let token ordinal = - Overlay.sync_token_of_string (Printf.sprintf "sync-token:v1:%d" ordinal) - |> Result.get_ok - in - let sync token submissions = Overlay.sync_view ~token ~checkpoint:cursor ~submissions in - let core, connection = restore_recovery_core (sync (token 0) []) in - let mutation : Overlay.submission_descriptor = - { mutation_id - ; fingerprint = - Overlay.Mutation_fingerprint.of_string "mutation-fingerprint:v1:delete-recovery" - |> Result.get_ok - ; state = Overlay.Queued - ; operation = Overlay.Delete_blocks_operation - ; dependency_eligible = true - ; attempt_count = 0 - ; plaintext_bytes = 128 - ; protected_bytes = Some 128 - } - in - let inspection = Core.step core Core.Local_outbox_changed in - let planned = - Core.step - inspection.next - (Core.Sync_inspected - { scope = connection.graph; sync = sync (token 1) [ mutation ] }) - in - let request = - List.find_map - (function - | Core.Delegate (Core.Apply_outbox_transition request) -> Some request - | Run _ | Delegate _ | Publish _ -> None) - planned.effects - |> Option.get - in - Alcotest.(check bool) - "fixture submits exactly the delete mutation" - true - (request.transition = Overlay.Submit_group [ mutation_id ]); - let batch_id = - Overlay.Submission_batch_id.of_string "submission-batch:v1:delete-recovery" - |> Result.get_ok - in - let wire = - Overlay.submission_wire - ~maximum_bytes:1024 - ~mutation_id - ~operation:Overlay.Delete_blocks_operation - ~protected_transaction:recovery_delete_transaction - |> Result.get_ok - in - let batch = - Overlay.submission_batch - ~maximum_wires:1 - ~maximum_bytes:1024 - ~batch_id - ~t_before:cursor - ~wires:[ wire ] - |> Result.get_ok - in - let submitted_sync = - sync - (token 2) - [ { mutation with state = Overlay.Submitted batch_id; attempt_count = 1 } ] - in - let projection = - Overlay.Projection_revision.of_string "projection:v1:1" |> Result.get_ok - in - let commit : Overlay.outbox_commit = - { generation = Overlay.Generation.of_string "generation:v1:1" |> Result.get_ok - ; before_projection_revision = projection - ; after_projection_revision = projection - ; sync_token = token 2 - ; transition = request.transition - ; activity = Overlay.Logically_active - ; logical_change_summary = Overlay.No_logical_change - ; submission_batch = Some batch - } - in - let submitted = - Core.step - planned.next - (Core.Outbox_transition_applied - { scope = request.scope; commit; sync = submitted_sync }) - in - Alcotest.(check bool) - "fixture sends the persisted delete batch" - true - (List.exists - (function - | Core.Run - (Core.Send_websocket - { scope; message = Protocol.Client.Tx_batch { txs = [ tx ]; _ } }) -> - scope = connection && tx.outliner_op = Some "delete-blocks" - | Run _ | Delegate _ | Publish _ -> false) - submitted.effects); - submitted, connection, submitted_sync, batch -;; - -let defer_recovered_delete core connection batch_id = - let pulled = - Core.step - core - (Core.Websocket_message - ( connection - , Protocol.Server.Pull_ok - { t = server_cursor_of_int 1 - ; checksum = Some "0123456789abcdef" - ; txs = - [ { t = server_cursor_of_int 1 - ; tx = recovery_delete_transaction - ; outliner_op = Some "delete-blocks" - } - ] - } )) - in - let batch = - List.find_map - (function - | Core.Delegate (Core.Apply_authoritative_batch request) -> Some request - | Run _ | Delegate _ | Publish _ -> None) - pulled.effects - |> Option.get - in - Alcotest.(check bool) - "pull reaches the authoritative worker boundary" - true - (batch.scope = connection); - (* The overlay's deferred outcome is an input at this pure policy boundary. *) - let event = - Core.Authoritative_batch_deferred - { scope = connection.graph; defer = Overlay.Await_submission_outcome batch_id } - in - let deferred = preview_step "submitted delete defer" pulled.next event in - ignore (check_replay "submitted delete defer" pulled.next event deferred); - deferred, batch -;; - -let live_submitted_delete_resumes_after_acknowledgement () = - let submitted, connection, sync, wire_batch = submitted_delete_recovery_fixture () in - let batch_id = Overlay.submission_batch_id wire_batch in - let deferred, batch = defer_recovered_delete submitted.next connection batch_id in - let deferred = deferred.next in - Alcotest.(check (option string)) - "live owner accepts the deferred result" - None - (Core.state deferred).snapshot.last_error; - let acknowledged = - Core.step - deferred - (Core.Websocket_message - ( connection - , Protocol.Server.Tx_batch_ok - { t = server_cursor_of_int 1; checksum = Some "0123456789abcdef" } )) - in - let request = - List.find_map - (function - | Core.Delegate - (Core.Apply_outbox_transition - ({ transition = Overlay.Accept_group { batch_id = accepted; _ }; _ } as - request)) - when Overlay.Submission_batch_id.equal accepted batch_id -> Some request - | Run _ | Delegate _ | Publish _ -> None) - acknowledged.effects - |> Option.get - in - let accepted_sync = - Overlay.sync_view - ~token:(Overlay.sync_token_of_string "sync-token:v1:3" |> Result.get_ok) - ~checkpoint:(Overlay.sync_view_checkpoint sync) - ~submissions: - (List.map - (fun (descriptor : Overlay.submission_descriptor) -> - { descriptor with state = Overlay.Accepted_pending_authoritative batch_id }) - (Overlay.sync_view_submissions sync)) - in - let projection = - Overlay.Projection_revision.of_string "projection:v1:1" |> Result.get_ok - in - let commit : Overlay.outbox_commit = - { generation = Overlay.Generation.of_string "generation:v1:1" |> Result.get_ok - ; before_projection_revision = projection - ; after_projection_revision = projection - ; sync_token = Overlay.sync_view_token accepted_sync - ; transition = request.transition - ; activity = Overlay.Logically_active - ; logical_change_summary = Overlay.No_logical_change - ; submission_batch = None - } - in - let resumed = - Core.step - acknowledged.next - (Core.Outbox_transition_applied - { scope = request.scope; commit; sync = accepted_sync }) - in - check_instructions - "live acknowledgement resumes the same deferred batch" - [ Core.Delegate (Core.Apply_authoritative_batch batch) ] - resumed.effects -;; - -let recovery_cursor n = - Overlay.Server_cursor.of_int64 (Int64.of_int n) - |> Result.map_error (fun `Negative_cursor -> "negative cursor") - |> Result.get_ok -;; - -let recovery_sync _sync ?(checkpoint = 0) ordinal submissions = - Overlay.sync_view - ~token: - (Overlay.sync_token_of_string (Printf.sprintf "sync-token:v1:%d" ordinal) - |> Result.get_ok) - ~checkpoint:(recovery_cursor checkpoint) - ~submissions -;; - -let recovery_request effects = - match - List.filter_map - (function - | Core.Delegate (Core.Apply_outbox_transition request) -> Some request - | Run _ | Delegate _ | Publish _ -> None) - effects - with - | [ request ] -> request - | _ -> Alcotest.fail "expected exactly one recovery outbox transition" -;; - -let recovery_commit (request : Core.outbox_transition_request) sync submission_batch = - let projection = - Overlay.Projection_revision.of_string "projection:v1:1" |> Result.get_ok - in - let commit : Overlay.outbox_commit = - { generation = Overlay.Generation.of_string "generation:v1:1" |> Result.get_ok - ; before_projection_revision = projection - ; after_projection_revision = projection - ; sync_token = Overlay.sync_view_token sync - ; transition = request.transition - ; activity = Overlay.Logically_active - ; logical_change_summary = Overlay.No_logical_change - ; submission_batch - } - in - Core.Outbox_transition_applied { scope = request.scope; commit; sync } -;; - -let require_no_new_submission effects = - Alcotest.(check bool) - "unresolved recovery blocks new mutations" - false - (List.exists - (function - | Core.Delegate - (Core.Apply_outbox_transition { transition = Overlay.Submit_group _; _ }) -> - true - | Run _ | Delegate _ | Publish _ -> false) - effects) -;; - -let recovery_apply core (connection : Core.connection_scope) sync = - let projection = - Overlay.Projection_revision.of_string "projection:v1:1" |> Result.get_ok - in - let commit : Overlay.authoritative_commit = - { generation = Overlay.Generation.of_string "generation:v1:1" |> Result.get_ok - ; before_projection_revision = projection - ; after_projection_revision = projection - ; checkpoint = Overlay.sync_view_checkpoint sync - ; sync_token = Overlay.sync_view_token sync - ; terminal_receipts = [] - ; replanned_queued_ids = [] - ; blocked_ids = [] - ; logical_change_summary = Overlay.No_logical_change - } - in - Core.step - core - (Core.Authoritative_batch_applied { scope = connection.Core.graph; commit; sync }) -;; - -let recover_retry core connection sync wire_batch effects = - let request = recovery_request effects in - Alcotest.(check bool) - "recover the original durable batch" - true - (request.transition = Overlay.Retry_group (Overlay.submission_batch_id wire_batch)); - let retried_sync = - recovery_sync - sync - 3 - (List.map - (fun (item : Overlay.submission_descriptor) -> - { item with attempt_count = item.attempt_count + 1 }) - (Overlay.sync_view_submissions sync)) - in - let retried = Core.step core (recovery_commit request retried_sync (Some wire_batch)) in - let sends = - List.filter_map - (function - | Core.Run (Core.Send_websocket send) -> Some send - | Run _ | Delegate _ | Publish _ -> None) - retried.effects - in - (match sends with - | [ { scope - ; message = Protocol.Client.Tx_batch { client_revision; t_before; txs = [ tx ] } - } - ] -> - Alcotest.(check bool) "retry uses the current connection" true (scope = connection); - Alcotest.(check (option string)) - "retry retains batch ID" - (Some - (Overlay.Submission_batch_id.to_string (Overlay.submission_batch_id wire_batch))) - client_revision; - Alcotest.(check cursor_testable) - "retry retains conditional baseline" - (server_cursor_of_int 0) - t_before; - Alcotest.(check string) - "retry retains protected bytes" - recovery_delete_transaction - tx.tx - | _ -> Alcotest.fail "retry did not send the original wire batch once"); - Alcotest.(check bool) - "retry waits for terminal outcome before replaying deletion" - false - (List.exists - (function - | Core.Delegate (Core.Apply_authoritative_batch _) -> true - | Run _ | Delegate _ | Publish _ -> false) - retried.effects); - retried, retried_sync -;; - -let finish_recovery ~reject core connection sync wire_batch effects batch = - let retried, sync = recover_retry core connection sync wire_batch effects in - let batch_id = Overlay.submission_batch_id wire_batch in - let response = - if reject - then - Protocol.Server.Tx_reject - { reason = Protocol.Stale - ; t = Some (server_cursor_of_int 1) - ; checksum = Some "0123456789abcdef" - ; success_tx_ids = [] - ; failed_tx_id = None - ; missing_block_uuids = [] - ; error_detail = None - ; data = None - } - else - Protocol.Server.Tx_batch_ok - { t = server_cursor_of_int 1; checksum = Some "0123456789abcdef" } - in - let terminal = Core.step retried.next (Core.Websocket_message (connection, response)) in - let request = recovery_request terminal.effects in - let state = - if reject - then - Overlay.Stale_rejected_pending_authoritative - { batch_id; through = recovery_cursor 1 } - else Overlay.Accepted_pending_authoritative batch_id - in - let terminal_sync = - recovery_sync - sync - 4 - (List.map - (fun (item : Overlay.submission_descriptor) -> { item with state }) - (Overlay.sync_view_submissions sync)) - in - let resumed = Core.step terminal.next (recovery_commit request terminal_sync None) in - Alcotest.(check bool) - "terminal outcome replays the same deferred authoritative batch" - true - (List.exists - (function - | Core.Delegate (Core.Apply_authoritative_batch actual) -> actual = batch - | Run _ | Delegate _ | Publish _ -> false) - resumed.effects); - Alcotest.(check bool) - "deferred replay is not Current" - false - ((Core.state resumed.next).snapshot.sync_phase = Core.Current); - require_no_new_submission resumed.effects; - let current = - recovery_apply resumed.next connection (recovery_sync sync ~checkpoint:1 5 []) - in - Alcotest.(check (option cursor_testable)) - "recovery advances the durable checkpoint" - (Some (server_cursor_of_int 1)) - (Core.state current.next).snapshot.applied_server_t; - Alcotest.(check bool) - "completed recovery becomes Current" - true - ((Core.state current.next).snapshot.sync_phase = Core.Current) -;; - -let check_submitted_delete_recovery ?(reject = false) core connection sync wire_batch = - let recovered, batch = - defer_recovered_delete core connection (Overlay.submission_batch_id wire_batch) - in - Alcotest.(check (option string)) - "a durable submitted delete must remain recoverable after its transport owner ends" - None - (Core.state recovered.next).snapshot.last_error; - Alcotest.(check bool) - "recovery does not enter Failed" - false - ((Core.state recovered.next).snapshot.sync_phase = Core.Failed); - finish_recovery - ~reject - recovered.next - connection - sync - wire_batch - recovered.effects - batch -;; - -let restart_after_retirement core connection = - let retired = Core.step core (Core.Websocket_retired connection) in - let timer = - List.find_map - (function - | Core.Run (Core.Schedule_timer timer) -> Some timer - | _ -> None) - retired.effects - |> Option.get - in - Core.step retired.next (Core.Timer_elapsed timer.id) -;; - -let submitted_delete_survives_disconnect_reconnect () = - let submitted, connection, sync, wire_batch = submitted_delete_recovery_fixture () in - let closed = - Core.step - submitted.next - (Core.Websocket_closed (connection, Core.Recoverable_close { message = None; attention = Requires_attention })) - in - let restarting = restart_after_retirement closed.next connection in - let current_connection = recovery_connection restarting.effects in - Alcotest.(check bool) - "reconnect advances the connection generation" - true - (current_connection.connection_generation > connection.connection_generation); - let opened = Core.step restarting.next (Core.Websocket_opened current_connection) in - check_submitted_delete_recovery opened.next current_connection sync wire_batch -;; - -let submitted_delete_survives_fresh_core_restore () = - let _, _, persisted_sync, wire_batch = submitted_delete_recovery_fixture () in - let fresh, connection = restore_recovery_core persisted_sync in - Alcotest.(check (option cursor_testable)) - "fresh Core restores the durable checkpoint" - (Some (server_cursor_of_int 0)) - (Core.state fresh).snapshot.applied_server_t; - check_submitted_delete_recovery fresh connection persisted_sync wire_batch -;; - -let recovery_stale_rejection_resumes_deletion () = - let _, _, sync, wire_batch = submitted_delete_recovery_fixture () in - let fresh, connection = restore_recovery_core sync in - check_submitted_delete_recovery ~reject:true fresh connection sync wire_batch -;; - -let recovery_empty_pull_prioritizes_submitted () = - let _, _, sync, wire_batch = submitted_delete_recovery_fixture () in - let original = List.hd (Overlay.sync_view_submissions sync) in - let queued = - { original with - mutation_id = List.nth mutation_ids 1 - ; state = Overlay.Queued - ; attempt_count = 0 - } - in - let sync = recovery_sync sync 3 [ queued; original ] in - let fresh, connection = restore_recovery_core sync in - let pulled = - Core.step - fresh - (Core.Websocket_message - ( connection - , Protocol.Server.Pull_ok - { t = server_cursor_of_int 0; checksum = None; txs = [] } )) - in - require_no_new_submission pulled.effects; - ignore (recover_retry pulled.next connection sync wire_batch pulled.effects) -;; - -let recovery_barrier_blocks_queued ~rejected () = - let _, _, sync, wire_batch = submitted_delete_recovery_fixture () in - let batch_id = Overlay.submission_batch_id wire_batch in - let original = List.hd (Overlay.sync_view_submissions sync) in - let state = - if rejected - then - Overlay.Stale_rejected_pending_authoritative - { batch_id; through = recovery_cursor 1 } - else Overlay.Accepted_pending_authoritative batch_id - in - let pending = { original with state } in - let queued = - { original with - mutation_id = List.nth mutation_ids 1 - ; state = Overlay.Queued - ; attempt_count = 0 - } - in - let sync = recovery_sync sync 3 [ queued; pending ] in - let fresh, connection = restore_recovery_core sync in - let pulled = - Core.step - fresh - (Core.Websocket_message - ( connection - , Protocol.Server.Pull_ok - { t = server_cursor_of_int 0; checksum = None; txs = [] } )) - in - require_no_new_submission pulled.effects; - Alcotest.(check bool) - "unresolved barrier is not Current" - false - ((Core.state pulled.next).snapshot.sync_phase = Core.Current) -;; - -let recovery_stale_connection_and_interrupted_retry () = - let _, old_connection, sync, wire_batch = submitted_delete_recovery_fixture () in - let fresh, connection = restore_recovery_core sync in - let retry, _ = - defer_recovered_delete fresh connection (Overlay.submission_batch_id wire_batch) - in - let request = recovery_request retry.effects in - let closed = - Core.step retry.next (Core.Websocket_closed (connection, Core.Recoverable_close { message = None; attention = Requires_attention })) - in - let committed = - Core.step closed.next (recovery_commit request sync (Some wire_batch)) - in - Alcotest.(check bool) - "offline durable completion emits no websocket send" - false - (List.exists - (function - | Core.Run (Core.Send_websocket _) -> true - | _ -> false) - committed.effects); - let restarted = restart_after_retirement committed.next connection in - let new_connection = recovery_connection restarted.effects in - let opened = Core.step restarted.next (Core.Websocket_opened new_connection) in - let late = - Core.step - opened.next - (Core.Websocket_message - ( old_connection - , Protocol.Server.Tx_batch_ok - { t = server_cursor_of_int 1; checksum = Some "0123456789abcdef" } )) - in - check_instructions "old response cannot accept recovered batch" [] late.effects; - check_submitted_delete_recovery late.next new_connection sync wire_batch -;; - -let recovery_submission_timeout () = - let submitted, connection, sync, wire_batch = submitted_delete_recovery_fixture () in - let timer = - List.find_map - (function - | Core.Run (Core.Schedule_timer timer) -> Some timer - | Run _ | Delegate _ | Publish _ -> None) - submitted.effects - in - Alcotest.(check bool) - "live submission has a bounded acknowledgement wait" - true - (Option.is_some timer); - let timer = Option.get timer in - let expired = Core.step submitted.next (Core.Timer_elapsed timer.id) in - Alcotest.(check bool) - "timeout closes the uncertain connection" - true - (List.exists - (function - | Core.Run (Core.Close_websocket scope) -> scope = connection - | _ -> false) - expired.effects); - let retired = Core.step expired.next (Core.Websocket_retired connection) in - let new_connection = recovery_connection retired.effects in - let opened = Core.step retired.next (Core.Websocket_opened new_connection) in - let replay = Core.step opened.next (Core.Timer_elapsed timer.id) in - check_instructions "old timeout is consumed" [] replay.effects; - check_submitted_delete_recovery replay.next new_connection sync wire_batch -;; - -let recovery_outbox_failure_is_correlated () = - let submitted, connection, sync, batch = submitted_delete_recovery_fixture () in - let acknowledged = - Core.step - submitted.next - (Core.Websocket_message - ( connection - , Protocol.Server.Tx_batch_ok - { t = server_cursor_of_int 1; checksum = Some "0123456789abcdef" } )) - in - let request = recovery_request acknowledged.effects in - let message = "The overlay outbox transition was rejected." in - let fail request = Core.Outbox_transition_failed { request; message } in - let foreign = - { request with - expected = Overlay.sync_token_of_string "sync-token:v1:999" |> Result.get_ok - } - in - let ignored = Core.step acknowledged.next (fail foreign) in - check_instructions - "foreign expected token cannot fail pending transition" - [] - ignored.effects; - Alcotest.(check bool) - "foreign failure leaves state unchanged" - true - (Core.state ignored.next = Core.state acknowledged.next); - let failed = Core.step ignored.next (fail request) in - Alcotest.(check bool) - "outbox failure is terminal and visible" - true - ((Core.state failed.next).snapshot.sync_phase = Core.Failed); - Alcotest.(check (option string)) - "outbox error is retained" - (Some message) - (Core.state failed.next).snapshot.last_error; - Alcotest.(check bool) - "failed response retires its connection" - true - (List.exists - (function - | Core.Run (Core.Close_websocket scope) -> scope = connection - | _ -> false) - failed.effects); - let late = Core.step failed.next (recovery_commit request sync None) in - check_instructions "late success cannot revive failed transition" [] late.effects; - let duplicate = Core.step late.next (fail request) in - check_instructions "duplicate failure is inert" [] duplicate.effects; - let old_ack = - Core.step - duplicate.next - (Core.Websocket_message - ( connection - , Protocol.Server.Tx_batch_ok - { t = server_cursor_of_int 1; checksum = Some "0123456789abcdef" } )) - in - check_instructions "old response cannot recreate a failed transition" [] old_ack.effects; - ignore batch -;; - -let recovery_retry_failure_is_visible () = - let _, _, sync, batch = submitted_delete_recovery_fixture () in - let fresh, connection = restore_recovery_core sync in - let retry, _ = - defer_recovered_delete fresh connection (Overlay.submission_batch_id batch) - in - let request = recovery_request retry.effects in - let failed = - Core.step - retry.next - (Core.Outbox_transition_failed { request; message = "Retry commit failed." }) - in - Alcotest.(check bool) - "retry failure does not remain Submitting" - true - ((Core.state failed.next).snapshot.sync_phase = Core.Failed); - let late = Core.step failed.next (recovery_commit request sync (Some batch)) in - check_instructions "failed retry completion cannot send" [] late.effects -;; - -let pull_sends effects = - List.filter_map - (function - | Core.Run (Core.Send_websocket { scope; message = Protocol.Client.Pull { since } }) - -> Some (scope, since) - | _ -> None) - effects -;; - -let check_pull_count message expected effects = - Alcotest.(check int) message expected (List.length (pull_sends effects)) -;; - -let pull_timer effects = - let timers = - List.filter_map - (function - | Core.Run (Core.Schedule_timer timer) -> Some timer - | _ -> None) - effects - in - Alcotest.(check int) "one bounded pull timer" 1 (List.length timers); - List.hd timers -;; - -let pull_fixture () = - let selected, _ = selected_graph graph in - let mirror = inspect_mirror_request selected.effects in - let inspected = - Core.step selected.next (Core.Mirror_inspected (Core.Mirror_available mirror)) - in - let sync = - Overlay.sync_view - ~token:(token Overlay.sync_token_of_string "sync-token:v1:pull-0") - ~checkpoint:(recovery_cursor 0) - ~submissions:[] - in - let attached = - Core.step inspected.next (Core.Graph_attached { scope = mirror.scope; sync }) - in - let connection = recovery_connection attached.effects in - Core.step attached.next (Core.Websocket_opened connection), connection, sync -;; - -let pull_message core connection message = - Core.step core (Core.Websocket_message (connection, message)) -;; - -let pull_response core connection ~after through = - let txs = - List.init (through - after) (fun offset -> - { Protocol.Server.t = server_cursor_of_int (after + offset + 1) - ; tx = "[]" - ; outliner_op = None - }) - in - pull_message - core - connection - (Protocol.Server.Pull_ok { t = server_cursor_of_int through; checksum = None; txs }) -;; - -let pull_open_and_notifications_are_singleflight () = - let opened, connection, _ = pull_fixture () in - check_pull_count "opening baseline pull" 1 opened.effects; - let hello = - pull_message - opened.next - connection - (Protocol.Server.Hello { t = server_cursor_of_int 100; checksum = None }) - in - check_pull_count "hello joins the opening pull" 0 hello.effects; - let state = ref hello.next in - for t = 1 to 100 do - let changed = - pull_message - !state - connection - (Protocol.Server.Changed { t = server_cursor_of_int t }) - in - check_pull_count "notification burst joins the same pull" 0 changed.effects; - state := changed.next - done; - Alcotest.(check (option cursor_testable)) - "notifications cannot advance the durable checkpoint" - (Some (server_cursor_of_int 0)) - (Core.state !state).snapshot.applied_server_t -;; - -let pull_demand_survives_authoritative_apply () = - let opened, connection, sync = pull_fixture () in - let notified = - pull_message - opened.next - connection - (Protocol.Server.Changed { t = server_cursor_of_int 1 }) - in - let received = pull_response notified.next connection ~after:0 1 in - let state = ref received.next in - for index = 1 to 100 do - let t = index + 1 in - let changed = - pull_message - !state - connection - (Protocol.Server.Changed { t = server_cursor_of_int t }) - in - check_pull_count "apply owns the pull until durable completion" 0 changed.effects; - state := changed.next - done; - let committed = - recovery_apply !state connection (recovery_sync sync ~checkpoint:1 1 []) - in - Alcotest.(check bool) - "latest target schedules one pull from committed cursor" - true - (pull_sends committed.effects = [ connection, Some (server_cursor_of_int 1) ]); - Alcotest.(check bool) - "catchup owner is represented by Pulling phase" - true - ((Core.state committed.next).snapshot.sync_phase = Core.Pulling); - Alcotest.(check bool) - "catchup phase is publicly published" - true - (List.exists - (function - | Core.Publish (Core.State_changed state) -> - state.snapshot.sync_phase = Core.Pulling - | _ -> false) - committed.effects); - let received = pull_response committed.next connection ~after:1 2 in - let committed = - recovery_apply received.next connection (recovery_sync sync ~checkpoint:2 2 []) - in - Alcotest.(check bool) - "partial catchup cannot drop the maximum requested target" - true - (pull_sends committed.effects = [ connection, Some (server_cursor_of_int 2) ]); - (* This pure policy test uses a sparse page to acknowledge the newest target. - The overlay separately enforces contiguous durable application. *) - let received = - pull_message - committed.next - connection - (Protocol.Server.Pull_ok - { t = server_cursor_of_int 101 - ; checksum = None - ; txs = - [ { Protocol.Server.t = server_cursor_of_int 101 - ; tx = "[]" - ; outliner_op = None - } - ] - }) - in - let committed = - recovery_apply received.next connection (recovery_sync sync ~checkpoint:101 3 []) - in - Alcotest.(check (option cursor_testable)) - "latest checkpoint is durably acknowledged" - (Some (server_cursor_of_int 101)) - (Core.state committed.next).snapshot.applied_server_t; - check_pull_count "caught up exactly to latest target" 0 committed.effects; - let stale = - pull_message - committed.next - connection - (Protocol.Server.Changed { t = server_cursor_of_int 100 }) - in - check_pull_count "old notifications need no pull" 0 stale.effects; - let same = - pull_message - stale.next - connection - (Protocol.Server.Changed { t = server_cursor_of_int 101 }) - in - check_pull_count "same cursor notification needs no pull" 0 same.effects -;; - -let pull_no_progress_is_delayed_and_bounded () = - let opened, connection, _ = pull_fixture () in - (* The opening response alone can report demand, without Hello/Changed. *) - let state = ref opened.next in - let previous_delay = ref 0. in - for round = 1 to 12 do - let reported_head = if round mod 2 = 1 then 10 else 0 in - let empty = - pull_message - !state - connection - (Protocol.Server.Pull_ok - { t = server_cursor_of_int reported_head; checksum = None; txs = [] }) - in - Alcotest.(check (option cursor_testable)) - "empty response cannot advance durable checkpoint" - (Some (server_cursor_of_int 0)) - (Core.state empty.next).snapshot.applied_server_t; - check_pull_count "no progress cannot immediately repeat the pull" 0 empty.effects; - Alcotest.(check bool) - "pending demand during retry is still Pulling" - true - ((Core.state empty.next).snapshot.sync_phase = Core.Pulling); - Alcotest.(check bool) - "retry delay cannot publish false Current" - true - (List.for_all - (function - | Core.Publish (Core.State_changed state) -> - state.snapshot.sync_phase <> Core.Current - | _ -> true) - empty.effects); - let timer = pull_timer empty.effects in - Alcotest.(check bool) - "retry delay is positive, bounded, and does not decrease" - true - (timer.delay_seconds >= 1. - && timer.delay_seconds <= 30. - && timer.delay_seconds >= !previous_delay); - previous_delay := timer.delay_seconds; - let notified = - pull_message - empty.next - connection - (Protocol.Server.Changed { t = server_cursor_of_int 10 }) - in - check_pull_count "notifications cannot bypass retry delay" 0 notified.effects; - let retried = Core.step notified.next (Core.Timer_elapsed timer.id) in - Alcotest.(check bool) - "retry retains demand and original durable cursor" - true - (pull_sends retried.effects = [ connection, Some (server_cursor_of_int 0) ]); - let duplicate = Core.step retried.next (Core.Timer_elapsed timer.id) in - check_instructions "retry timer is consumed once" [] duplicate.effects; - state := duplicate.next - done -;; - -let pull_timeout_reconnects_and_fences_old_work () = - let opened, connection, _ = pull_fixture () in - let timer = pull_timer opened.effects in - Alcotest.(check bool) - "response timeout is finite and positive" - true - (Float.is_finite timer.delay_seconds && timer.delay_seconds > 0.); - let expired = Core.step opened.next (Core.Timer_elapsed timer.id) in - Alcotest.(check bool) - "response timeout closes its exact connection" - true - (List.exists - (function - | Core.Run (Core.Close_websocket scope) -> scope = connection - | _ -> false) - expired.effects); - let retired = Core.step expired.next (Core.Websocket_retired connection) in - let replacement = recovery_connection retired.effects in - let reopened = Core.step retired.next (Core.Websocket_opened replacement) in - check_pull_count "reconnect starts one fresh baseline pull" 1 reopened.effects; - let old_timeout = Core.step reopened.next (Core.Timer_elapsed timer.id) in - check_instructions "old timeout cannot clear replacement owner" [] old_timeout.effects; - let old_response = pull_response old_timeout.next connection ~after:0 1 in - check_instructions "old connection response is inert" [] old_response.effects; - let changed = - pull_message - old_response.next - replacement - (Protocol.Server.Changed { t = server_cursor_of_int 2 }) - in - check_pull_count "replacement still owns its single pull" 0 changed.effects -;; - -let pull_response_timer_does_not_interrupt_apply () = - let opened, connection, sync = pull_fixture () in - let timer = pull_timer opened.effects in - let received = pull_response opened.next connection ~after:0 1 in - let late_timer = Core.step received.next (Core.Timer_elapsed timer.id) in - check_instructions - "response arrived before timer while apply remains active" - [] - late_timer.effects; - let changed = - pull_message - late_timer.next - connection - (Protocol.Server.Changed { t = server_cursor_of_int 2 }) - in - check_pull_count "notification waits for durable apply" 0 changed.effects; - let committed = - recovery_apply changed.next connection (recovery_sync sync ~checkpoint:1 1 []) - in - Alcotest.(check bool) - "apply completion releases exactly one catchup" - true - (pull_sends committed.effects = [ connection, Some (server_cursor_of_int 1) ]) -;; - -let pull_terminal_outcome_joins_existing_round ~reject () = - let core, connection = submitted_core () in - let response = - if reject - then - Protocol.Server.Tx_reject - { reason = Protocol.Stale - ; t = Some (server_cursor_of_int 1) - ; checksum = Some "0123456789abcdef" - ; success_tx_ids = [] - ; failed_tx_id = None - ; missing_block_uuids = [] - ; error_detail = None - ; data = None - } - else - Protocol.Server.Tx_batch_ok - { t = server_cursor_of_int 1; checksum = Some "0123456789abcdef" } - in - let terminal = pull_message core connection response in - let request = recovery_request terminal.effects in - let sync = - Overlay.sync_view - ~token:request.expected - ~checkpoint:(recovery_cursor 0) - ~submissions:[] - in - let committed = Core.step terminal.next (recovery_commit request sync None) in - check_pull_count - "terminal outbox completion joins already outstanding pull" - 0 - committed.effects; - let empty = pull_response committed.next connection ~after:0 0 in - check_pull_count "unresolved acceptance or rejection target cannot spin" 0 empty.effects; - let timer = pull_timer empty.effects in - let catchup = Core.step empty.next (Core.Timer_elapsed timer.id) in - Alcotest.(check bool) - "terminal outcome target survives until catchup" - true - (pull_sends catchup.effects = [ connection, Some (server_cursor_of_int 0) ]) -;; - -let pull_terminal_barrier_already_applied_is_current ~reject () = - List.iter - (fun through -> - let core, connection = submitted_core () in - let received = pull_response core connection ~after:0 1 in - let sync = - Overlay.sync_view - ~token:(Overlay.sync_token_of_string "sync-token:v1:F5-phase" |> Result.get_ok) - ~checkpoint:(recovery_cursor 1) - ~submissions:[] - in - (* This is a public durable completion fact, not a SQLite/checksum test. *) - let applied = recovery_apply received.next connection sync in - let response = - if reject - then - Protocol.Server.Tx_reject - { reason = Protocol.Stale - ; t = Some through - ; checksum = Some "0123456789abcdef" - ; success_tx_ids = [] - ; failed_tx_id = None - ; missing_block_uuids = [] - ; error_detail = None - ; data = None - } - else - Protocol.Server.Tx_batch_ok { t = through; checksum = Some "0123456789abcdef" } - in - let terminal = pull_message applied.next connection response in - let request = recovery_request terminal.effects in - let completed = Core.step terminal.next (recovery_commit request sync None) in - check_pull_count "already applied barrier needs no new pull" 0 completed.effects; - Alcotest.(check bool) - "completed barrier remains Current without a future response" - true - ((Core.state completed.next).snapshot.sync_phase = Core.Current); - Alcotest.(check bool) - "published terminal phase matches the settled owner" - true - (List.exists - (function - | Core.Publish (Core.State_changed state) -> - state.snapshot.sync_phase = Core.Current - | _ -> false) - completed.effects)) - [ server_cursor_of_int 0; server_cursor_of_int 1 ] -;; - -let pull_shutdown_and_graph_retirement_cancel_demand () = - let opened, connection, _ = pull_fixture () in - let timer = pull_timer opened.effects in - List.iter - (fun event -> - let retired = Core.step opened.next event in - let late_timer = Core.step retired.next (Core.Timer_elapsed timer.id) in - check_instructions "retired graph timer is inert" [] late_timer.effects; - let late_changed = - pull_message - late_timer.next - connection - (Protocol.Server.Changed { t = server_cursor_of_int 20 }) - in - check_instructions "retired graph demand is inert" [] late_changed.effects; - let late_response = pull_response late_changed.next connection ~after:0 1 in - check_instructions "retired graph response is inert" [] late_response.effects) - [ Core.Shutdown; Core.Graph_picker_requested ] -;; - -(* The existing explicit-online event is the shared manual-sync entry. *) -let manual_sync core = - Core.step core (Core.Online_recovery_requested (timeline_scope core)) -;; - -let manual_sync_online_forces_a_fresh_pull () = - let opened, connection, _ = pull_fixture () in - let current = pull_response opened.next connection ~after:0 0 in - let requested = manual_sync current.next in - check_pull_count "manual online sync sends a fresh pull" 1 requested.effects; - Alcotest.(check bool) "request is not prematurely Current" true - ((Core.state requested.next).snapshot.sync_phase = Core.Pulling); - let completed = pull_response requested.next connection ~after:0 0 in - check_pull_count "one confirmed empty response settles" 0 completed.effects; - Alcotest.(check bool) "confirmed empty pull is Current" true - ((Core.state completed.next).snapshot.sync_phase = Core.Current) -;; - -let manual_sync_busy_force_is_coalesced () = - let opened, connection, _ = pull_fixture () in - let pending = ref opened.next in - for _ = 1 to 50 do - let requested = manual_sync !pending in - check_pull_count "busy gestures never send concurrently" 0 requested.effects; - pending := requested.next - done; - let received = pull_response !pending connection ~after:0 0 in - check_pull_count "force survives old empty response" 1 received.effects; - Alcotest.(check bool) "post-gesture probe remains Pulling" true - ((Core.state received.next).snapshot.sync_phase = Core.Pulling); - let confirmed = pull_response received.next connection ~after:0 0 in - check_pull_count "fifty gestures merge into one probe" 0 confirmed.effects; - Alcotest.(check bool) "new probe confirmation settles" true - ((Core.state confirmed.next).snapshot.sync_phase = Core.Current) -;; - -let manual_sync_waits_for_authoritative_commit () = - let opened, connection, sync = pull_fixture () in - let received = pull_response opened.next connection ~after:0 1 in - let requested = manual_sync received.next in - check_pull_count "applying authoritative data keeps its owner" 0 requested.effects; - let committed = recovery_apply requested.next connection - (recovery_sync sync ~checkpoint:1 1 []) in - Alcotest.(check bool) "probe starts from committed cursor" true - (pull_sends committed.effects = [connection, Some (server_cursor_of_int 1)]); - Alcotest.(check bool) "no intermediate Current publication" false - (List.exists (function Core.Publish (Core.State_changed state) -> - state.snapshot.sync_phase = Core.Current | _ -> false) committed.effects); - let confirmed = pull_response committed.next connection ~after:1 1 in - Alcotest.(check bool) "commit plus fresh response settles" true - ((Core.state confirmed.next).snapshot.sync_phase = Core.Current) -;; - -let manual_sync_offline_uses_existing_admission_and_socket_owner () = - let opened, connection, _ = pull_fixture () in - let current = pull_response opened.next connection ~after:0 0 in - let closed = Core.step current.next - (Core.Websocket_closed (connection, Core.Recoverable_close { message = Some "synthetic offline"; attention = Requires_attention })) in - let requested = manual_sync closed.next in - check_pull_count "offline never sends through old socket" 0 requested.effects; - (* Application's existing online request re-verifies the native identity. *) - let admitted = Core.step requested.next - (Core.Account_authenticated {user_id=Some "user"}) in - Alcotest.(check bool) "replacement waits for physical retirement" false - (List.exists (function Core.Run (Core.Start_websocket _) -> true | _ -> false) admitted.effects); - let retired = Core.step admitted.next (Core.Websocket_retired connection) in - let fresh = recovery_connection retired.effects in - let connected = Core.step retired.next (Core.Websocket_opened fresh) in - check_pull_count "recovered connection owns the single baseline pull" 1 connected.effects; - let late = pull_response connected.next connection ~after:0 5 in - check_pull_count "old socket response is fenced" 0 late.effects; - let completed = pull_response late.next fresh ~after:0 0 in - check_pull_count "recovery does not queue an extra manual pull" 0 completed.effects; - Alcotest.(check bool) "reconnected confirmation settles" true - ((Core.state completed.next).snapshot.sync_phase = Core.Current) -;; - -let manual_sync_scope_and_retirement_fence_force () = - let opened, connection, _ = pull_fixture () in - let current = pull_response opened.next connection ~after:0 0 in - let scope = timeline_scope current.next in - let stale = {scope with startup={scope.startup with - presentation_generation=scope.startup.presentation_generation+1}} in - let ignored = Core.step current.next (Core.Online_recovery_requested stale) in - check_instructions "stale manual intent is inert" [] ignored.effects; - let requested = manual_sync opened.next in - List.iter (fun event -> - let retired = Core.step requested.next event in - let late = pull_response retired.next connection ~after:0 1 in - check_pull_count "retired force cannot restart pull" 0 late.effects) - [Core.Graph_picker_requested; Core.Shutdown; - Core.Online_cancelled (startup_scope requested.next)] -;; - -let manual_sync_does_not_bypass_submission_owner () = - let submitted, connection, sync, batch = submitted_delete_recovery_fixture () in - let requested = manual_sync submitted.next in - check_pull_count "manual force waits for shared submission response" 0 requested.effects; - Alcotest.(check bool) "submission remains active" true - ((Core.state requested.next).snapshot.sync_phase = Core.Submitting); - let accepted = pull_message requested.next connection - (Protocol.Server.Tx_batch_ok {t=server_cursor_of_int 1;checksum=Some "0123456789abcdef"}) in - let request = recovery_request accepted.effects in - let accepted_sync = recovery_sync sync 3 - (List.map (fun (item : Overlay.submission_descriptor) -> - {item with state=Overlay.Accepted_pending_authoritative (Overlay.submission_batch_id batch)}) - (Overlay.sync_view_submissions sync)) in - let applied = Core.step accepted.next (recovery_commit request accepted_sync None) in - check_pull_count "acceptance joins the still-outstanding baseline pull" 0 applied.effects; - let received = pull_response applied.next connection ~after:0 1 in - let committed = recovery_apply received.next connection - (recovery_sync sync ~checkpoint:1 4 []) in - check_pull_count "submission and manual force drain through shared commit" 1 committed.effects -;; - -let pull_singleflight_scenarios = - [ Alcotest.test_case "manual sync online fresh pull" `Quick manual_sync_online_forces_a_fresh_pull - ; Alcotest.test_case "manual sync busy force coalesces" `Quick manual_sync_busy_force_is_coalesced - ; Alcotest.test_case "manual sync waits for commit" `Quick manual_sync_waits_for_authoritative_commit - ; Alcotest.test_case "manual sync offline admission and owner" `Quick manual_sync_offline_uses_existing_admission_and_socket_owner - ; Alcotest.test_case "manual sync stale and retired scope" `Quick manual_sync_scope_and_retirement_fence_force - ; Alcotest.test_case "manual sync preserves submission owner" `Quick manual_sync_does_not_bypass_submission_owner - ; Alcotest.test_case - "F5 opening and notification burst share one pull" - `Quick - pull_open_and_notifications_are_singleflight - ; Alcotest.test_case - "F5 newest demand survives authoritative apply" - `Quick - pull_demand_survives_authoritative_apply - ; Alcotest.test_case - "F5 no progress has bounded reducer retry delay" - `Quick - pull_no_progress_is_delayed_and_bounded - ; Alcotest.test_case - "F5 response timeout reconnects and fences old work" - `Quick - pull_timeout_reconnects_and_fences_old_work - ; Alcotest.test_case - "F5 response timer cannot interrupt durable apply" - `Quick - pull_response_timer_does_not_interrupt_apply - ; Alcotest.test_case - "F5 acceptance joins an outstanding pull" - `Quick - (pull_terminal_outcome_joins_existing_round ~reject:false) - ; Alcotest.test_case - "F5 rejection joins an outstanding pull" - `Quick - (pull_terminal_outcome_joins_existing_round ~reject:true) - ; Alcotest.test_case - "F5 completed acceptance barrier is Current" - `Quick - (pull_terminal_barrier_already_applied_is_current ~reject:false) - ; Alcotest.test_case - "F5 completed rejection barrier is Current" - `Quick - (pull_terminal_barrier_already_applied_is_current ~reject:true) - ; Alcotest.test_case - "F5 shutdown and picker retire pending pull demand" - `Quick - pull_shutdown_and_graph_retirement_cancel_demand - ] -;; - -let sync_recovery_reproductions = - [ Alcotest.test_case - "outbox failures are correlated and terminal" - `Quick - recovery_outbox_failure_is_correlated - ; Alcotest.test_case - "retry commit failure is visible" - `Quick - recovery_retry_failure_is_visible - ; Alcotest.test_case - "live submitted delete resumes after acknowledgement" - `Quick - live_submitted_delete_resumes_after_acknowledgement - ; Alcotest.test_case - "M02 submitted delete survives disconnect and reconnect" - `Quick - submitted_delete_survives_disconnect_reconnect - ; Alcotest.test_case - "M02 submitted delete survives fresh Core restore" - `Quick - submitted_delete_survives_fresh_core_restore - ; Alcotest.test_case - "recovery Stale rejection resumes deletion" - `Quick - recovery_stale_rejection_resumes_deletion - ; Alcotest.test_case - "empty pull recovers Submitted before Queued" - `Quick - recovery_empty_pull_prioritizes_submitted - ; Alcotest.test_case - "restored acceptance barrier blocks Queued" - `Quick - (recovery_barrier_blocks_queued ~rejected:false) - ; Alcotest.test_case - "restored rejection barrier blocks Queued" - `Quick - (recovery_barrier_blocks_queued ~rejected:true) - ; Alcotest.test_case - "interrupted retry and old response are fenced" - `Quick - recovery_stale_connection_and_interrupted_retry - ; Alcotest.test_case - "missing batch response expires its connection" - `Quick - recovery_submission_timeout - ] - @ pull_singleflight_scenarios -;; - -let rejected_submission_feedback () = - let core, connection = submitted_core () in - let rejection : Protocol.rejection = - { reason = Db_transact_failed - ; t = Some (server_cursor_of_int 0) - ; checksum = None - ; success_tx_ids = [] - ; failed_tx_id = Some (List.hd mutation_ids) - ; missing_block_uuids = [] - ; error_detail = Some "transaction validation failed" - ; data = None - } - in - let rejected = - Core.step - core - (Core.Websocket_message (connection, Protocol.Server.Tx_reject rejection)) - in - let message = (Core.state rejected.next).snapshot.last_error in - Alcotest.(check bool) "definitive rejection is visible" true (Option.is_some message); - Alcotest.(check bool) - "rejection publishes feedback" - true - (List.exists - (function - | Core.Publish (Core.State_changed state) -> state.snapshot.last_error = message - | _ -> false) - rejected.effects); - Alcotest.(check bool) - "rejection does not turn a usable graph into startup failure" - true - ((Core.state rejected.next).snapshot.startup.failure = None); - let request = recovery_request rejected.effects in - let submissions = - List.map - (fun mutation_id -> - { Overlay.mutation_id - ; fingerprint = - Overlay.Mutation_fingerprint.of_string - ("mutation-fingerprint:v1:" - ^ Logseq_db_types.Graph_types.Uuid.to_string mutation_id) - |> Result.get_ok - ; state = Overlay.Blocked - ; operation = Overlay.Save_block_operation - ; dependency_eligible = false - ; attempt_count = 1 - ; plaintext_bytes = 16 - ; protected_bytes = Some 16 - }) - mutation_ids - in - let sync = - Overlay.sync_view - ~token:(Overlay.sync_token_of_string "sync-token:v1:feedback" |> Result.get_ok) - ~checkpoint:(recovery_cursor 0) - ~submissions - in - let committed = Core.step rejected.next (recovery_commit request sync None) in - let pulled = - Core.step - committed.next - (Core.Websocket_message - ( connection - , Protocol.Server.Pull_ok - { t = server_cursor_of_int 0; checksum = None; txs = [] } )) - in - Alcotest.(check bool) - "empty catch-up does not hide the rejected write" - true - (Option.is_some (Core.state pulled.next).snapshot.last_error); - require_no_new_submission pulled.effects; - let restored, restored_connection = restore_recovery_core sync in - let caught_up = - Core.step - restored - (Core.Websocket_message - ( restored_connection - , Protocol.Server.Pull_ok - { t = server_cursor_of_int 0; checksum = None; txs = [] } )) - in - Alcotest.(check bool) - "restored blocked writes retain visible feedback" - true - (Option.is_some (Core.state caught_up.next).snapshot.last_error); - require_no_new_submission caught_up.effects -;; - -let stale_rejection_is_not_a_failed_save_notice () = - let core, connection = submitted_core () in - let rejection : Protocol.rejection = - { reason = Stale - ; t = Some (server_cursor_of_int 1) - ; checksum = None - ; success_tx_ids = [] - ; failed_tx_id = None - ; missing_block_uuids = [] - ; error_detail = None - ; data = None - } - in - let transition = - Core.step - core - (Core.Websocket_message (connection, Protocol.Server.Tx_reject rejection)) - in - Alcotest.(check (option string)) - "stale catch-up is recoverable" - None - (Core.state transition.next).snapshot.last_error; - Alcotest.(check bool) - "stale transition remains delegated" - true - (match rejection_transition transition.effects with - | Overlay.Stale _ -> true - | _ -> false) -;; - -let decimal_boundary_pull_is_admitted () = - List.iter - (fun (first, last) -> - let checkpoint = recovery_cursor (first - 1) in - let sync = - Overlay.sync_view - ~token:(token Overlay.sync_token_of_string "sync-token:v1:numeric-cursor") - ~checkpoint - ~submissions:[] - in - let core, connection = restore_recovery_core sync in - let txs = - List.map - (fun t -> - Protocol.Server. - { t = server_cursor_of_int t; tx = "[]"; outliner_op = None }) - [ first; last ] - in - let wire = - Protocol.encode_server_message - (Protocol.Server.Pull_ok - { t = server_cursor_of_int last; checksum = None; txs }) - |> Result.get_ok - in - let message = Protocol.decode_server_message wire |> Result.get_ok in - let received = Core.step core (Core.Websocket_message (connection, message)) in - Alcotest.(check bool) - "normal numeric pull reaches authoritative apply" - true - (List.exists - (function - | Core.Delegate (Core.Apply_authoritative_batch _) -> true - | _ -> false) - received.effects); - Alcotest.(check bool) - "normal numeric pull does not fail" - false - ((Core.state received.next).snapshot.sync_phase = Core.Failed)) - [ 9, 10; 99, 100; 999, 1000 ] -;; - -let unordered_pull_is_rejected () = - List.iter - (fun values -> - let opened, connection, _ = pull_fixture () in - let txs = - List.map - (fun t -> - Protocol.Server. - { t = server_cursor_of_int t; tx = "[]"; outliner_op = None }) - values - in - let received = - pull_message - opened.next - connection - (Protocol.Server.Pull_ok - { t = server_cursor_of_int (List.hd (List.rev values)) - ; checksum = None - ; txs - }) - in - Alcotest.(check bool) - "duplicate or reversed pull fails" - true - ((Core.state received.next).snapshot.sync_phase = Core.Failed)) - [ [ 9; 9 ]; [ 10; 9 ] ] -;; - -let scenarios = - [ Alcotest.test_case - "snapshot baseline retains HTTP response budget" - `Quick - snapshot_baseline_decoder_retains_http_budget - ; Alcotest.test_case - "numeric cursor decimal boundaries" - `Quick - decimal_boundary_pull_is_admitted - ; Alcotest.test_case - "numeric cursor duplicate and reverse order" - `Quick - unordered_pull_is_rejected - ; Alcotest.test_case "rejected submission feedback" `Quick rejected_submission_feedback - ; Alcotest.test_case - "stale rejection remains recoverable" - `Quick - stale_rejection_is_not_a_failed_save_notice - ; Alcotest.test_case - "pure reducer canonical overlay happy path" - `Quick - canonical_overlay_happy_path - ; Alcotest.test_case - "selected graph survives picker and codec restart" - `Quick - selected_graph_survives_picker_and_codec_restart - ; Alcotest.test_case - "invalid cached selections fail closed" - `Quick - invalid_cached_selections_fail_closed - ; Alcotest.test_case - "same-account authentication preserves pending warm restore" - `Quick - same_account_authentication_preserves_pending_warm_restore - ; Alcotest.test_case - "catalog refresh preserves or revokes selection" - `Quick - catalog_refresh_preserves_or_revokes_selection - ; Alcotest.test_case - "catalog save failure keeps selected graph usable" - `Quick - catalog_save_failure_keeps_selected_graph_usable - ; Pure_reducer_bad_case_01.scenario - ; Pure_reducer_bad_case_02.scenario - ; Pure_reducer_bad_case_03.scenario - ; Pure_reducer_bad_case_04.scenario - ; Pure_reducer_bad_case_05.scenario - ; Pure_reducer_bad_case_06.scenario - ; Pure_reducer_bad_case_07.scenario - ; Pure_reducer_bad_case_08.scenario - ; Pure_reducer_bad_case_09.scenario - ; Pure_reducer_bad_case_10.scenario - ; Alcotest.test_case "config is bounded" `Quick config_is_bounded - ; Alcotest.test_case - "authentication delegates catalog transport" - `Quick - authentication_delegates_catalog_transport - ; Alcotest.test_case - "websocket connect uses the deployed graph path" - `Quick - websocket_connect_uses_deployed_graph_path - ; Alcotest.test_case - "sign-out captures identity and fences old work" - `Quick - sign_out_captures_identity_advances_generations_and_fences_old_work - ; Alcotest.test_case - "sign-out cleanup completion is inert and sanitized" - `Quick - sign_out_cleanup_completion_is_inert_and_sanitized - ; Alcotest.test_case - "local cache deletion rejects unopened graphs" - `Quick - local_cache_deletion_rejects_unopened_graphs - ; Alcotest.test_case - "deletion discards unsynchronized work" - `Quick - local_deletion_discards_unsynchronized_work - ; Alcotest.test_case - "plain local deletion ordering" - `Quick - (local_deletion_orders_cleanup false) - ; Alcotest.test_case - "encrypted local deletion ordering" - `Quick - (local_deletion_orders_cleanup true) - ; Alcotest.test_case - "plain local deletion failures" - `Quick - (local_deletion_failures_stop false) - ; Alcotest.test_case - "encrypted local deletion failures" - `Quick - (local_deletion_failures_stop true) - ; Alcotest.test_case - "local deletion account change fences completion" - `Quick - local_deletion_account_change_fences_completion - ; Alcotest.test_case - "cold start absent mirror reaches pull after snapshot activation" - `Quick - cold_start_absent_mirror_reaches_pull_after_snapshot_activation - ; Alcotest.test_case - "invalid snapshot metadata fails bootstrap" - `Quick - invalid_snapshot_metadata_fails_bootstrap - ; Alcotest.test_case - "E2EE password retry after failed unlock" - `Quick - e2ee_password_retry_recovers_after_failed_unlock - ; Alcotest.test_case - "encrypted recovery unlocks graph key before bootstrap" - `Quick - encrypted_graph_recovery_fetches_and_unlocks_key_before_bootstrap - ; Alcotest.test_case - "warm encrypted graph loads key before attach and protects outbox" - `Quick - warm_encrypted_graph_loads_key_before_attach_and_protects_outbox - ; Alcotest.test_case - "graph picker clears terminal graph failure" - `Quick - graph_picker_clears_terminal_graph_failure - ; Alcotest.test_case - "graph picker fences late warm key completion" - `Quick - graph_picker_fences_late_warm_key_completion - ; Alcotest.test_case - "warm encrypted graph online recovery attaches existing mirror" - `Quick - warm_encrypted_graph_online_recovery_attaches_existing_mirror - ; Alcotest.test_case - "partial rejection is normalized exactly" - `Quick - partial_rejection_is_normalized_exactly - ; Alcotest.test_case - "local outbox change requests sync inspection" - `Quick - local_outbox_change_requests_sync_inspection - ; Alcotest.test_case - "accepted submission pulls its authoritative transaction" - `Quick - accepted_submission_pulls_its_authoritative_transaction - ; Alcotest.test_case - "whole-batch invalid tx partitions every member" - `Quick - whole_batch_invalid_tx_partitions_every_member - ] -;; - -(* Core owns admission and acceptance; filesystem execution is tested by the runner. *) -let asset_version = - Logseq_db_types.Asset_descriptor.version ~checksum:(String.make 64 'a') ~file_type:"bin" - |> Result.get_ok -;; - -let asset_fixture () = - let selected, scope = selected_graph graph in - selected.next, scope -;; - -let asset_issue core scope operation action = - let request : Core.asset_request = { scope; operation; action } in - let transition = Core.step core (Core.Asset_requested request) in - let ticket = - List.find_map - (function - | Core.Run (Core.Asset_io (ticket, io)) when io.operation = operation -> - Some ticket - | _ -> None) - transition.effects - |> Option.get - in - transition, ticket, request -;; - -let asset_outputs effects = - List.filter_map - (function - | Core.Publish (Core.Asset_finished output) -> Some output - | _ -> None) - effects -;; - -let asset_complete core ticket result = - Core.step core (Core.Runner_completed (Core.Asset_completion (ticket, result))) -;; - -let asset_accepts_exact_identity_once () = - let core, scope = asset_fixture () in - let first, ticket, request = - asset_issue core scope "cache-1" (Core.Check_asset_cache (graph_id, asset_version)) - in - Alcotest.(check int) - "request emits no premature result" - 0 - (List.length (asset_outputs first.effects)); - let second, other, other_request = - asset_issue - first.next - scope - "cache-2" - (Core.Check_asset_cache (graph_id, asset_version)) - in - let completed = asset_complete second.next other (Ok (Core.Asset_cached None)) in - Alcotest.(check bool) - "second ticket resolves only second operation" - true - (asset_outputs completed.effects - = [ { Core.request = other_request; result = Ok (Core.Asset_cached None) } ]); - let first_done = asset_complete completed.next ticket (Ok (Core.Asset_cached None)) in - Alcotest.(check bool) - "first ticket remains pending" - true - (asset_outputs first_done.effects - = [ { Core.request; result = Ok (Core.Asset_cached None) } ]); - let repeated = asset_complete first_done.next ticket (Ok (Core.Asset_cached None)) in - check_instructions "asset completion replay" [] repeated.effects -;; - -let asset_rejects_foreign_scope_and_duplicate_request () = - let core, scope = asset_fixture () in - let pending, _, _ = - asset_issue - core - scope - "same" - (Core.Fetch_asset - { asset = graph_id; version = asset_version; maximum_plaintext_bytes = 32 }) - in - let duplicate = - Core.step - pending.next - (Core.Asset_requested - { scope - ; operation = "same" - ; action = - Core.Fetch_asset - { asset = graph_id; version = asset_version; maximum_plaintext_bytes = 32 } - }) - in - Alcotest.(check bool) - "duplicate request does not execute twice" - false - (List.exists - (function - | Core.Run (Core.Asset_io _) -> true - | _ -> false) - duplicate.effects); - let foreign_scopes = - [ { scope with graph_generation = scope.graph_generation + 1 } - ; { scope with graph_id = other_graph_id } - ; { scope with - account = - { scope.account with account_generation = scope.account.account_generation + 1 } - } - ; { scope with - account = - { scope.account with - presentation_generation = scope.account.presentation_generation + 1 - } - } - ; { scope with - account = - { scope.account with - lifecycle_generation = Int64.succ scope.account.lifecycle_generation - } - } - ; { scope with account = { scope.account with user_id = "other-account" } } - ; { scope with - account = - { scope.account with - managed_sync_origin = Uri.of_string "https://other-origin.invalid" - } - } - ] - in - List.iter - (fun foreign_scope -> - let rejected = - Core.step - duplicate.next - (Core.Asset_requested - { scope = foreign_scope - ; operation = "foreign" - ; action = - Core.Fetch_asset - { asset = graph_id - ; version = asset_version - ; maximum_plaintext_bytes = 32 - } - }) - in - Alcotest.(check bool) - "foreign scope identity cannot execute IO" - false - (List.exists - (function - | Core.Run (Core.Asset_io _) -> true - | _ -> false) - rejected.effects)) - foreign_scopes -;; - -let asset_failure_is_published_once () = - let core, scope = asset_fixture () in - let pending, ticket, request = - asset_issue - core - scope - "failed" - (Core.Put_asset_file - { asset = graph_id - ; version = asset_version - ; file = "resource" - ; maximum_plaintext_bytes = 32 - }) - in - let failed = asset_complete pending.next ticket (Error Core.Asset_network) in - Alcotest.(check bool) - "typed failure preserves request identity" - true - (asset_outputs failed.effects - = [ { Core.request; result = Error Core.Asset_network } ]); - let repeated = asset_complete failed.next ticket (Error Core.Asset_network) in - check_instructions "asset failure replay" [] repeated.effects -;; - -let asset_cancelled_completion_cleans_resources () = - List.iter - (fun (action, value, cleanup) -> - let core, scope = asset_fixture () in - let pending, ticket, _ = asset_issue core scope "cancelled" action in - let cancelled, _, _ = - asset_issue pending.next scope "cancel" (Core.Cancel_asset_operation "cancelled") - in - let late = asset_complete cancelled.next ticket (Ok value) in - Alcotest.(check int) - "cancelled work never publishes success" - 0 - (List.length (asset_outputs late.effects)); - Alcotest.(check bool) - "cancelled completion releases its resource" - true - (List.exists - (function - | Core.Run (Core.Asset_io (_, io)) -> io.action = cleanup - | _ -> false) - late.effects)) - [ ( Core.Fetch_asset - { asset = graph_id; version = asset_version; maximum_plaintext_bytes = 32 } - , Core.Asset_downloaded "file" - , Core.Release_asset_file "file" ) - ; ( Core.Stage_asset_file - { operation = graph_id; file_type = "bin"; source_file = "source" } - , Core.Asset_staged { file = "stage"; checksum = String.make 64 'a'; size = 4L } - , Core.Release_staged_file "stage" ) - ; ( Core.Retain_asset_file "file" - , Core.Asset_retained (Some ("lease", "path")) - , Core.Release_asset_file "lease" ) - ] -;; - -let asset_scope_switch_cleans_late_resources () = - List.iter - (fun (action, result, cleanup) -> - let core, scope = asset_fixture () in - let pending, ticket, _ = asset_issue core scope "old-resource" action in - let picker = Core.step pending.next Core.Graph_picker_requested in - let late = asset_complete picker.next ticket (Ok result) in - Alcotest.(check int) - "old graph publishes no resource" - 0 - (List.length (asset_outputs late.effects)); - Alcotest.(check bool) - "old graph resource is released via runner effect" - true - (List.exists - (function - | Core.Run (Core.Asset_io (_, io)) -> io.action = cleanup - | _ -> false) - late.effects)) - [ ( Core.Fetch_asset - { asset = graph_id; version = asset_version; maximum_plaintext_bytes = 32 } - , Core.Asset_downloaded "late-file" - , Core.Release_asset_file "late-file" ) - ; ( Core.Stage_asset_file - { operation = graph_id; file_type = "bin"; source_file = "source" } - , Core.Asset_staged - { file = "late-stage"; checksum = String.make 64 'a'; size = 4L } - , Core.Release_staged_file "late-stage" ) - ; ( Core.Retain_asset_file "file" - , Core.Asset_retained (Some ("late-lease", "path")) - , Core.Release_asset_file "late-lease" ) - ] -;; - -let asset_mismatched_completion_cleans_resource () = - let core, scope = asset_fixture () in - let pending, ticket, _ = - asset_issue core scope "cache" (Core.Check_asset_cache (graph_id, asset_version)) - in - let wrong = - asset_complete - pending.next - ticket - (Ok - (Core.Asset_staged - { file = "unexpected-stage"; checksum = String.make 64 'a'; size = 4L })) - in - Alcotest.(check bool) - "wrong resource kind is not accepted" - true - (match asset_outputs wrong.effects with - | [ { result = Error (Core.Asset_invalid_content _); _ } ] -> true - | _ -> false); - Alcotest.(check bool) - "wrong resource is cleaned through runner" - true - (List.exists - (function - | Core.Run (Core.Asset_io (_, io)) -> - io.action = Core.Release_staged_file "unexpected-stage" - | _ -> false) - wrong.effects) -;; - -let asset_retry_completion_is_consumed_once () = - let core, scope = asset_fixture () in - let pending, ticket, request = - asset_issue core scope "retry" (Core.Asset_retry_after 0.1) - in - let elapsed = asset_complete pending.next ticket (Ok Core.Asset_retry_elapsed) in - Alcotest.(check bool) - "timer returns to requesting owner" - true - (asset_outputs elapsed.effects - = [ { Core.request; result = Ok Core.Asset_retry_elapsed } ]); - let duplicate = asset_complete elapsed.next ticket (Ok Core.Asset_retry_elapsed) in - check_instructions "asset timer replay" [] duplicate.effects -;; - -let asset_scope_close_keeps_pending_release () = - List.iter(fun action-> - let core,scope=asset_fixture() in - let releasing,ticket,request=asset_issue core scope "preview-release" action in - let acquiring,_,acquire=asset_issue releasing.next scope "preview-retain" (Core.Retain_asset_file "preview") in - let closing,_,_=asset_issue acquiring.next scope "scope-close" Core.Close_asset_scope in - Alcotest.(check bool) "closing a scope cannot cancel its physical release ACK" false - (List.exists(fun(output:Core.asset_output)->output.request.operation=request.operation) - (asset_outputs closing.effects)); - Alcotest.(check bool) "closing still cancels an in-flight acquisition" true - (List.exists(fun(output:Core.asset_output)->output.request=acquire && output.result=Error Core.Asset_cancelled) - (asset_outputs closing.effects)); - let completed=asset_complete closing.next ticket(Ok Core.Asset_unit) in - Alcotest.(check bool) "actual release completion reaches its original owner" true - (asset_outputs completed.effects=[{Core.request;result=Ok Core.Asset_unit}]); - check_instructions "release terminal is consumed once" [] - (asset_complete completed.next ticket(Ok Core.Asset_unit)).effects) - [Core.Release_asset_file "preview";Core.Release_staged_file "staged-preview"] -;; - -let asset_cleanup_requires_admitted_scope () = - let core, scope = asset_fixture () in - let foreign = - { scope with account = { scope.account with user_id = "forged-account" } } - in - List.iter - (fun action -> - let rejected = - Core.step - core - (Core.Asset_requested { scope = foreign; operation = "forged-cleanup"; action }) - in - Alcotest.(check bool) - "foreign account cannot execute destructive or resource cleanup" - false - (List.exists - (function - | Core.Run (Core.Asset_io _) -> true - | _ -> false) - rejected.effects); - Alcotest.(check bool) - "rejected cleanup reports its own cancelled identity" - true - (match asset_outputs rejected.effects with - | [ output ] -> - output.request.scope = foreign && output.result = Error Core.Asset_cancelled - | _ -> false)) - [ Core.Release_asset_file "resource" - ; Core.Release_staged_file "stage" - ; Core.Close_asset_scope - ; (Core.Prepare_graph_asset_deletion (Core.retirement_id_of_string "96000000-0000-4000-8000-000000000001" |> Result.get_ok)) - ; Core.Delete_account_assets - ; Core.Cancel_asset_operation "pending" - ]; - let picker = Core.step core Core.Graph_picker_requested in - List.iter - (fun action -> - let accepted = - Core.step - picker.next - (Core.Asset_requested { scope; operation = "known-cleanup"; action }) - in - Alcotest.(check bool) - "previously admitted scope still permits required cleanup" - true - (List.exists - (function - | Core.Run (Core.Asset_io (_, io)) -> - io.context.scope = scope && io.action = action - | _ -> false) - accepted.effects)) - [ Core.Release_asset_file "resource" - ; Core.Release_staged_file "stage" - ; Core.Close_asset_scope - ] -;; - -let asset_deletion_cancels_stage core scope deletion_scope action = - let pending, ticket, request = - asset_issue - core - scope - "stage-before-deletion" - (Core.Stage_asset_file - { operation = graph_id; file_type = "bin"; source_file = "source" }) - in - let deleting, _, _ = asset_issue pending.next deletion_scope "delete-assets" action in - let late = - asset_complete - deleting.next - ticket - (Ok - (Core.Asset_staged - { file = "late-deleted-stage"; checksum = String.make 64 'a'; size = 4L })) - in - Alcotest.(check int) - "deleted asset operation cannot publish late staging success" - 0 - (List.length (asset_outputs late.effects)); - Alcotest.(check bool) - "deletion cancels the exact pending staging request" - true - (asset_outputs deleting.effects - = [ { Core.request; result = Error Core.Asset_cancelled } ]); - Alcotest.(check bool) - "late staging completion cleans through the original resource scope" - true - (List.exists - (function - | Core.Run (Core.Asset_io (_, io)) -> - io.context.scope = scope - && io.action = Core.Release_staged_file "late-deleted-stage" - | _ -> false) - late.effects); - let repeated = - asset_complete - late.next - ticket - (Ok - (Core.Asset_staged - { file = "late-deleted-stage"; checksum = String.make 64 'a'; size = 4L })) - in - check_instructions "deleted staging completion replay" [] repeated.effects -;; - -let asset_graph_deletion_cancels_other_generation () = - let core, scope = asset_fixture () in - asset_deletion_cancels_stage - core - scope - { scope with graph_generation = 0 } - (Core.Prepare_graph_asset_deletion (Core.retirement_id_of_string "96000000-0000-4000-8000-000000000001" |> Result.get_ok)) -;; - -let asset_account_deletion_cancels_other_known_graph () = - let core, old_scope = asset_fixture () in - let catalogued = refresh_catalog core [ graph; other_graph ] in - let picker = Core.step catalogued.next Core.Graph_picker_requested in - let selected = Core.step picker.next (Core.Graph_selected other_graph_id) in - let current_scope = Core.admitted_graph_scope selected.next |> Option.get in - Alcotest.(check bool) - "account deletion fixture keeps the same account across graph selection" - true - (Uri.equal - current_scope.account.managed_sync_origin - old_scope.account.managed_sync_origin - && String.equal current_scope.account.user_id old_scope.account.user_id - && current_scope.graph_id <> old_scope.graph_id); - asset_deletion_cancels_stage - selected.next - current_scope - old_scope - Core.Delete_account_assets -;; - -let completed_retirement_cannot_cancel_replacement_work () = - let core, scope = asset_fixture () in - let retirement = Core.retirement_id_of_string "96000000-0000-4000-8000-000000000003" |> Result.get_ok in - let deletion = Core.asset_deletion_reference ~scope ~retirement in - let preparing, ticket, _ = asset_issue core scope "prepare-old" (Core.Prepare_graph_asset_deletion retirement) in - let prepared = asset_complete preparing.next ticket (Ok (Core.Asset_deletion_prepared deletion)) in - let committing, ticket, _ = asset_issue prepared.next scope "commit-old" (Core.Commit_graph_asset_deletion deletion) in - let committed = asset_complete committing.next ticket (Ok Core.Asset_unit) in - let pending, _, fresh = asset_issue committed.next scope "replacement-fetch" - (Core.Fetch_asset {asset=graph_id;version=asset_version;maximum_plaintext_bytes=8}) in - let replay = Core.step pending.next (Core.Asset_requested - {scope;operation="replay-old-prepare";action=Core.Prepare_graph_asset_deletion retirement}) in - Alcotest.(check bool) "old retirement cannot cancel replacement work" false - (List.exists (fun (output : Core.asset_output) -> output.request = fresh && output.result = Error Core.Asset_cancelled) - (asset_outputs replay.effects)) -;; - -let retirement_replay_preserves_inflight_and_account_cleanup () = - List.iter (fun after_account_cleanup -> - let core, scope = asset_fixture () in - let retirement = Core.retirement_id_of_string "96000000-0000-4000-8000-000000000014" |> Result.get_ok in - let deletion = Core.asset_deletion_reference ~scope ~retirement in - let preparing, ticket, original = asset_issue core scope "initial-prepare" (Core.Prepare_graph_asset_deletion retirement) in - let owner, pending = - if not after_account_cleanup then preparing.next, original - else ( - let prepared = asset_complete preparing.next ticket (Ok (Core.Asset_deletion_prepared deletion)) in - let clearing, ticket, _ = asset_issue prepared.next scope "account-cleanup" Core.Delete_account_assets in - let cleared = asset_complete clearing.next ticket (Ok Core.Asset_unit) in - let fresh, _, request = asset_issue cleared.next scope "new-account-fetch" - (Core.Fetch_asset {asset=graph_id;version=asset_version;maximum_plaintext_bytes=8}) in - fresh.next, request) in - let replay = Core.step owner (Core.Asset_requested - {scope;operation="replay-initial-prepare";action=Core.Prepare_graph_asset_deletion retirement}) in - Alcotest.(check bool) "replay does not cancel the existing owner" false - (List.exists (fun (output : Core.asset_output) -> output.request=pending && output.result=Error Core.Asset_cancelled) - (asset_outputs replay.effects))) [false;true] -;; - -let retirement_terminal_rejects_mismatched_token_scope () = - let core, scope = asset_fixture () in - let foreign = {scope with account={scope.account with user_id="foreign-token-account"}} in - let retirement = Core.retirement_id_of_string "96000000-0000-4000-8000-000000000015" |> Result.get_ok in - let deletion = Core.asset_deletion_reference ~scope:foreign ~retirement in - List.iter (fun action -> - let transition = Core.step core (Core.Asset_requested {scope;operation="foreign-token";action}) in - Alcotest.(check bool) "current scope cannot execute another token's cleanup" false - (List.exists (function Core.Run (Core.Asset_io _) -> true | _ -> false) transition.effects)) - [Core.Commit_graph_asset_deletion deletion;Core.Cancel_graph_asset_deletion deletion] -;; - -let asset_scenarios = - [ Alcotest.test_case "scope close keeps pending physical release" `Quick asset_scope_close_keeps_pending_release - ; Alcotest.test_case "retirement replay preserves inflight and account cleanup" `Quick - retirement_replay_preserves_inflight_and_account_cleanup - ; Alcotest.test_case "retirement token scope is checked for active context" `Quick - retirement_terminal_rejects_mismatched_token_scope - ; Alcotest.test_case "completed retirement preserves replacement work" `Quick - completed_retirement_cannot_cancel_replacement_work - ; Alcotest.test_case - "graph asset deletion cancels pending work across generations" - `Quick - asset_graph_deletion_cancels_other_generation - ; Alcotest.test_case - "account asset deletion cancels pending work on another known graph" - `Quick - asset_account_deletion_cancels_other_known_graph - ; Alcotest.test_case - "asset cleanup rejects foreign account and accepts retired known scope" - `Quick - asset_cleanup_requires_admitted_scope - ; Alcotest.test_case - "asset exact completion identity and replay" - `Quick - asset_accepts_exact_identity_once - ; Alcotest.test_case - "asset foreign scope and duplicate admission" - `Quick - asset_rejects_foreign_scope_and_duplicate_request - ; Alcotest.test_case - "asset failure resolves once" - `Quick - asset_failure_is_published_once - ; Alcotest.test_case - "asset cancellation cleans file staging and lease" - `Quick - asset_cancelled_completion_cleans_resources - ; Alcotest.test_case - "asset scope switch cleans stale resources" - `Quick - asset_scope_switch_cleans_late_resources - ; Alcotest.test_case - "asset mismatched completion cleans resource" - `Quick - asset_mismatched_completion_cleans_resource - ; Alcotest.test_case - "asset retry completion resolves once" - `Quick - asset_retry_completion_is_consumed_once - ] -;; - -(* Reconnect policy belongs to Core. Physical retirement and socket peaks are - checked separately at the public Runner boundary, which executes resources. *) -let websocket_starts effects = - List.filter_map - (function - | Core.Run (Core.Start_websocket r) -> Some r - | _ -> None) - effects -;; - -let websocket_retry_timer effects = - List.find_map - (function - | Core.Run (Core.Schedule_timer r) -> Some r - | _ -> None) - effects - |> function - | Some r -> r - | None -> Alcotest.fail "recoverable close did not schedule retry" -;; - -let websocket_retirement_required scope effects = - Alcotest.(check bool) - "exact owner retirement requested" - true - (List.exists - (function - | Core.Run (Core.Close_websocket s) -> s = scope - | _ -> false) - effects); - Alcotest.(check int) - "replacement waits for retirement" - 0 - (List.length (websocket_starts effects)) -;; - -let reconnect_fixture () = - let fixture, current = Pure_reducer_bad_case_support.current_connected_fixture () in - fixture.connection, current.next -;; - -let reconnect_waits_for_retirement_and_consumes_retry () = - let scope, core = reconnect_fixture () in - let closed = - Core.step core (Core.Websocket_closed (scope, Core.Recoverable_close { message = None; attention = Requires_attention })) - in - websocket_retirement_required scope closed.effects; - List.iter (fun attention -> - let message = Some "Synthetic closed transport" in - let terminated = Core.step core (Core.Websocket_closed (scope, - Core.Recoverable_close { message; attention })) in - let snapshot = (Core.state terminated.next).snapshot in - Alcotest.(check (option string)) "classification preserves exact message" message snapshot.last_error; - Alcotest.(check bool) "typed attention reaches public snapshot" true - (snapshot.last_error_attention = Some attention); - websocket_retirement_required scope terminated.effects; - let retired = Core.step terminated.next (Core.Websocket_retired scope) in - Alcotest.(check (float 0.)) "attention cannot alter retry delay" 1. - (websocket_retry_timer retired.effects).delay_seconds) - [Core.Transient_connection; Core.Requires_attention]; - let duplicate = - Core.step closed.next (Core.Websocket_closed (scope, Core.Recoverable_close { message = None; attention = Requires_attention })) - in - check_instructions "duplicate termination" [] duplicate.effects; - let late = Core.step duplicate.next (Core.Websocket_opened scope) in - check_instructions "late open cannot resurrect retiring owner" [] late.effects; - let retired = Core.step late.next (Core.Websocket_retired scope) in - let timer = websocket_retry_timer retired.effects in - Alcotest.(check (float 0.)) "first retry delay" 1. timer.delay_seconds; - let expired = Core.step retired.next (Core.Timer_elapsed timer.id) in - let replacement = - match websocket_starts expired.effects with - | [ r ] -> r.scope - | _ -> Alcotest.fail "retry must start exactly one owner" - in - Alcotest.(check bool) - "replacement generation is fresh" - true - (replacement.connection_generation > scope.connection_generation); - check_instructions - "duplicate retry timer" - [] - (Core.step expired.next (Core.Timer_elapsed timer.id)).effects; - check_instructions - "duplicate retirement" - [] - (Core.step expired.next (Core.Websocket_retired scope)).effects; - List.iter - (fun event -> - check_instructions - "late old connection event" - [] - (Core.step expired.next event).effects) - [ Core.Websocket_opened scope - ; Core.Websocket_closed (scope, Core.Recoverable_close { message = Some "late"; attention = Requires_attention }) - ; Core.Websocket_message - ( scope - , Logseq_sync_pure_reducer.Sync_protocol.Server.Changed - { t = server_cursor_of_int 42 } ) - ]; - let repeated = - Core.step - expired.next - (Core.Foreground_changed { foreground = true; lifecycle_generation = 1L }) - in - Alcotest.(check int) - "foreground cannot duplicate an opening owner" - 0 - (List.length (websocket_starts repeated.effects)) -;; - -let reconnect_backoff_is_capped_and_not_reset_by_handshake () = - let scope, core = reconnect_fixture () in - let rec attempt scope core = function - | [] -> () - | delay :: rest -> - let failed = - Core.step - core - (Core.Websocket_closed (scope, Core.Recoverable_close { message = Some "EOF"; attention = Requires_attention })) - in - let retired = Core.step failed.next (Core.Websocket_retired scope) in - let timer = websocket_retry_timer retired.effects in - Alcotest.(check (float 0.)) "capped retry backoff" delay timer.delay_seconds; - let opening = Core.step retired.next (Core.Timer_elapsed timer.id) in - let scope = - match websocket_starts opening.effects with - | [ r ] -> r.scope - | _ -> Alcotest.fail "missing retry" - in - let opened = Core.step opening.next (Core.Websocket_opened scope) in - attempt scope opened.next rest - in - attempt scope core [ 1.; 2.; 4.; 8.; 16.; 30.; 30. ] -;; - -let reconnect_terminal_failures_do_not_retry () = - List.iter - (fun reason -> - let scope, core = reconnect_fixture () in - let closed = Core.step core (Core.Websocket_closed (scope, reason)) in - websocket_retirement_required scope closed.effects; - let retired = Core.step closed.next (Core.Websocket_retired scope) in - Alcotest.(check bool) - "terminal close has no automatic retry" - false - (List.exists - (function - | Core.Run (Core.Schedule_timer _ | Core.Start_websocket _) -> true - | _ -> false) - retired.effects)) - [ Core.Local_close - ; Core.Authentication_failure "401" - ; Core.Protocol_failure "bad frame" - ] -;; - -let reconnect_background_shutdown_and_scope_fence () = - let scope, core = reconnect_fixture () in - let closed = - Core.step core (Core.Websocket_closed (scope, Core.Recoverable_close { message = None; attention = Requires_attention })) - in - let retired = Core.step closed.next (Core.Websocket_retired scope) in - let timer = websocket_retry_timer retired.effects in - let background = - Core.step - retired.next - (Core.Foreground_changed { foreground = false; lifecycle_generation = 1L }) - in - Alcotest.(check bool) - "background cancels scoped retry" - true - (List.exists - (function - | Core.Run (Core.Cancel_effects _) -> true - | _ -> false) - background.effects); - check_instructions - "background retry timer is inert" - [] - (Core.step background.next (Core.Timer_elapsed timer.id)).effects; - let foreground = - Core.step - background.next - (Core.Foreground_changed { foreground = true; lifecycle_generation = 2L }) - in - Alcotest.(check int) - "foreground starts one owner" - 1 - (List.length (websocket_starts foreground.effects)); - let duplicate = - Core.step - foreground.next - (Core.Foreground_changed { foreground = true; lifecycle_generation = 3L }) - in - Alcotest.(check int) - "repeated foreground does not dial again" - 0 - (List.length (websocket_starts duplicate.effects)); - let stopped = Core.step retired.next Core.Shutdown in - check_instructions - "shutdown retry timer is inert" - [] - (Core.step stopped.next (Core.Timer_elapsed timer.id)).effects; - let switched = Core.step retired.next Core.Graph_picker_requested in - check_instructions - "old graph retry timer is inert" - [] - (Core.step switched.next (Core.Timer_elapsed timer.id)).effects; - let logged_out = - Core.step retired.next (Core.Account_authenticated { user_id = None }) - in - check_instructions - "old account retry timer is inert" - [] - (Core.step logged_out.next (Core.Timer_elapsed timer.id)).effects -;; - -let reconnect_scope_switch_waits_for_old_opening_owner () = - let fixture = Pure_reducer_bad_case_support.connected_fixture () in - let switched = Core.step fixture.attached.next (Core.Graph_selected graph_id) in - websocket_retirement_required fixture.connection switched.effects; - let request = - List.find_map - (function - | Core.Delegate (Core.Inspect_mirror r) -> Some r - | _ -> None) - switched.effects - |> Option.get - in - let inspected = - Core.step switched.next (Core.Mirror_inspected (Core.Mirror_available request)) - in - let attached = - Core.step - inspected.next - (Core.Graph_attached { scope = request.scope; sync = fixture.attachment.sync }) - in - Alcotest.(check int) - "attachment cannot overlap old setup" - 0 - (List.length (websocket_starts attached.effects)); - check_instructions - "late opening completion is fenced" - [] - (Core.step attached.next (Core.Websocket_opened fixture.connection)).effects; - let wrong = - { fixture.connection with - connection_generation = fixture.connection.connection_generation + 100 - } - in - check_instructions - "unrelated retirement cannot release owner" - [] - (Core.step attached.next (Core.Websocket_retired wrong)).effects; - let retired = Core.step attached.next (Core.Websocket_retired fixture.connection) in - let next = - match websocket_starts retired.effects with - | [ r ] -> r - | _ -> Alcotest.fail "retirement must start latest attached target" - in - Alcotest.(check bool) - "latest graph generation selected" - true - (next.scope.graph = request.scope) -;; - -let scenarios = - scenarios - @ [ Alcotest.test_case - "reconnect waits for retirement and consumes retry" - `Quick - reconnect_waits_for_retirement_and_consumes_retry - ; Alcotest.test_case - "reconnect capped backoff survives short handshakes" - `Quick - reconnect_backoff_is_capped_and_not_reset_by_handshake - ; Alcotest.test_case - "reconnect terminal failures do not retry" - `Quick - reconnect_terminal_failures_do_not_retry - ; Alcotest.test_case - "reconnect background shutdown and scope fencing" - `Quick - reconnect_background_shutdown_and_scope_fence - ; Alcotest.test_case - "reconnect scope switch waits for opening owner" - `Quick - reconnect_scope_switch_waits_for_old_opening_owner - ] -;; - -let reconnect_resets_backoff_after_durable_nonempty_pull () = - let opened, old_connection, sync = pull_fixture () in - let closed = - Core.step - opened.next - (Core.Websocket_closed (old_connection, Core.Recoverable_close { message = None; attention = Requires_attention })) - in - let restarting = restart_after_retirement closed.next old_connection in - let connection = recovery_connection restarting.effects in - let opened = Core.step restarting.next (Core.Websocket_opened connection) in - let received = pull_response opened.next connection ~after:0 1 in - let committed = - recovery_apply received.next connection (recovery_sync sync ~checkpoint:1 1 []) - in - Alcotest.(check bool) - "nonempty opening pull is durably current" - true - ((Core.state committed.next).snapshot.sync_phase = Core.Current); - let closed = - Core.step - committed.next - (Core.Websocket_closed (connection, Core.Recoverable_close { message = None; attention = Requires_attention })) - in - let retired = Core.step closed.next (Core.Websocket_retired connection) in - Alcotest.(check (float 0.)) - "confirmed healthy connection resets reconnect backoff" - 1. - (websocket_retry_timer retired.effects).delay_seconds -;; - -let scenarios = - scenarios - @ [ Alcotest.test_case - "reconnect resets after durable nonempty pull" - `Quick - reconnect_resets_backoff_after_durable_nonempty_pull - ] -;; - -let generated_catalog_cache_rejects_legacy_and_invalid_domain_values () = - let cache = - Core.catalog_cache ~user_id:"user" ~graphs:[ graph ] ~selected_graph:(Some graph_id) - in - let encoded = Core.encode_catalog_cache cache in - let restored = Core.decode_catalog_cache encoded |> Result.get_ok in - Alcotest.(check bool) - "generated cache restores validated graph and selection" - true - (Core.catalog_cache_graphs restored = [ graph ] - && Core.catalog_cache_selected_graph restored = Some graph_id); - let legacy = {|{"userId":"user","graphs":[],"selectedGraph":null}|} in - Alcotest.(check bool) - "old internal cache is explicitly rejected" - true - (Result.is_error (Core.decode_catalog_cache legacy)); - let native_invalid_selection = - {|{"cache_user_id":"user","cache_graphs":[],"cache_selected_graph":"invalid"}|} - in - Alcotest.(check bool) - "invalid optional UUID does not become no selection" - true - (Result.is_error (Core.decode_catalog_cache native_invalid_selection)); - let native_invalid_schema = - {|{"cache_user_id":"user","cache_graphs":[{"graph_id":"11111111-1111-4111-8111-111111111111","name":"Journal","schema":{"major":-1,"minor":0,"exact":true},"encrypted":false}],"cache_selected_graph":null}|} - in - Alcotest.(check bool) - "schema domain constraints remain enforced" - true - (Result.is_error (Core.decode_catalog_cache native_invalid_schema)) -;; - -let scenarios = - scenarios - @ [ Alcotest.test_case - "generated catalog cache rejects old format and invalid domains" - `Quick - generated_catalog_cache_rejects_legacy_and_invalid_domain_values - ] -;; - -(* Core owns challenge validity/recovery. Only its public events and opaque - effect tickets reproduce these state defects; no runner/UI duplication. *) -let e2ee_preparation_fixture () = - let selected, scope = selected_graph encrypted_graph in - let absent = Core.step selected.next (Core.Mirror_inspected (Mirror_absent scope)) in - let cached = List.find_map (function - | Core.Run (Core.Request (ticket, Core.Load_and_unlock_graph_key _)) -> - Some (Core.step absent.next (Core.Runner_completed (Core.Completion - (ticket, Error (Core.Effect_failed "synthetic absent cache"))))) - | _ -> None) absent.effects |> Option.get in - cached, scope -;; - -let e2ee_finish_graph_fetch preparation (result : (string, Core.effect_error) result) = - List.find_map (function - | Core.Run (Core.Request (ticket, Core.Fetch_e2ee_graph_key _)) -> - Some (Core.step preparation.Core.next - (Core.Runner_completed (Core.Completion (ticket, result)))) - | _ -> None) preparation.effects |> Option.get -;; - -let e2ee_finish_user_fetch preparation (result : (string, Core.effect_error) result) = - List.find_map (function - | Core.Run (Core.Request (ticket, Core.Fetch_e2ee_user_keys _)) -> - Some (Core.step preparation.Core.next - (Core.Runner_completed (Core.Completion (ticket, result)))) - | _ -> None) preparation.effects |> Option.get -;; - -let e2ee_preparation_failure_retries fetch_user () = - let preparation, scope = e2ee_preparation_fixture () in - let error = Error (Core.Effect_failed "synthetic preparation failed") in - let failed = if fetch_user then - e2ee_finish_user_fetch (e2ee_finish_graph_fetch preparation (Ok "wrapped")) error - else e2ee_finish_graph_fetch preparation error in - let rejected = Core.step failed.next (password_event failed.next "unused-synthetic") in - Alcotest.(check bool) "no challenge is advertised" false - (Core.state rejected.next).snapshot.startup.awaiting_e2ee_password; - Alcotest.(check int) "password without challenge starts no work" 0 - (List.length rejected.effects); - let retry = Core.step rejected.next (Core.Online_recovery_requested (timeline_scope rejected.next)) in - let fetched = List.filter (function - | Core.Run (Core.Request (_, Core.Fetch_e2ee_graph_key actual)) -> actual = scope - | _ -> false) retry.effects in - Alcotest.(check int) "Retry reacquires graph key in retained scope" 1 (List.length fetched); - Alcotest.(check bool) "Retry clears failure" true - ((Core.state retry.next).snapshot.startup.failure = None); - Alcotest.(check bool) "selected graph retained" true - ((Core.state retry.next).snapshot.selected_graph = Some scope.graph_id); - Alcotest.(check bool) "Retry performs no cache or credential deletion" false - (List.exists (function - | Core.Run (Core.Request (_, Core.Delete_account_secrets _)) -> true - | _ -> false) retry.effects) -;; - -let e2ee_invalidated_challenge_is_not_advertised () = - let preparation, _ = e2ee_preparation_fixture () in - let prompt = e2ee_finish_user_fetch - (e2ee_finish_graph_fetch preparation (Ok "wrapped")) (Ok "package") in - let refresh = Core.step prompt.next (Core.Catalog_refresh_requested (startup_scope prompt.next)) in - let failed = List.find_map (function - | Core.Run (Core.Request (ticket, Core.Fetch_catalog _)) -> - Some (Core.step refresh.next (Core.Runner_completed (Core.Completion - (ticket, Error (Core.Effect_failed "synthetic catalog refresh failed"))))) - | _ -> None) refresh.effects |> Option.get in - Alcotest.(check bool) "cleared key package cannot advertise a challenge" false - (Core.state failed.next).snapshot.startup.awaiting_e2ee_password; - let submit = Core.step failed.next (password_event failed.next "unused-synthetic") in - Alcotest.(check int) "invalidated challenge starts no crypto" 0 (List.length submit.effects) -;; - -let scenarios = scenarios @ - [ Alcotest.test_case "E2EE graph preparation failure Retry" `Quick - (e2ee_preparation_failure_retries false) - ; Alcotest.test_case "E2EE user preparation failure Retry" `Quick - (e2ee_preparation_failure_retries true) - ; Alcotest.test_case "E2EE invalidated challenge is not advertised" `Quick - e2ee_invalidated_challenge_is_not_advertised ] -;; - - -let snapshot_metadata_open_fields_keep_declared_constraints () = - List.iter - (fun ((body : string), success) -> - let requested, _ = begin_snapshot_bootstrap () in - let downloaded = - List.find_map - (function - | Core.Run (Core.Request (ticket, Core.Fetch_snapshot_metadata _)) -> - Some - (Core.step - requested.next - (Core.Runner_completed (Core.Completion (ticket, Ok body)))) - | _ -> None) - requested.effects - |> Option.get - in - Alcotest.(check bool) - "public Core owns metadata decoding and download decision" - success - (List.exists - (function - | Core.Run (Core.Request (_, Core.Download_snapshot _)) -> true - | _ -> false) - downloaded.effects)) - [ ( {|{"ok":true,"t":0,"url":"https://example.com/snapshot","key":"synthetic","future":{"nested":true}}|} - , true ) - ; {|{"ok":true,"url":"https://example.com/snapshot"}|}, false - ; {|{"ok":true,"t":null,"url":"https://example.com/snapshot"}|}, false - ; {|{"ok":true,"t":-1,"url":"https://example.com/snapshot"}|}, false - ; {|{"ok":true,"t":1.5,"url":"https://example.com/snapshot"}|}, false - ; {|{"ok":true,"t":"42","url":"https://example.com/snapshot"}|}, false - ; {|{"ok":true,"t":true,"url":"https://example.com/snapshot"}|}, false - ; {|{"ok":true,"t":0,"t":1,"url":"https://example.com/snapshot"}|}, false - ; {|{"ok":true,"t":9223372036854775808,"url":"https://example.com/snapshot"}|}, false - ; {|{"ok":true,"t":0,"future":true}|}, false - ; {|{"ok":true,"t":0,"url":null,"future":true}|}, false - ; {|{"ok":"true","t":0,"url":"https://example.com/snapshot","future":true}|}, false - ; {|{"ok":true,"t":0,"url":"http://example.com/snapshot","future":true}|}, false - ] -;; - -let scenarios = - scenarios - @ [ Alcotest.test_case - "snapshot metadata open fields keep declared constraints" - `Quick - snapshot_metadata_open_fields_keep_declared_constraints - ] -;; - - -(* A missing wrapped-key cache is a valid local-access failure. Core owns whether - to fetch the challenge or advertise a terminal failure; no runner/UI replica. *) -let e2ee_auto_preparation () = - let selected, scope = selected_graph encrypted_graph in - let absent = Core.step selected.next (Core.Mirror_inspected (Mirror_absent scope)) in - let completion = List.find_map (function - | Core.Run (Core.Request (ticket, Core.Load_and_unlock_graph_key _)) -> - Some (Core.Runner_completed (Core.Completion - (ticket, Error (Core.Effect_failed "wrappedGraphKeyUnavailable")))) - | _ -> None) absent.effects |> Option.get in - Core.step absent.next completion, scope, completion -;; - -let e2ee_missing_cache_prepares_without_retry () = - let preparation, scope, missing = e2ee_auto_preparation () in - let snapshot = (Core.state preparation.next).snapshot in - Alcotest.(check bool) "cache miss is not a terminal UI failure" true - (snapshot.startup.failure = None && snapshot.last_error = None); - Alcotest.(check bool) "no password prompt before materials exist" false - snapshot.startup.awaiting_e2ee_password; - Alcotest.(check int) "one automatic graph-key fetch" 1 - (List.length (List.filter (function - | Core.Run (Core.Request (_, Core.Fetch_e2ee_graph_key actual)) -> actual = scope - | _ -> false) preparation.effects)); - let repeated = Core.step preparation.next (Core.Online_recovery_requested (timeline_scope preparation.next)) in - Alcotest.(check int) "recovery during preparation is inert" 0 - (List.length repeated.effects); - let duplicate = Core.step preparation.next missing in - Alcotest.(check int) "duplicate cache completion starts no work" 0 - (List.length duplicate.effects); - let user_fetch = e2ee_finish_graph_fetch preparation (Ok "wrapped") in - Alcotest.(check bool) "graph key alone cannot advertise password" false - (Core.state user_fetch.next).snapshot.startup.awaiting_e2ee_password; - let prompt = e2ee_finish_user_fetch user_fetch (Ok "private-package") in - Alcotest.(check bool) "both materials directly advertise password" true - (Core.state prompt.next).snapshot.startup.awaiting_e2ee_password; - Alcotest.(check bool) "password prompt has no obsolete failure" true - ((Core.state prompt.next).snapshot.startup.failure = None) -;; - -let e2ee_auto_preparation_failure_is_bounded fetch_user message () = - let preparation, _, _ = e2ee_auto_preparation () in - let failed = if fetch_user then - e2ee_finish_user_fetch (e2ee_finish_graph_fetch preparation (Ok "wrapped")) - (Error (Core.Effect_failed message)) - else e2ee_finish_graph_fetch preparation (Error (Core.Effect_failed message)) in - let snapshot = (Core.state failed.next).snapshot in - Alcotest.(check bool) "remote error is visible" true - (snapshot.startup.failure = Some Core.During_e2ee && snapshot.last_error = Some message); - Alcotest.(check bool) "missing challenge never shows password" false - snapshot.startup.awaiting_e2ee_password; - Alcotest.(check bool) "failure does not automatically retry" false - (List.exists (function Core.Run _ -> true | _ -> false) failed.effects); - let blocked = Core.step failed.next (password_event failed.next "unused") in - Alcotest.(check int) "no unusable password submission" 0 (List.length blocked.effects); - let retry = Core.step failed.next (Core.Online_recovery_requested (timeline_scope failed.next)) in - Alcotest.(check bool) "explicit Retry can reacquire materials" true - (List.exists (function Core.Run (Core.Request (_, Core.Fetch_e2ee_graph_key _)) -> true - | _ -> false) retry.effects); - let picker = Core.step failed.next Core.Graph_picker_requested in - Alcotest.(check bool) "failure can return to graph selection" true - (Core.state picker.next).snapshot.startup.awaiting_selection -;; - -let e2ee_late_preparation_cannot_reopen_challenge stage cancel result () = - let preparation, scope, _ = e2ee_auto_preparation () in - let flight, complete = - if stage = 0 then preparation, (fun origin -> - let finished = e2ee_finish_graph_fetch { preparation with next = origin } result in - finished) - else let user_fetch = e2ee_finish_graph_fetch preparation (Ok "old-wrapped") in - if stage = 1 then user_fetch, (fun origin -> - e2ee_finish_user_fetch { user_fetch with next = origin } result) - else let prompt = e2ee_finish_user_fetch user_fetch (Ok "old-package") in - let unlocking = Core.step prompt.next (password_event prompt.next "synthetic") in - let completion = List.find_map (function - | Core.Run (Core.Request (ticket, Core.Unlock_private_key _)) -> - Some (Core.Runner_completed (Core.Completion (ticket, - match result with Ok _ -> Ok () | Error e -> Error e))) - | _ -> None) unlocking.effects |> Option.get in - unlocking, (fun origin -> Core.step origin completion) - in - let left = Core.step flight.next - (if cancel then Core.Graph_picker_requested else Core.Graph_selected scope.graph_id) in - let before = Core.state left.next in - let late = complete left.next in - Alcotest.(check bool) "late completion preserves current public state" true - (before = Core.state late.next); - Alcotest.(check int) "late completion starts no next phase" 0 (List.length late.effects); - let submit = Core.step late.next (password_event late.next "abandoned") in - Alcotest.(check int) "obsolete challenge cannot submit" 0 (List.length submit.effects) -;; - -let scenarios = scenarios @ - [ Alcotest.test_case "E2EE missing cache auto-prepares without Retry" `Quick - e2ee_missing_cache_prepares_without_retry - ; Alcotest.test_case "E2EE automatic graph-key network failure is bounded" `Quick - (e2ee_auto_preparation_failure_is_bounded false "synthetic network unavailable") - ; Alcotest.test_case "E2EE automatic user-key authentication failure is bounded" `Quick - (e2ee_auto_preparation_failure_is_bounded true "synthetic authentication required") - ] @ List.concat_map (fun stage -> List.concat_map (fun cancel -> - List.map (fun success -> Alcotest.test_case - (Printf.sprintf "E2EE late stage %d %s %s is fenced" stage - (if cancel then "cancel" else "reentry") (if success then "success" else "error")) - `Quick (e2ee_late_preparation_cannot_reopen_challenge stage cancel - (if success then Ok "late-material" else Error (Core.Effect_failed "late error")))) - [true; false]) [true; false]) [0;1;2] -;; - -(* Startup policy is owned by Sync; these use the raw cold-start state rather - than the explicit-online fixture used by the older synchronization tests. *) -let cold_start () = - Core.config ~managed_sync_origin:(Uri.of_string "https://api.logseq.io") ~limits:(limits ()) - |> Result.get_ok |> Core.initial |> Result.get_ok -;; -let startup_ticket core : Core.startup_scope = - let f = (Core.state core).snapshot.startup in - { account_generation=f.account_generation; presentation_generation=f.presentation_generation } -;; -let timeline_ticket core : Core.timeline_scope = - let s = (Core.state core).snapshot in - { startup=startup_ticket core; graph_id=Option.get s.selected_graph - ; graph_generation=s.startup.graph_generation } -;; -let local_selected ?(encrypted=false) () = - let restoring = Core.step (cold_start ()) (Core.Restore_local_account {user_id="user"}) in - let g = if encrypted then encrypted_graph else graph in - let cache = Core.catalog_cache ~user_id:"user" ~graphs:[g] ~selected_graph:(Some graph_id) in - let selected = Core.step restoring.next (load_catalog_completion restoring.effects (Some cache)) in - selected, inspect_mirror_request selected.effects -;; -let local_attached () = - let selected, request = local_selected () in - let attaching = Core.step selected.next (Core.Mirror_inspected (Core.Mirror_available request)) in - let token = Overlay.sync_token_of_string "sync-token:v1:local-startup" |> Result.get_ok in - let sync = Overlay.sync_view ~token ~checkpoint:(server_cursor_of_int 0) ~submissions:[] in - Core.step attaching.next (Core.Graph_attached {scope=request.scope;sync}) -;; -let no_network message effects = - Alcotest.(check bool) message false (List.exists (function - | Core.Run (Core.Start_websocket _) -> true - | Core.Run (Core.Request (_, (Core.Fetch_catalog _ | Core.Fetch_snapshot_baseline _ - | Core.Fetch_e2ee_graph_key _ | Core.Fetch_e2ee_user_keys _))) -> true - | _ -> false) effects) -;; -let local_startup_has_no_network () = - let cold = cold_start () in - no_network "authentication notification cannot authorize cold-start networking" - (Core.step cold (Core.Account_authenticated {user_id=Some "user"})).effects; - let attached = local_attached () in - no_network "attachment stays local" attached.effects; - Alcotest.(check bool) "local access does not forge online credentials" false - (Core.state attached.next).snapshot.startup.authenticated; - Alcotest.(check bool) "local attachment is usable" true - ((Core.state attached.next).snapshot.startup.local_access = Core.Local_available); - no_network "feed completion does not authorize networking" - (Core.step attached.next Core.Local_feed_acknowledged).effects; - no_network "foreground does not authorize networking" - (Core.step attached.next (Core.Foreground_changed {foreground=true;lifecycle_generation=1L})).effects -;; -let missing_local_data_stops_loading () = - let selected, request = local_selected () in - let absent = Core.step selected.next (Core.Mirror_inspected (Core.Mirror_absent request.scope)) in - no_network "no automatic snapshot download" absent.effects; - let f = (Core.state absent.next).snapshot.startup in - Alcotest.(check bool) "missing mirror is terminal and accurate" true - (not f.restoring_local && not f.bootstrapping && f.local_access=Core.Local_unavailable Core.Local_mirror_unavailable); - let selected, request = local_selected ~encrypted:true () in - let keys = Core.step selected.next (Core.Mirror_inspected (Core.Mirror_available request)) in - let missing = List.find_map (function - | Core.Run (Core.Request (ticket, Core.Load_and_unlock_graph_key _)) -> - Some (Core.Runner_completed (Core.Completion (ticket, Error (Core.Effect_failed "local key absent")))) - | _ -> None) keys.effects |> Option.get in - let failed = Core.step keys.next missing in - no_network "no automatic remote key recovery" failed.effects; - Alcotest.(check bool) "key failure stops loading" true - ((Core.state failed.next).snapshot.startup.local_access=Core.Local_unavailable Core.Local_key_unavailable); - let restoring = Core.step (cold_start ()) (Core.Restore_local_account {user_id="user"}) in - let absent = Core.step restoring.next (load_catalog_completion restoring.effects None) in - Alcotest.(check bool) "missing catalog stops loading" true - ((Core.state absent.next).snapshot.startup.local_access=Core.Local_unavailable Core.Local_catalog_unavailable) -;; -let appearance_is_scoped_and_authentication_is_required () = - let attached = local_attached () in - let ticket = timeline_ticket attached.next in - let stale = {ticket with startup={ticket.startup with presentation_generation=ticket.startup.presentation_generation+1}} in - let ignored = Core.step attached.next (Core.Timeline_presented stale) in - Alcotest.(check bool) "stale appearance leaves gate closed" true - ((Core.state ignored.next).snapshot.startup.network_phase=Core.Local_only); - let shown = Core.step attached.next (Core.Timeline_presented ticket) in - no_network "appearance authorizes authentication, not anonymous sync" shown.effects; - Alcotest.(check bool) "current appearance opens phase" true - ((Core.state shown.next).snapshot.startup.network_phase=Core.Online_after_timeline); - let online = Core.step shown.next (Core.Account_authenticated {user_id=Some "user"}) in - Alcotest.(check bool) "verified authentication starts websocket" true - (List.exists (function Core.Run (Core.Start_websocket _) -> true | _ -> false) online.effects); - let stopped=Core.step online.next (Core.Online_cancelled (startup_ticket online.next)) in - Alcotest.(check bool) "closing permission retires its socket owner" true - (List.exists (function Core.Run (Core.Close_websocket _) -> true | _ -> false) stopped.effects); - let connection=List.find_map (function Core.Run(Core.Start_websocket request)->Some request.scope|_->None) online.effects |> Option.get in - let retry=Core.step stopped.next (Core.Sign_in_requested (startup_ticket stopped.next)) in - let retry=Core.step retry.next (Core.Account_authenticated {user_id=Some "user"}) in - let retired=Core.step retry.next (Core.Websocket_retired connection) in - Alcotest.(check bool) "retry while old socket retires is resumed after retirement" true - (List.exists (function Core.Run(Core.Start_websocket _)->true|_->false) retired.effects); - let duplicate = Core.step online.next (Core.Timeline_presented ticket) in - Alcotest.(check int) "duplicate appearance is inert" 0 (List.length duplicate.effects); - let changed = Core.step shown.next (Core.Graph_selected graph_id) in - let stale_permission=Core.step changed.next (Core.Sign_in_requested ticket.startup) in - Alcotest.(check bool) "old presentation cannot authorize a replacement graph" true - ((Core.state stale_permission.next).snapshot.startup.network_phase=Core.Local_only); - let late = Core.step changed.next (Core.Timeline_presented ticket) in - Alcotest.(check bool) "old graph cannot open replacement gate" true - ((Core.state late.next).snapshot.startup.network_phase=Core.Local_only); - let cancelled = Core.step shown.next (Core.Online_cancelled (startup_ticket shown.next)) in - no_network "late authentication after cancellation is inert" - (Core.step cancelled.next (Core.Account_authenticated {user_id=Some "user"})).effects; - let first = cold_start () in - let missing = Core.step first (Core.Local_account_unavailable (startup_ticket first, Core.No_local_account)) in - Alcotest.(check bool) "first installation has a terminal local result" true - ((Core.state missing.next).snapshot.startup.local_access=Core.Local_unavailable Core.No_local_account); - let granted = Core.step missing.next (Core.Sign_in_requested (startup_ticket missing.next)) in - let authenticated = Core.step granted.next (Core.Account_authenticated {user_id=Some "new-user"}) in - Alcotest.(check bool) "explicit sign-in still fetches catalog" true (has_catalog_request authenticated.effects) -;; -let cancelled_local_key_crypto_still_completes () = - let preparation, scope = e2ee_preparation_fixture () in - let prompt = e2ee_finish_user_fetch (e2ee_finish_graph_fetch preparation (Ok "wrapped")) (Ok "private-package") in - let unlocking = Core.step prompt.next (password_event prompt.next "synthetic") in - let finish_private (result : (unit, Core.effect_error) result) core = List.find_map (function - | Core.Run (Core.Request (ticket, Core.Unlock_private_key _)) -> - Some (Core.step core (Core.Runner_completed (Core.Completion (ticket, result)))) - | _ -> None) unlocking.effects |> Option.get in - let cancelled = Core.step unlocking.next (Core.Online_cancelled (startup_ticket unlocking.next)) in - let failed = finish_private (Error (Core.Effect_failed "synthetic failure")) cancelled.next in - Alcotest.(check bool) "cancelled private-key failure cannot reopen unusable prompt" false - (Core.state failed.next).snapshot.startup.awaiting_e2ee_password; - let key = finish_private (Ok ()) cancelled.next in - let loaded = List.find_map (function - | Core.Run (Core.Request (ticket, Core.Fetch_and_unlock_graph_key _)) -> - let cancelled = Core.step key.next (Core.Online_cancelled (startup_ticket key.next)) in - Some (Core.step cancelled.next (Core.Runner_completed (Core.Completion - (ticket, Ok (Core.graph_key_handle ~id:"recovered-local-key" ~scope))))) - | _ -> None) key.effects |> Option.get in - no_network "local key completion cannot begin remote snapshot after cancellation" loaded.effects; - Alcotest.(check bool) "local wrapped-key unlock result remains usable" true - (Option.fold ~none:false ~some:(fun (context : Core.asset_context) -> Option.is_some context.key) (Core.asset_context loaded.next)) -;; -let local_staged_asset_publication_is_available_offline () = - let attached = local_attached () in - let scope = Core.admitted_graph_scope attached.next |> Option.get in - let issued = Core.step attached.next (Core.Asset_requested - {scope; operation="offline-publish"; - action=Core.Publish_staged_asset {asset=graph_id; version=asset_version; file="synthetic-staged-reference"}}) in - Alcotest.(check bool) "local staged content can enter local cache while offline" true - (List.exists (function Core.Run (Core.Asset_io (_, {action=Core.Publish_staged_asset _; _})) -> true | _ -> false) issued.effects) -;; -let cancellation_fences_deferred_catalog_and_snapshot () = - let restoring = Core.step (cold_start ()) (Core.Restore_local_account {user_id="user"}) in - let granted = Core.step restoring.next (Core.Sign_in_requested (startup_ticket restoring.next)) in - let verified = Core.step granted.next (Core.Account_authenticated {user_id=Some "user"}) in - let cancelled = Core.step verified.next (Core.Online_cancelled (startup_ticket verified.next)) in - let cache = Core.catalog_cache ~user_id:"user" ~graphs:[graph] ~selected_graph:(Some graph_id) in - let loaded = Core.step cancelled.next (load_catalog_completion restoring.effects (Some cache)) in - no_network "deferred authentication cannot restart catalog after cancellation" loaded.effects; - Alcotest.(check bool) "local completion cannot restore online authentication" false - (Core.state loaded.next).snapshot.startup.authenticated; - let requested, _ = begin_snapshot_bootstrap () in - let cancelled = Core.step requested.next (Core.Online_cancelled (startup_ticket requested.next)) in - List.iter (function - | Core.Run (Core.Request (ticket, Core.Fetch_snapshot_metadata _)) -> - let late = Core.step cancelled.next (Core.Runner_completed - (Core.Completion (ticket, Ok {|{"ok":true,"t":42,"url":"https://example.com/snapshot"}|}))) in - check_instructions "cancelled snapshot metadata completion" [] late.effects - | _ -> ()) requested.effects; - let facts = (Core.state cancelled.next).snapshot.startup in - Alcotest.(check bool) "cancelled metadata has a terminal local result" true - (match facts.local_access with Core.Local_unavailable _ -> true | _ -> false) -;; -let online_cancellation_preserves_local_restore () = - let restoring = Core.step (cold_start ()) (Core.Restore_local_account {user_id="user"}) in - let cancelled = Core.step restoring.next (Core.Online_cancelled (startup_ticket restoring.next)) in - let facts = (Core.state cancelled.next).snapshot.startup in - Alcotest.(check bool) "local catalog load remains pending" true - (facts.restoring_local && facts.local_access = Core.Local_pending); - Alcotest.(check bool) "online cancellation does not retire local resources" false - (List.exists (function Core.Run (Core.Cancel_effects _) -> true | _ -> false) cancelled.effects); - let cache = Core.catalog_cache ~user_id:"user" ~graphs:[graph] ~selected_graph:(Some graph_id) in - let loaded = Core.step cancelled.next (load_catalog_completion restoring.effects (Some cache)) in - ignore (inspect_mirror_request loaded.effects); - let selected, request = local_selected ~encrypted:true () in - let keys = Core.step selected.next (Core.Mirror_inspected (Core.Mirror_available request)) in - let cancelled = Core.step keys.next (Core.Online_cancelled (startup_ticket keys.next)) in - let completion = List.find_map (function - | Core.Run (Core.Request (ticket, Core.Load_and_unlock_graph_key scope)) -> - let key = Core.graph_key_handle ~id:"retained-local-key" ~scope in - Some (Core.Runner_completed (Core.Completion (ticket, Ok key))) - | _ -> None) keys.effects |> Option.get in - let loaded = Core.step cancelled.next completion in - Alcotest.(check bool) "pending local key completion still attaches graph" true - (List.exists (function Core.Delegate (Core.Attach_graph _) -> true | _ -> false) loaded.effects) -;; -(* The pure owner can reproduce authenticated Local_only by public timeline, - identity and graph-selection events; no native-result forgery is required. *) -let authenticated_local_catalog_refresh_waits_for_permission () = - let attached=local_attached () in - let shown=Core.step attached.next (Core.Timeline_presented (timeline_ticket attached.next)) in - let verified=Core.step shown.next (Core.Account_authenticated {user_id=Some "user"}) in - let selected=Core.step verified.next (Core.Graph_selected graph_id) in - let startup=(Core.state selected.next).snapshot.startup in - Alcotest.(check bool) "graph switch preserves identity while returning local" true - (startup.authenticated && startup.network_phase=Core.Local_only); - let refresh=Core.step selected.next (Core.Catalog_refresh_requested (startup_ticket selected.next)) in - no_network "catalog refresh waits for native permission and identity round-trip" refresh.effects; - Alcotest.(check bool) "native permission phase is published" true - (List.exists (function Core.Publish (Core.State_changed state) -> - state.snapshot.startup.network_phase=Core.Online_by_user | _ -> false) refresh.effects); - let duplicate=Core.step refresh.next (Core.Catalog_refresh_requested (startup_ticket refresh.next)) in - no_network "repeated refresh cannot bypass pending native permission" duplicate.effects; - let confirmed=Core.step duplicate.next (Core.Account_authenticated {user_id=Some "user"}) in - Alcotest.(check bool) "confirmed identity starts catalog refresh" true - (has_catalog_request confirmed.effects); - let repeated=Core.step confirmed.next (Core.Catalog_refresh_requested (startup_ticket confirmed.next)) in - Alcotest.(check bool) "already online refresh remains direct" true (has_catalog_request repeated.effects) -;; -let scenarios = scenarios @ - [ Alcotest.test_case "authenticated local catalog refresh waits for permission" `Quick authenticated_local_catalog_refresh_waits_for_permission - ; Alcotest.test_case "cancelled local key crypto completes without reopening password" `Quick cancelled_local_key_crypto_still_completes - ; Alcotest.test_case "local staged asset publishes offline" `Quick local_staged_asset_publication_is_available_offline - ; Alcotest.test_case "cancellation fences deferred catalog and snapshot metadata" `Quick cancellation_fences_deferred_catalog_and_snapshot - ; Alcotest.test_case "online cancellation retains pending local restoration" `Quick online_cancellation_preserves_local_restore - ; Alcotest.test_case "local startup emits zero network work" `Quick local_startup_has_no_network - ; Alcotest.test_case "local insufficiency stops loading without network fallback" `Quick missing_local_data_stops_loading - ; Alcotest.test_case "native appearance and explicit online permissions are scoped" `Quick appearance_is_scoped_and_authentication_is_required ] -;; - -(* User graph selection owns the decision to download an absent mirror. A cold - restore and unrelated storage/key failures still require explicit recovery. *) -let user_selected_missing_mirror () = - let selected, _ = local_selected () in - let chosen = Core.step selected.next (Core.Graph_selected graph_id) in - let request = inspect_mirror_request chosen.effects in - let missing = Core.step chosen.next (Core.Mirror_inspected (Core.Mirror_absent request.scope)) in - no_network "selection requests native permission before remote work" missing.effects; - let facts = (Core.state missing.next).snapshot.startup in - Alcotest.(check bool) "absent user selection publishes online permission without error" true - (facts.network_phase = Core.Online_by_user && facts.failure = None - && facts.local_access = Core.Local_pending && not facts.authenticated); - let confirmed = Core.step missing.next (Core.Account_authenticated {user_id=Some "user"}) in - let downloading = confirmed in - Alcotest.(check bool) "verified selection starts one metadata request" true - (List.exists (function Core.Run(Core.Request(_,Core.Fetch_snapshot_metadata _))->true|_->false) downloading.effects); - let failure = List.find_map (function - | Core.Run(Core.Request(ticket,Core.Fetch_snapshot_metadata _)) -> - Some (Core.Runner_completed(Core.Completion(ticket,Error(Core.Effect_failed "synthetic offline")))) - | _ -> None) downloading.effects |> Option.get in - let failed = Core.step downloading.next failure in - Alcotest.(check bool) "download failure remains terminal instead of auto-retrying" true - ((Core.state failed.next).snapshot.startup.failure = Some Core.During_bootstrap); - no_network "failed request has no automatic retry" failed.effects; - let retry = Core.step failed.next (Core.Online_recovery_requested (timeline_ticket failed.next)) in - ignore (inspect_mirror_request retry.effects) -;; - -let selected_download_cancellation_and_stale_inspections () = - let selected, old_request = local_selected () in - let chosen = Core.step selected.next (Core.Graph_selected graph_id) in - let request = inspect_mirror_request chosen.effects in - let stale = Core.step chosen.next (Core.Mirror_inspected(Core.Mirror_absent old_request.scope)) in - Alcotest.(check bool) "old inspection cannot authorize selected graph" true - ((Core.state stale.next).snapshot.startup.network_phase = Core.Local_only); - let missing = Core.step stale.next (Core.Mirror_inspected(Core.Mirror_absent request.scope)) in - let cancelled = Core.step missing.next (Core.Online_cancelled(startup_ticket missing.next)) in - no_network "cancellation keeps remote work stopped" cancelled.effects; - let late = Core.step cancelled.next (Core.Account_authenticated {user_id=Some "user"}) in - no_network "late identity cannot resume cancelled download" late.effects; - Alcotest.(check bool) "cancelled download offers explicit mirror recovery" true - ((Core.state late.next).snapshot.startup.local_access = Core.Local_unavailable Core.Local_mirror_unavailable); - let retry=Core.step late.next (Core.Online_recovery_requested(timeline_ticket late.next)) in - let confirmed=Core.step retry.next (Core.Account_authenticated {user_id=Some "user"}) in - ignore(inspect_mirror_request confirmed.effects); - let replacement = Core.step chosen.next (Core.Graph_selected graph_id) in - let retired = Core.step replacement.next (Core.Mirror_inspected(Core.Mirror_absent request.scope)) in - no_network "replaced selection ignores previous inspection" retired.effects; - Alcotest.(check bool) "replacement still starts locally" true - ((Core.state retired.next).snapshot.startup.network_phase = Core.Local_only) -;; - -let selected_available_mirror_keeps_local_key_failures () = - let selected, _ = local_selected ~encrypted:true () in - let chosen = Core.step selected.next (Core.Graph_selected graph_id) in - let request = inspect_mirror_request chosen.effects in - let available = Core.step chosen.next (Core.Mirror_inspected(Core.Mirror_available request)) in - no_network "available selected mirror keeps local-only key access" available.effects; - let completion = List.find_map (function - | Core.Run(Core.Request(ticket,Core.Load_and_unlock_graph_key _)) -> - Some(Core.Runner_completed(Core.Completion(ticket,Error(Core.Effect_failed "key missing")))) - | _ -> None) available.effects |> Option.get in - let failed=Core.step available.next completion in - no_network "key failure is not a download request" failed.effects; - Alcotest.(check bool) "key failure stays explicit" true - ((Core.state failed.next).snapshot.startup.local_access=Core.Local_unavailable Core.Local_key_unavailable) -;; - -let scenarios = scenarios @ - [ Alcotest.test_case "user-selected missing mirror requests direct download" `Quick user_selected_missing_mirror - ; Alcotest.test_case "selected download cancellation and stale inspections are fenced" `Quick selected_download_cancellation_and_stale_inspections - ; Alcotest.test_case "selected existing mirror retains local key failure" `Quick selected_available_mirror_keeps_local_key_failures ] -;; - -let selected_encrypted_download_retains_crypto_and_failure () = - let selected, _ = local_selected ~encrypted:true () in - let chosen = Core.step selected.next (Core.Graph_selected graph_id) in - let request=inspect_mirror_request chosen.effects in - let missing=Core.step chosen.next (Core.Mirror_inspected(Core.Mirror_absent request.scope)) in - Alcotest.(check bool) "encrypted selection opens scoped permission" true - ((Core.state missing.next).snapshot.startup.network_phase=Core.Online_by_user); - let confirmed=Core.step missing.next (Core.Account_authenticated {user_id=Some "user"}) in - let keys=confirmed in - Alcotest.(check bool) "encrypted selection still requires local key validation" true - (List.exists (function Core.Run(Core.Request(_,Core.Load_and_unlock_graph_key _))->true|_->false) keys.effects); - let missing_key=List.find_map (function - | Core.Run(Core.Request(ticket,Core.Load_and_unlock_graph_key _)) -> - Some(Core.Runner_completed(Core.Completion(ticket,Error(Core.Effect_failed "key missing")))) - | _ -> None) keys.effects |> Option.get in - let remote=Core.step keys.next missing_key in - let failed=List.find_map (function - | Core.Run(Core.Request(ticket,Core.Fetch_e2ee_graph_key _)) -> - Some(Core.step remote.next (Core.Runner_completed(Core.Completion(ticket,Error(Core.Effect_failed "synthetic key server offline"))))) - | _ -> None) remote.effects |> Option.get in - Alcotest.(check bool) "key recovery failure remains visible" true - ((Core.state failed.next).snapshot.startup.failure=Some Core.During_e2ee); - no_network "key recovery failure does not auto-repeat" failed.effects -;; -let scenarios=scenarios @ [Alcotest.test_case "selected encrypted download retains key validation and bounded failure" `Quick selected_encrypted_download_retains_crypto_and_failure] -;; - -(* The stream is live: a write after metadata may already be present in the - artifact. Core must still replay every journal position after metadata t, - and only durable application may advance that cursor. *) -let snapshot_live_stream_catchup_keeps_metadata_cursor () = - let opened, connection, sync = snapshot_catchup_fixture () in - Alcotest.(check bool) - "initial catchup starts at metadata t" - true - (pull_sends opened.effects = [ connection, Some (server_cursor_of_int 42) ]); - let hello = - pull_message - opened.next - connection - (Protocol.Server.Hello { t = server_cursor_of_int 44; checksum = None }) - in - let changed = - pull_message - hello.next - connection - (Protocol.Server.Changed { t = server_cursor_of_int 45 }) - in - check_pull_count "head changes join the in-flight catchup" 0 changed.effects; - Alcotest.(check (option cursor_testable)) - "later live stream writes cannot skip journal positions" - (Some (server_cursor_of_int 42)) - (Core.state changed.next).snapshot.applied_server_t; - let received = pull_response changed.next connection ~after:42 44 in - let batches = - List.filter_map - (function - | Core.Delegate (Core.Apply_authoritative_batch batch) -> Some batch - | _ -> None) - received.effects - in - let batch = - match batches with - | [ batch ] -> batch - | _ -> Alcotest.fail "one catchup batch" - in - Alcotest.(check (list int64)) - "every post-metadata transaction reaches the owner in order" - [ 43L; 44L ] - (List.map - (fun tx -> - Overlay.authoritative_transaction_cursor tx - |> Logseq_db_types.Server_cursor.to_int64) - (Overlay.authoritative_batch_transactions batch.input)); - Alcotest.(check (option cursor_testable)) - "delegation is not a durable commit" - (Some (server_cursor_of_int 42)) - (Core.state received.next).snapshot.applied_server_t; - let committed = - recovery_apply received.next connection (recovery_sync sync ~checkpoint:44 1 []) - in - Alcotest.(check bool) - "next catchup starts at committed t, including concurrent write" - true - (pull_sends committed.effects = [ connection, Some (server_cursor_of_int 44) ]); - let final = pull_response committed.next connection ~after:44 45 in - let committed = - recovery_apply final.next connection (recovery_sync sync ~checkpoint:45 2 []) - in - Alcotest.(check (option cursor_testable)) - "all concurrent writes are durably acknowledged" - (Some (server_cursor_of_int 45)) - (Core.state committed.next).snapshot.applied_server_t; - check_pull_count "caught up has no redundant pull" 0 committed.effects; - let repeated = pull_response committed.next connection ~after:42 45 in - Alcotest.(check bool) - "repeated catchup cannot reapply committed transactions" - false - (List.exists - (function - | Core.Delegate (Core.Apply_authoritative_batch _) -> true - | _ -> false) - repeated.effects) -;; - -let snapshot_failure_and_retry_keep_cursor_uncommitted () = - List.iter - (fun activation_failure -> - let requested, _scope = begin_snapshot_bootstrap () in - let download = - List.find_map - (function - | Core.Run (Core.Request (ticket, Core.Fetch_snapshot_metadata _)) -> - Some - (Core.step - requested.next - (Core.Runner_completed - (Core.Completion - ( ticket - , Ok - {|{"ok":true,"t":42,"key":"stream/graph.snapshot","url":"https://example.com/snapshot","content-encoding":"gzip"}|} - )))) - | _ -> None) - requested.effects - |> Option.get - in - let failed, late_completion = - List.find_map - (function - | Core.Run (Core.Request (ticket, Core.Download_snapshot download_request)) -> - let artifact = - Core.stage_snapshot_download download_request ~path:"/tmp/synthetic-snapshot" ~expected_rows:1 |> Result.get_ok - in - let completion = - Core.Runner_completed (Core.Completion (ticket, Ok artifact)) - in - let failed = - if activation_failure - then ( - let activating = Core.step download.next completion in - Core.step - activating.next - (Core.Snapshot_activation_failed (Core.snapshot_activation_failed - (snapshot_activation_request activating.effects) - ~message:"synthetic activation failure"))) - else - Core.step - download.next - (Core.Runner_completed - (Core.Completion - (ticket, Error (Core.Effect_failed "synthetic stream failure")))) - in - Some (failed, completion) - | _ -> None) - download.effects - |> Option.get - in - Alcotest.(check (option cursor_testable)) - "failed bootstrap cannot publish a cursor" - None - (Core.state failed.next).snapshot.applied_server_t; - Alcotest.(check bool) - "failure stays visible" - true - ((Core.state failed.next).snapshot.startup.failure = Some Core.During_bootstrap); - let stale = Core.step failed.next late_completion in - check_instructions - "completed download ticket cannot be consumed twice" - [] - stale.effects; - let retry = - Core.step - stale.next - (Core.Online_recovery_requested (timeline_ticket stale.next)) - in - let mirror = inspect_mirror_request retry.effects in - let missing = - Core.step retry.next (Core.Mirror_inspected (Core.Mirror_absent mirror.scope)) - in - Alcotest.(check bool) - "retry obtains fresh metadata without a preliminary pull" - true - (List.exists - (function - | Core.Run (Core.Request (_, Core.Fetch_snapshot_metadata _)) -> true - | _ -> false) - missing.effects - && not - (List.exists - (function - | Core.Run (Core.Request (_, Core.Fetch_snapshot_baseline _)) -> true - | _ -> false) - missing.effects))) - [ false; true ] -;; - -let scenarios = - scenarios - @ [ Alcotest.test_case - "live snapshot catchup preserves metadata cursor" - `Quick - snapshot_live_stream_catchup_keeps_metadata_cursor - ; Alcotest.test_case - "snapshot stream and activation failure retry" - `Quick - snapshot_failure_and_retry_keep_cursor_uncommitted - ] -;; - -let replaced_snapshot_completions_are_inert () = - let other = { graph with graph_id = - Logseq_db_types.Graph_types.Uuid.of_string "22222222-2222-4222-8222-222222222222" - |> Result.get_ok } in - List.iter (fun download_pending -> - let requested, _old_scope = begin_snapshot_bootstrap ~graphs:[graph; other] () in - let metadata (result : (Core.snapshot_metadata, Core.effect_error) result) = List.find_map (function - | Core.Run (Core.Request (ticket, Core.Fetch_snapshot_metadata _)) -> - Some (Core.Runner_completed (Core.Completion (ticket, result))) - | _ -> None) requested.effects |> Option.get in - let old_owner, successes, failure = - if download_pending then ( - let downloading = Core.step requested.next - (metadata (Ok {|{"ok":true,"t":42,"url":"https://example.com/old"}|})) in - List.find_map (function - | Core.Run (Core.Request (ticket, Core.Download_snapshot download_request)) -> - let artifact = Core.stage_snapshot_download download_request ~path:"/tmp/synthetic-old" ~expected_rows:1 |> Result.get_ok in - Some (downloading.next, - [Core.Runner_completed (Core.Completion (ticket, Ok artifact))], - Core.Runner_completed (Core.Completion - (ticket, Error (Core.Effect_failed "obsolete stream")))) - | _ -> None) downloading.effects |> Option.get) - else requested.next, - [metadata (Ok {|{"ok":true,"t":42,"url":"https://example.com/old"}|}); - metadata (Ok {|{"ok":true,"url":"https://example.com/invalid"}|})], - metadata (Error (Core.Effect_failed "obsolete metadata")) in - let selected = Core.step old_owner (Core.Graph_selected other.graph_id) in - let inspection = inspect_mirror_request selected.effects in - let current = Core.step selected.next - (Core.Mirror_inspected (Core.Mirror_absent inspection.scope)) in - List.iter (fun completion -> - let late = Core.step current.next completion in - check_instructions "replaced snapshot completion has no effects" [] late.effects; - check_observed "replaced snapshot" "current owner is unchanged" - (observe current.next) (observe late.next); - let fresh = List.find_map (function - | Core.Run (Core.Request (ticket, Core.Fetch_snapshot_metadata _)) -> - Some (Core.step late.next (Core.Runner_completed (Core.Completion - (ticket, Ok {|{"ok":true,"t":84,"url":"https://example.com/current"}|})))) - | _ -> None) current.effects |> Option.get in - List.find_map (function - | Core.Run (Core.Request (ticket, Core.Download_snapshot request)) -> - Alcotest.(check bool) "fresh download retains graph and URL" true - (Core.snapshot_download_scope request = inspection.scope && Core.snapshot_download_uri request = Uri.of_string "https://example.com/current"); - let artifact = Core.stage_snapshot_download request ~path:"/tmp/synthetic-current" ~expected_rows:1 |> Result.get_ok in - Some (Core.step fresh.next (Core.Runner_completed - (Core.Completion (ticket, Ok artifact)))) - | _ -> None) fresh.effects |> Option.get |> fun activation -> - Alcotest.(check bool) "fresh activation keeps its own graph and cursor" true - (List.exists (function - | Core.Delegate (Core.Activate_snapshot request) -> - Core.snapshot_activation_scope request = inspection.scope && Core.snapshot_activation_cursor request = server_cursor_of_int 84 - | _ -> false) activation.effects)) (successes @ [failure])) [false; true] -;; - -let scenarios = scenarios @ [ Alcotest.test_case - "replaced snapshot metadata and download completions are inert" `Quick - replaced_snapshot_completions_are_inert ] -;; - -let snapshot_artifact_from_retried_attempt_is_rejected () = - let requested, _ = begin_snapshot_bootstrap () in - let metadata (origin : Core.transition) cursor = - List.find_map (function - | Core.Run (Core.Request (ticket, Core.Fetch_snapshot_metadata _)) -> - Some (Core.step origin.next (Core.Runner_completed (Core.Completion - (ticket, Ok (Printf.sprintf "{\"ok\":true,\"t\":%d,\"url\":\"https://example.com/snapshot\"}" cursor))))) - | _ -> None) origin.effects |> Option.get in - let old = metadata requested 42 in - let failed, old_artifact, old_scope = List.find_map (function - | Core.Run (Core.Request (ticket, Core.Download_snapshot request)) -> - let artifact = Core.stage_snapshot_download request ~path:"/tmp/synthetic-attempt-a" - ~expected_rows:1 |> Result.get_ok in - Some (Core.step old.next (Core.Runner_completed (Core.Completion - (ticket, Error (Core.Effect_failed "synthetic interrupted stream")))), artifact, - Core.snapshot_download_scope request) - | _ -> None) old.effects |> Option.get in - let retry = Core.step failed.next (Core.Online_recovery_requested (timeline_scope failed.next)) in - let mirror = inspect_mirror_request retry.effects in - let missing = Core.step retry.next (Core.Mirror_inspected (Core.Mirror_absent mirror.scope)) in - let current = metadata missing 84 in - let mismatch = List.find_map (function - | Core.Run (Core.Request (ticket, Core.Download_snapshot request)) -> - Alcotest.(check bool) "retry changes the generation of this production graph scope" false - (Core.snapshot_download_scope request = old_scope); - Some (Core.step current.next (Core.Runner_completed (Core.Completion (ticket, Ok old_artifact)))) - | _ -> None) current.effects |> Option.get in - Alcotest.(check bool) "foreign attempt cannot activate" false - (List.exists (function Core.Delegate (Core.Activate_snapshot _) -> true | _ -> false) mismatch.effects); - Alcotest.(check bool) "mismatch terminates bootstrap with visible failure" true - ((Core.state mismatch.next).snapshot.startup.failure = Some Core.During_bootstrap) -;; - -let scenarios = scenarios @ [ Alcotest.test_case - "old snapshot artifact cannot complete a fresh request" `Quick - snapshot_artifact_from_retried_attempt_is_rejected ] -;; - -let failed_snapshot_activation_receipts_are_inert () = - let activating = snapshot_activation_fixture () in - let request = snapshot_activation_request activating.effects in - let failure = Core.Snapshot_activation_failed - (Core.snapshot_activation_failed request ~message:"synthetic activation failure") in - let failed = Core.step activating.next failure in - let success = Core.Snapshot_activated (Core.snapshot_activation_completed request) in - List.iter (fun event -> - let late = Core.step failed.next event in - check_observed "retired activation" "old receipt preserves failed owner" - (observe failed.next) (observe late.next); - check_instructions "retired activation receipt" [] late.effects) [success; failure] -;; - -let scenarios = scenarios @ [ Alcotest.test_case - "late activation success and failure cannot reopen failed bootstrap" `Quick - failed_snapshot_activation_receipts_are_inert ] -;; - -let sync_cursor_diagnostic core label = - (Core.state core).diagnostics.groups - |> List.concat_map (fun (group : Core.diagnostic_group) -> group.entries) - |> List.assoc_opt label -;; - -let diagnostics_observe_remote_cursor_without_advancing_local () = - let selected, _ = selected_graph graph in - List.iter (fun label -> - Alcotest.(check (option string)) (label ^ " starts unknown") - (Some "Not available") (sync_cursor_diagnostic selected.next label)) - ["Local tx"; "Remote tx"]; - let opened, connection, _ = pull_fixture () in - let largest = Logseq_db_types.Server_cursor.of_int64 Int64.max_int |> Result.get_ok in - let observed = pull_message opened.next connection - (Protocol.Server.Hello { t = largest; checksum = None }) in - Alcotest.(check (option string)) "remote cursor retains all int64 digits" - (Some "9223372036854775807") (sync_cursor_diagnostic observed.next "Remote tx"); - Alcotest.(check (option string)) "server notification is not a local commit" - (Some "0") (sync_cursor_diagnostic observed.next "Local tx"); - Alcotest.(check bool) "new cursor is published while an existing pull is occupied" true - (List.exists (function - | Core.Publish (Core.State_changed state) -> - List.exists (fun (group : Core.diagnostic_group) -> - List.assoc_opt "Remote tx" group.entries = Some "9223372036854775807") state.diagnostics.groups - | _ -> false) observed.effects); - let lower = pull_message observed.next connection - (Protocol.Server.Changed { t = server_cursor_of_int 1 }) in - Alcotest.(check (option string)) "older notifications cannot lower known remote maximum" - (Some "9223372036854775807") (sync_cursor_diagnostic lower.next "Remote tx"); - let failed = pull_message lower.next connection - (Protocol.Server.Error { message = "synthetic server failure" }) in - Alcotest.(check (option string)) "failure retains the last server observation" - (Some "9223372036854775807") (sync_cursor_diagnostic failed.next "Remote tx"); - let catalog = refresh_catalog lower.next [graph; other_graph] in - let switched = Core.step catalog.next (Core.Graph_selected other_graph.graph_id) in - List.iter (fun label -> - Alcotest.(check (option string)) (label ^ " clears on graph switch") - (Some "Not available") (sync_cursor_diagnostic switched.next label)) - ["Local tx"; "Remote tx"]; - let stale = pull_message switched.next connection - (Protocol.Server.Changed { t = largest }) in - Alcotest.(check (option string)) "old graph messages cannot repopulate remote cursor" - (Some "Not available") (sync_cursor_diagnostic stale.next "Remote tx"); - let account = Core.step observed.next (Core.Account_authenticated {user_id=Some "other-user"}) in - Alcotest.(check (option string)) "account switch removes old graph cursor rows" - None (sync_cursor_diagnostic account.next "Remote tx") -;; - -let diagnostics_local_cursor_tracks_authoritative_commit () = - let opened, connection, _ = pull_fixture () in - let received = pull_response opened.next connection ~after:0 1 in - Alcotest.(check (option string)) "received authoritative data is not committed yet" - (Some "0") (sync_cursor_diagnostic received.next "Local tx"); - Alcotest.(check (option string)) "pull response observes the remote cursor" - (Some "1") (sync_cursor_diagnostic received.next "Remote tx"); - let sync = Overlay.sync_view - ~token:(token Overlay.sync_token_of_string "sync-token:v1:diagnostic-commit") - ~checkpoint:(server_cursor_of_int 1) ~submissions:[] in - let committed = recovery_apply received.next connection sync in - Alcotest.(check (option string)) "only authoritative completion advances local cursor" - (Some "1") (sync_cursor_diagnostic committed.next "Local tx"); - Alcotest.(check (option string)) "completed catchup retains remote cursor" - (Some "1") (sync_cursor_diagnostic committed.next "Remote tx"); - let selected, scope = selected_graph graph in - let inspected = Core.step selected.next - (Core.Mirror_inspected (Core.Mirror_available {graph;scope})) in - let largest = Logseq_db_types.Server_cursor.of_int64 Int64.max_int |> Result.get_ok in - let sync = Overlay.sync_view ~token:(Overlay.sync_view_token sync) - ~checkpoint:largest ~submissions:[] in - let attached = Core.step inspected.next (Core.Graph_attached {scope;sync}) in - Alcotest.(check (option string)) "restored checkpoint retains all int64 digits" - (Some "9223372036854775807") (sync_cursor_diagnostic attached.next "Local tx"); - Alcotest.(check (option string)) "local checkpoint does not invent server evidence" - (Some "Not available") (sync_cursor_diagnostic attached.next "Remote tx") -;; - -let scenarios = scenarios @ - [ Alcotest.test_case "diagnostics observes scoped remote cursor" `Quick - diagnostics_observe_remote_cursor_without_advancing_local - ; Alcotest.test_case "diagnostics local cursor follows durable commit" `Quick - diagnostics_local_cursor_tracks_authoritative_commit - ] -;; - -let diagnostics_snapshot_metadata_is_remote_evidence () = - let activating = snapshot_activation_fixture () in - Alcotest.(check (option string)) "snapshot metadata observes remote progress before activation" - (Some "42") (sync_cursor_diagnostic activating.next "Remote tx"); - Alcotest.(check (option string)) "download cannot publish an uncommitted local checkpoint" - (Some "Not available") (sync_cursor_diagnostic activating.next "Local tx") -;; - -let scenarios = scenarios @ [ Alcotest.test_case - "diagnostics snapshot metadata is remote evidence" `Quick - diagnostics_snapshot_metadata_is_remote_evidence ] -;; - -let diagnostics_rejection_observes_remote_cursor () = - let core, connection = submitted_core () in - let rejection : Protocol.rejection = - { reason = Stale; t = Some (server_cursor_of_int 7); checksum = None - ; success_tx_ids = []; failed_tx_id = None; missing_block_uuids = [] - ; error_detail = None; data = None } in - let rejected = pull_message core connection (Protocol.Server.Tx_reject rejection) in - Alcotest.(check (option string)) "valid stale rejection observes server progress" - (Some "7") (sync_cursor_diagnostic rejected.next "Remote tx"); - Alcotest.(check (option string)) "rejection does not commit authoritative changes" - (Some "0") (sync_cursor_diagnostic rejected.next "Local tx") -;; - -let scenarios = scenarios @ [ Alcotest.test_case - "diagnostics rejection observes remote cursor" `Quick - diagnostics_rejection_observes_remote_cursor ] -;; - -(* Catalog advisory diagnostics are owned by this public reducer. A valid - producer error is not an injected incorrect external result. *) -let catalog_save_keeps_the_safe_producer_cause () = - let module D = Logseq_db_types.Failure_diagnostic in - let selected, _ = selected_graph graph in - let event = List.find_map (function - | Core.Run (Core.Request (ticket, Core.Save_catalog _)) -> - let diagnostic = D.create ~operation:"saveCatalog" ~reason:"persistenceFailed" - ~source:"Unix.Unix_error(Unix.ENOSPC,PRIVATE_PATH,PRIVATE_PATH)" () in - Some (Core.Runner_completed (Core.Completion (ticket, Error (Core.Effect_failed (D.encode diagnostic))))) - | _ -> None) selected.effects |> Option.get in - let transition = Core.step selected.next event in - ignore (check_replay "catalog-safe-error" selected.next event transition); - let before = Core.state selected.next and after = Core.state transition.next in - Alcotest.(check bool) "advisory failure preserves graph and startup state" true - (before.snapshot = after.snapshot); - let text = after.diagnostics.groups |> List.concat_map (fun (g:Core.diagnostic_group) -> List.map snd g.entries) - |> String.concat "\n" in - let contains text part = - let rec loop n = n + String.length part <= String.length text && - (String.sub text n (String.length part)=part || loop(n+1)) in loop 0 in - Alcotest.(check bool) "catalog advisory preserves original safe reason" true - (contains text "Persistence failed" && contains text "Storage has no free space"); - Alcotest.(check bool) "catalog advisory excludes raw path" false (contains text "PRIVATE_PATH"); - Alcotest.(check bool) "catalog advisory displays message, not a wire carrier" false - (contains text "journal-failure:v1:") -;; -let scenarios = scenarios @ [Alcotest.test_case "catalog save retains safe producer cause" `Quick - catalog_save_keeps_the_safe_producer_cause] -;; - -(* Resident membership is independent of graph lifetime and presentation. *) -let third_graph = { other_graph with graph_id = - Logseq_db_types.Graph_types.Uuid.of_string "33333333-3333-4333-8333-333333333333" |> Result.get_ok } -let fourth_graph = { other_graph with graph_id = - Logseq_db_types.Graph_types.Uuid.of_string "44444444-4444-4444-8444-444444444444" |> Result.get_ok } -let resident_catalog () = - let selected, _ = selected_graph graph in - (refresh_catalog selected.next [graph;other_graph;third_graph;fourth_graph]).next -let resident_scope core id = Core.scope_for_graph core ~graph_id:id |> Option.get -let resident_ids = [graph.graph_id;other_graph.graph_id;third_graph.graph_id] -let check_residents expected core = - Alcotest.(check bool) "configured resident order" true (Core.resident_graphs core = expected) -let resident_membership_limits_and_order () = - let original = resident_catalog () in - let one = Core.step original (Core.Resident_graphs_changed [graph.graph_id]) in - check_residents [graph.graph_id] one.next; - let two = Core.step one.next (Core.Resident_graphs_changed [graph.graph_id;other_graph.graph_id]) in - check_residents [graph.graph_id;other_graph.graph_id] two.next; - let three = Core.step two.next (Core.Resident_graphs_changed resident_ids) in - check_residents resident_ids three.next; - List.iter (fun invalid -> - let result = Core.step three.next (Core.Resident_graphs_changed invalid) in - check_residents resident_ids result.next; - check_instructions "invalid configuration emits no work" [] result.effects) - [resident_ids @ [fourth_graph.graph_id];[graph.graph_id;graph.graph_id];[fourth_graph.graph_id;Logseq_db_types.Graph_types.Uuid.of_string "55555555-5555-4555-8555-555555555555" |> Result.get_ok]]; - let reordered = Core.step three.next (Core.Resident_graphs_changed (List.rev resident_ids)) in - check_residents (List.rev resident_ids) reordered.next -let resident_focus_preserves_lifetimes () = - let configured = Core.step (resident_catalog ()) (Core.Resident_graphs_changed resident_ids) in - let owners = List.map (resident_scope configured.next) resident_ids in - let focused = List.fold_left (fun core id -> - let next = Core.step core (Core.Visible_graph_changed id) in - Alcotest.(check bool) "focus does not close/cancel graph operations" false - (List.exists (function Core.Delegate (Core.Detach_graph _) | Run (Core.Close_websocket _ | Cancel_effects _) -> true | _ -> false) next.effects); - Alcotest.(check bool) "requested graph becomes visible" true - ((Core.state next.next).snapshot.selected_graph = Some id); - List.iter2 (fun id owner -> Alcotest.(check bool) "original graph lifetime survives focus" true - (Core.scope_for_graph next.next ~graph_id:id = Some owner)) resident_ids owners; - next.next) configured.next (resident_ids @ [graph.graph_id]) in - check_residents resident_ids focused -let resident_removal_reopen_and_late_callback () = - let configured = Core.step (resident_catalog ()) (Core.Resident_graphs_changed resident_ids) in - let old = resident_scope configured.next other_graph.graph_id in - let removed = Core.step configured.next (Core.Resident_graphs_changed [graph.graph_id;third_graph.graph_id]) in - Alcotest.(check bool) "removal revokes only removed graph" true - (Core.scope_for_graph removed.next ~graph_id:other_graph.graph_id = None); - Alcotest.(check bool) "removal never deletes mirror/account" false - (List.exists (function Core.Delegate (Core.Delete_mirror _ | Reset_managed_account _) | Run (Core.Request (_, Core.Delete_account_secrets _)) -> true | _ -> false) removed.effects); - let reopened = Core.step removed.next (Core.Resident_graphs_changed resident_ids) in - let fresh = resident_scope reopened.next other_graph.graph_id in - Alcotest.(check bool) "reopened graph gets fresh lifetime" true (old <> fresh); - let late = Core.step reopened.next (Core.Graph_attached {scope=old;sync=Overlay.sync_view ~token:(Overlay.sync_token_of_string "sync-token:v1:1" |> Result.get_ok) ~checkpoint:(server_cursor_of_int 0) ~submissions:[]}) in - Alcotest.(check bool) "late old attachment cannot replace fresh scope" true - (Core.scope_for_graph late.next ~graph_id:other_graph.graph_id = Some fresh) -let resident_logout_clears_membership () = - let configured = Core.step (resident_catalog ()) (Core.Resident_graphs_changed resident_ids) in - let cleared = Core.step configured.next (Core.Account_authenticated {user_id=None}) in - check_residents [] cleared.next; - List.iter (fun id -> Alcotest.(check bool) "account retirement removes all scopes" true - (Core.scope_for_graph cleared.next ~graph_id:id = None)) resident_ids -let resident_catalog_codec_keeps_order () = - let configured = Core.step (resident_catalog ()) (Core.Resident_graphs_changed resident_ids) in - let focused = Core.step configured.next (Core.Visible_graph_changed other_graph.graph_id) in - let cache = List.find_map (function Core.Run (Core.Request (_, Core.Save_catalog {cache;_})) -> Some cache | _ -> None) focused.effects |> Option.get in - let decoded = Core.decode_catalog_cache (Core.encode_catalog_cache cache) |> Result.get_ok in - Alcotest.(check bool) "codec preserves resident order" true (Core.catalog_cache_resident_graphs decoded = resident_ids); - Alcotest.(check bool) "codec preserves visible graph" true (Core.catalog_cache_selected_graph decoded = Some other_graph.graph_id); - let old = Core.catalog_cache ~user_id:"user" ~graphs:[graph] ~selected_graph:(Some graph.graph_id) in - Alcotest.(check bool) "legacy single selection migrates" true (Core.catalog_cache_resident_graphs old = [graph.graph_id]) -let scenarios = scenarios @ - [ Alcotest.test_case "resident membership limits and order" `Quick resident_membership_limits_and_order - ; Alcotest.test_case "resident focus preserves lifetimes" `Quick resident_focus_preserves_lifetimes - ; Alcotest.test_case "resident remove reopen fences callbacks" `Quick resident_removal_reopen_and_late_callback - ; Alcotest.test_case "resident logout clears membership" `Quick resident_logout_clears_membership - ; Alcotest.test_case "resident catalog codec preserves order" `Quick resident_catalog_codec_keeps_order ] -;; - -let resident_restart_restores_membership_and_focus () = - let configured=Core.step (resident_catalog ()) (Core.Resident_graphs_changed (List.rev resident_ids)) in - let focused=Core.step configured.next (Core.Visible_graph_changed other_graph.graph_id) in - let cache=save_catalog_cache focused.effects |> Core.encode_catalog_cache |> Core.decode_catalog_cache |> Result.get_ok in - let restoring=Core.step (cold_start ()) (Core.Restore_local_account {user_id="user"}) in - let restored=Core.step restoring.next (load_catalog_completion restoring.effects (Some cache)) in - check_residents (List.rev resident_ids) restored.next; - Alcotest.(check bool) "restart restores selected resident" true - ((Core.state restored.next).snapshot.selected_graph=Some other_graph.graph_id); - List.iter (fun id -> Alcotest.(check bool) "restart creates each independent scope" true - (Option.is_some(Core.scope_for_graph restored.next ~graph_id:id))) resident_ids -;; -let resident_cache_rejects_invalid_membership () = - let configured=Core.step (resident_catalog ()) (Core.Resident_graphs_changed resident_ids) in - let cache=save_catalog_cache configured.effects |> Core.encode_catalog_cache |> Yojson.Safe.from_string in - let fields=Yojson.Safe.Util.to_assoc cache in - let uuid id=Logseq_db_types.Graph_types.Uuid.to_string id |> fun value -> `String value in - List.iter (fun ids -> - let source=`Assoc (("cache_resident_graphs",`List(List.map uuid ids))::List.remove_assoc "cache_resident_graphs" fields) - |> Yojson.Safe.to_string in - Alcotest.(check bool) "invalid durable membership is rejected" true - (Result.is_error(Core.decode_catalog_cache source))) - [resident_ids @ [fourth_graph.graph_id];[graph_id;graph_id];[third_graph.graph_id]] -;; -let resident_hidden_graphs_receive_shared_network_permission () = - let configured=Core.step (resident_catalog ()) (Core.Resident_graphs_changed [graph_id;other_graph_id]) in - let cache=save_catalog_cache configured.effects in - let restoring=Core.step (cold_start ()) (Core.Restore_local_account {user_id="user"}) in - let restored=Core.step restoring.next (load_catalog_completion restoring.effects (Some cache)) in - let attached=List.fold_left (fun core id -> - let scope=resident_scope core id in - let request=List.find_map (function Core.Delegate(Core.Inspect_mirror request) when request.scope=scope -> Some request|_->None) restored.effects |> Option.get in - let attaching=Core.step core (Core.Mirror_inspected(Core.Mirror_available request)) in - let token=Overlay.sync_token_of_string ("sync-token:v1:" ^ Logseq_db_types.Graph_types.Uuid.to_string id) |> Result.get_ok in - Core.step attaching.next (Core.Graph_attached {scope;sync=Overlay.sync_view ~token ~checkpoint:(server_cursor_of_int 0) ~submissions:[]}) - |> fun result->result.next) restored.next [graph_id;other_graph_id] in - let shown=Core.step attached (Core.Timeline_presented(timeline_ticket attached)) in - let online=Core.step shown.next (Core.Account_authenticated {user_id=Some "user"}) in - Alcotest.(check bool) "shared permission does not retire a connection still opening" false - (List.exists(function Core.Run(Core.Close_websocket _)->true|_->false) online.effects); - let sockets=List.filter_map (function Core.Run(Core.Start_websocket request)->Some request.scope.graph.graph_id|_->None) online.effects in - Alcotest.(check bool) "visible and hidden graph start independent sockets" true - (List.sort_uniq compare sockets=List.sort compare [graph_id;other_graph_id]); - List.iter (fun id -> let scope=resident_scope online.next id in - let state=Core.state_for_scope online.next scope |> Option.get in - Alcotest.(check bool) "network permission shared with hidden graph" true - (state.snapshot.startup.network_phase<>Core.Local_only)) [graph_id;other_graph_id] -;; -let scenarios=scenarios @ - [Alcotest.test_case "resident restart restores order and focus" `Quick resident_restart_restores_membership_and_focus - ;Alcotest.test_case "resident durable membership validation" `Quick resident_cache_rejects_invalid_membership - ;Alcotest.test_case "resident hidden network permission" `Quick resident_hidden_graphs_receive_shared_network_permission] -;; - -let resident_online_fixture () = - let configured=Core.step (resident_catalog ()) (Core.Resident_graphs_changed [graph_id;other_graph_id]) in - let cache=save_catalog_cache configured.effects in - let restoring=Core.step (cold_start ()) (Core.Restore_local_account {user_id="user"}) in - let restored=Core.step restoring.next (load_catalog_completion restoring.effects (Some cache)) in - let attached=List.fold_left (fun core id -> - let scope=resident_scope core id in - let request=List.find_map (function Core.Delegate(Core.Inspect_mirror request) when request.scope=scope -> Some request|_->None) restored.effects |> Option.get in - let attaching=Core.step core (Core.Mirror_inspected(Core.Mirror_available request)) in - let token=Overlay.sync_token_of_string ("sync-token:v1:" ^ Logseq_db_types.Graph_types.Uuid.to_string id) |> Result.get_ok in - Core.step attaching.next (Core.Graph_attached {scope;sync=Overlay.sync_view ~token ~checkpoint:(server_cursor_of_int 0) ~submissions:[]}) - |> fun result->result.next) restored.next [graph_id;other_graph_id] in - let shown=Core.step attached (Core.Timeline_presented(timeline_ticket attached)) in - Core.step shown.next (Core.Account_authenticated {user_id=Some "user"}) - -;; -let resident_online_cancel_retires_and_resumes_hidden () = - let online=resident_online_fixture () in - let scopes=List.filter_map(function Core.Run(Core.Start_websocket request)->Some request.scope|_->None) online.effects in - let opened=List.fold_left(fun core scope->(Core.step core (Core.Websocket_opened scope)).next) online.next scopes in - let cancelled=Core.step opened (Core.Online_cancelled(startup_ticket opened)) in - let closes=List.filter_map(function Core.Run(Core.Close_websocket scope)->Some scope|_->None) cancelled.effects in - Alcotest.(check int) "all graph policies enter physical retirement" 2 (List.length(List.sort_uniq compare closes)); - List.iter(fun early-> - let authenticated core=let sign_in=Core.step core (Core.Sign_in_requested(startup_ticket core)) in - Core.step sign_in.next (Core.Account_authenticated {user_id=Some "user"}) in - let ready,initial_effects=if early then let auth=authenticated cancelled.next in auth.next,auth.effects - else cancelled.next,[] in - let retired=List.fold_left(fun result scope-> - let part=Core.step result.Core.next (Core.Websocket_retired scope) in - {Core.next=part.next;effects=result.effects @ part.effects}) {Core.next=ready;effects=initial_effects} scopes in - let resumed=if early then retired else let auth=authenticated retired.next in {auth with effects=retired.effects @ auth.effects} in - let starts=List.filter_map(function Core.Run(Core.Start_websocket request)->Some request.scope.graph.graph_id|_->None) resumed.effects in - Alcotest.(check bool) "hidden graph resumes both before and after terminal retirement" true - (List.sort_uniq compare starts=List.sort compare [graph_id;other_graph_id])) [false;true] -;; -let scenarios=scenarios @ [Alcotest.test_case "resident online cancellation resumes every graph" `Quick resident_online_cancel_retires_and_resumes_hidden] -;; - -(* PR142 recovery must preserve resident lifetime and account presentation rules. *) -let resident_recovery_allocates_above_all_live_generations () = - let configured=Core.step (resident_catalog ()) (Core.Resident_graphs_changed resident_ids) in - let owners=List.map(resident_scope configured.next) resident_ids in - let a=List.hd owners in - let failed=Core.step configured.next (Core.Graph_attachment_failed {scope=Core.effect_scope_of_graph a;message="Synthetic local open failure"}) in - let recovered=Core.step failed.next (Core.Online_recovery_requested(timeline_scope failed.next)) in - let fresh=inspect_mirror_request recovered.effects in - Alcotest.(check bool) "recovery allocates above every resident, not the failing session" true - (List.for_all(fun (owner:Core.graph_scope)->fresh.scope.graph_generation>owner.graph_generation) owners); - Alcotest.(check bool) "fresh recovery scope remains admitted" true - (Core.scope_for_graph recovered.next ~graph_id=Some fresh.scope); - List.iter(fun (owner:Core.graph_scope)->if owner.graph_id<>graph_id then - Alcotest.(check bool) "other lifetimes unchanged" true - (Core.scope_for_graph recovered.next ~graph_id:owner.graph_id=Some owner)) owners; - let stale=Core.step recovered.next (Core.Mirror_inspected(Core.Mirror_available {fresh with scope=a})) in - Alcotest.(check bool) "old recovery callback cannot attach" false - (List.exists(function Core.Delegate(Core.Attach_graph _)->true|_->false) stale.effects); - let attached=Core.step stale.next (Core.Mirror_inspected(Core.Mirror_available fresh)) in - Alcotest.(check bool) "new recovery callback keeps its graph" true - (List.exists(function Core.Delegate(Core.Attach_graph r)->r.scope=fresh.scope|_->false) attached.effects) -;; -let resident_focus_keeps_native_presentation_monotonic () = - let configured=Core.step (resident_catalog ()) (Core.Resident_graphs_changed resident_ids) in - let owners=List.map(resident_scope configured.next) resident_ids in - let newer=Core.step configured.next (Core.Visible_graph_changed graph_id) in - let native=startup_scope newer.next in - let focused=Core.step newer.next (Core.Visible_graph_changed other_graph_id) in - let visible=startup_scope focused.next in - Alcotest.(check bool) "native scope never regresses on focus" true - (visible.presentation_generation>=native.presentation_generation); - List.iter2(fun id scope->Alcotest.(check bool) "presentation refresh preserves graph lifetime" true - (Core.scope_for_graph focused.next ~graph_id:id=Some scope)) resident_ids owners; - let cancel=Core.step focused.next (Core.Online_cancelled visible) in - Alcotest.(check bool) "native current-scope cancellation reaches focused graph" true - ((Core.state cancel.next).snapshot.startup.network_phase=Core.Local_only); - let stale=Core.step cancel.next (Core.Sign_in_requested visible) in - Alcotest.(check bool) "cancel retires old presentation tickets" true - ((Core.state stale.next).snapshot.startup.network_phase=Core.Local_only); - let resumed=Core.step cancel.next (Core.Sign_in_requested(startup_scope cancel.next)) in - Alcotest.(check bool) "fresh native authentication can resume" true - ((Core.state resumed.next).snapshot.startup.network_phase=Core.Online_by_user) -;; -let resident_local_retry_restarts_only_failed_graph () = - let configured=Core.step (resident_catalog ()) (Core.Resident_graphs_changed resident_ids) in - let old=resident_scope configured.next graph_id in - let failed=Core.step configured.next (Core.Graph_attachment_failed {scope=Core.effect_scope_of_graph old;message="Synthetic read failure"}) in - let retry=Core.step failed.next (Core.Graph_selected graph_id) in - let request=inspect_mirror_request retry.effects in - Alcotest.(check bool) "retry admits its new local-open scope" true - (Core.scope_for_graph retry.next ~graph_id=Some request.scope); - Alcotest.(check bool) "retry renews only failed lifetime" true (request.scope<>old); - Alcotest.(check bool) "retry retains configured order" true (Core.resident_graphs retry.next=resident_ids); - let inspected=Core.step retry.next (Core.Mirror_inspected(Core.Mirror_available request)) in - Alcotest.(check bool) "retried mirror advances to attachment" true - (List.exists(function Core.Delegate(Core.Attach_graph r)->r.scope=request.scope|_->false) inspected.effects) -;; -let scenarios=scenarios @ [ - Alcotest.test_case "resident recovery uses global lifetime allocator" `Quick resident_recovery_allocates_above_all_live_generations; - Alcotest.test_case "resident focus preserves native presentation scope" `Quick resident_focus_keeps_native_presentation_monotonic; - Alcotest.test_case "resident local retry keeps renewed scope admitted" `Quick resident_local_retry_restarts_only_failed_graph] -;; - -let resident_authentication_recovery_renews_scope () = - let configured=Core.step (resident_catalog ()) (Core.Resident_graphs_changed resident_ids) in - let old=resident_scope configured.next graph_id in - let cancelled=Core.step configured.next (Core.Online_cancelled(startup_scope configured.next)) in - let failed=Core.step cancelled.next (Core.Graph_attachment_failed{scope=Core.effect_scope_of_graph old;message="Synthetic local restore failure"}) in - let online=Core.step failed.next (Core.Sign_in_requested(startup_scope failed.next)) in - let recovered=Core.step online.next (Core.Account_authenticated{user_id=Some "user"}) in - let request=inspect_mirror_request recovered.effects in - Alcotest.(check bool) "authentication recovery registers fresh graph lifetime" true - (Core.scope_for_graph recovered.next ~graph_id=Some request.scope); - List.iter(fun id->let owner=resident_scope configured.next id in - Alcotest.(check bool) "authentication recovery generation unique" true - (request.scope.graph_generation>owner.graph_generation)) resident_ids -;; -let scenarios=scenarios @ [Alcotest.test_case "resident authentication recovery uses global lifetime" `Quick resident_authentication_recovery_renews_scope] -;; - -let resident_recovery_keeps_other_graph_completion () = - let catalog=refresh_catalog (resident_catalog ()) [graph;{other_graph with encrypted=true};third_graph] in - let configured=Core.step catalog.next (Core.Resident_graphs_changed resident_ids) in - let b=resident_scope configured.next other_graph_id in - let b_request=List.find_map(function Core.Delegate(Core.Inspect_mirror r) when r.scope=b->Some r|_->None) configured.effects |> Option.get in - let unlocking=Core.step configured.next (Core.Mirror_inspected(Core.Mirror_available b_request)) in - let fail_b=List.find_map(function Core.Run(Core.Request(ticket,Core.Load_and_unlock_graph_key _))-> - Some(Core.Runner_completed(Core.Completion(ticket,Ok(Core.graph_key_handle ~id:"synthetic-B-key" ~scope:b))))|_->None) unlocking.effects |> Option.get in - let a=resident_scope unlocking.next graph_id in - let failed=Core.step unlocking.next (Core.Graph_attachment_failed{scope=Core.effect_scope_of_graph a;message="Synthetic A open failure"}) in - let recovered=Core.step failed.next (Core.Online_recovery_requested(timeline_scope failed.next)) in - let fresh=inspect_mirror_request recovered.effects in - let completed=Core.step recovered.next fail_b in - Alcotest.(check bool) "B completion advances only B attachment after A recovery" true - (List.exists(function Core.Delegate(Core.Attach_graph request)->request.scope=b|_->false) completed.effects); - Alcotest.(check bool) "B completion keeps A recovery pending" true - ((Core.state completed.next).snapshot.startup.restoring_local); - Alcotest.(check bool) "A scope still admits its own mirror response" true - (Core.scope_for_graph completed.next ~graph_id=Some fresh.scope) -;; -let scenarios=scenarios @ [Alcotest.test_case "resident recovery preserves other key completion" `Quick resident_recovery_keeps_other_graph_completion] -;; - -let resident_reopened_lifetime_finishes_owned_retirement () = - let online=resident_online_fixture () in - let sockets=List.filter_map(function Core.Run(Core.Start_websocket r)->Some r.scope|_->None) online.effects in - let opened=List.fold_left(fun c socket->(Core.step c (Core.Websocket_opened socket)).next) online.next sockets in - let old=resident_scope opened graph_id in - let pending,ticket,_=asset_issue opened old "old-resident-lease" (Core.Retain_asset_file "source") in - let replaced=Core.step pending.next (Core.Graph_selected graph_id) in - let mirror=inspect_mirror_request replaced.effects in - let stale=Core.step replaced.next (Core.Local_graph_outbox_changed old) in - check_instructions "retired outbox cannot request work in new lifetime" [] stale.effects; - let late=asset_complete stale.next ticket (Ok(Core.Asset_retained(Some("late-resident-lease","synthetic-path")))) in - Alcotest.(check int) "retired success never publishes the resource" 0 (List.length(asset_outputs late.effects)); - let cleanup=List.find_map(function Core.Run(Core.Asset_io(ticket,request)) when request.action=Core.Release_asset_file "late-resident-lease"->Some ticket|_->None) late.effects in - Alcotest.(check bool) "new lifetime retains exact old asset cleanup owner" true (Option.is_some cleanup); - let released=asset_complete late.next (Option.get cleanup) (Ok Core.Asset_unit) in - let duplicate=asset_complete released.next ticket (Ok(Core.Asset_retained(Some("late-resident-lease","synthetic-path")))) in - check_instructions "retired asset completion is consumed once" [] duplicate.effects; - let attaching=Core.step duplicate.next (Core.Mirror_inspected(Core.Mirror_available mirror)) in - let module O=Logseq_overlay_db.Types in - let sync=O.sync_view ~token:(O.sync_token_of_string "sync-token:v1:reopened" |> Result.get_ok) - ~checkpoint:(server_cursor_of_int 0) ~submissions:[] in - let attached=Core.step attaching.next (Core.Graph_attached{scope=mirror.scope;sync}) in - let shown=Core.step attached.next (Core.Timeline_presented(timeline_scope attached.next)) in - let authenticated=Core.step shown.next (Core.Account_authenticated{user_id=Some "user"}) in - let socket=List.find(fun(s:Core.connection_scope)->s.graph=old) sockets in - let retired=Core.step authenticated.next (Core.Websocket_retired socket) in - Alcotest.(check int) "old physical retirement starts new socket once" 1 - (List.length(List.filter(function Core.Run(Core.Start_websocket r)->r.scope.graph=mirror.scope|_->false) retired.effects)); - check_instructions "duplicate old physical retirement is inert" [] (Core.step retired.next (Core.Websocket_retired socket)).effects -;; -let scenarios=scenarios @ [Alcotest.test_case "resident refreshed lifetime drains exact old owners" `Quick resident_reopened_lifetime_finishes_owned_retirement] -;; - -let resident_cancel_publishes_final_presentation () = - List.iter(fun count->let ids=List.filteri(fun i _->i - let configured=Core.step (resident_catalog ()) (Core.Resident_graphs_changed ids) in - let focused=Core.step configured.next (Core.Visible_graph_changed visible) in - let cancelled=Core.step focused.next (Core.Online_cancelled(startup_scope focused.next)) in - let published=List.fold_left(fun state->function Core.Publish(Core.State_changed value)->Some value|_->state) None cancelled.effects |> Option.get in - Alcotest.(check bool) "native receives final account presentation after every cancellation order" true - (published.snapshot.startup=(Core.state cancelled.next).snapshot.startup)) ids) [1;2;3] -;; -let resident_retry_retains_new_catalog_completion () = - let configured=Core.step (resident_catalog ()) (Core.Resident_graphs_changed resident_ids) in - let retry=Core.step configured.next (Core.Graph_selected graph_id) in - let error=Core.Effect_failed "Synthetic resident catalog persistence failure" in - let completion=List.find_map(function Core.Run(Core.Request(ticket,Core.Save_catalog _))-> - Some(Core.Runner_completed(Core.Completion(ticket,Error error)))|_->None) retry.effects |> Option.get in - let failed=Core.step retry.next completion in - Alcotest.(check bool) "new retry catalog ticket preserves persistence failure diagnostic" true - ((Core.state failed.next).diagnostics<>(Core.state retry.next).diagnostics); - check_instructions "catalog persistence completion consumed once" [] (Core.step failed.next completion).effects -;; -let scenarios=scenarios @ [ - Alcotest.test_case "resident cancellation publishes final native scope" `Quick resident_cancel_publishes_final_presentation; - Alcotest.test_case "resident retry preserves catalog persistence ticket" `Quick resident_retry_retains_new_catalog_completion] -;; - -let resident_graph_failure_preserves_account_catalog_request () = - let configured=Core.step (resident_catalog ()) (Core.Resident_graphs_changed resident_ids) in - let requested=Core.step configured.next (Core.Account_authenticated{user_id=Some "user"}) in - let updated_graph={graph with name="Authoritative catalog update"} in - let completion=List.find_map(function Core.Run(Core.Request(ticket,Core.Fetch_catalog _))-> - Some(Core.Runner_completed(Core.Completion(ticket,Ok[updated_graph;other_graph;third_graph;fourth_graph])))|_->None) requested.effects |> Option.get in - let hidden=resident_scope requested.next other_graph_id in - let failed=Core.step requested.next (Core.Graph_attachment_failed{scope=Core.effect_scope_of_graph hidden;message="Synthetic hidden graph failure"}) in - let completed=Core.step failed.next completion in - Alcotest.(check bool) "hidden graph failure cannot discard shared account catalog completion" true - (List.exists(fun(g:Core.graph)->g.graph_id=graph_id && g.name=updated_graph.name)(Core.state completed.next).snapshot.catalog); - Alcotest.(check bool) "shared catalog completion releases loading" false - (Core.state completed.next).snapshot.startup.catalog_loading; - check_instructions "catalog request completion consumed once" [] (Core.step completed.next completion).effects -;; -let scenarios=scenarios @ [Alcotest.test_case "resident graph failure preserves account catalog ticket" `Quick resident_graph_failure_preserves_account_catalog_request] -;; - -(* The reducer owns deletion failure projection; valid producer diagnostics must - survive its public completion boundary without exposing arbitrary source text. *) -let local_deletion_preserves_cause () = - let module D = Logseq_db_types.Failure_diagnostic in - List.iter (fun source -> - let original, _, _, _ = deletion_ready_graph false in - let ready, _, _, _ = deletion_noncurrent original in - let deleting = Core.step ready (Core.Local_cache_deletion_requested graph_id) in - let request = deletion_mirror deleting.effects in - let event = Core.Mirror_deleted (request, Error source) in - let failed = Core.step deleting.next event in - ignore (check_replay "deletion-cause" deleting.next event failed); - let diagnostics = (Core.state failed.next).diagnostics.groups in - let diagnostic = List.find_map (fun (group : Core.diagnostic_group) -> - List.assoc_opt "deleteMirror" group.entries) diagnostics in - let expected = D.of_source ~operation:"deleteMirror" ~reason:"deleteFailed" source in - Alcotest.(check (option string)) "deletion publishes the bounded concrete cause" - (Some (D.message expected)) diagnostic; - check_no_work failed.next [event]) - [ D.create ~operation:"deleteMirror" ~reason:"persistenceFailed" - ~source:"Unix.Unix_error(Unix.ENOSPC,PRIVATE_PATH,PRIVATE_PATH)" () |> D.encode - ; D.create ~operation:"deleteMirror" ~reason:"ownershipConflict" () |> D.encode - ; "Bearer PRIVATE_TOKEN https://user:PRIVATE_PASSWORD@host/PRIVATE_GRAPH" - ] -;; -let scenarios = scenarios @ [Alcotest.test_case "local deletion preserves concrete safe cause" - `Quick local_deletion_preserves_cause] -;; - -let authoritative_input_retains_validation_cause () = - let transaction t tx = Protocol.Server.{ t = server_cursor_of_int t; tx; outliner_op = None } in - List.iter (fun (txs, through, expected) -> - let opened, connection, _ = pull_fixture () in - let event = Core.Websocket_message (connection, Protocol.Server.Pull_ok - { t = server_cursor_of_int through; checksum = None; txs }) in - let failed = Core.step opened.next event in - ignore (check_replay "authoritative-input-cause" opened.next event failed); - Alcotest.(check (option string)) "typed validation cause survives to caller" - (Some expected) (Core.state failed.next).snapshot.last_error) - [ [transaction 1 ""], 1, "authoritative transaction is empty or its byte limit is invalid" - ; [transaction 1 (String.make (Logseq_db_types.Limits.maximum_response_bytes + 1) 'x')], 1, - "authoritative transaction exceeds the byte limit" - ; [transaction 1 "[]"; transaction 1 "[]"], 1, - "authoritative transaction cursors are not strictly ordered" - ; [transaction 2 "[]"; transaction 1 "[]"], 1, - "authoritative transaction cursors are not strictly ordered" - ; [transaction 1 "[]"], 2, "authoritative batch final cursor does not match its through cursor" - ; List.init 4097 (fun i -> transaction (i + 1) "[]"), 4097, - "authoritative batch exceeds its transaction count or byte limit" - ] -;; -let scenarios = scenarios @ [Alcotest.test_case "authoritative input retains validation cause" - `Quick authoritative_input_retains_validation_cause] -;; - -let cache_decoder_keeps_safe_field_cause () = - List.iter (fun ((source : string), expected) -> - Alcotest.(check (option string)) "cache parser retains field cause without data" - (Some expected) - (match Core.decode_catalog_cache source with Error message -> Some message | Ok _ -> None)) - [ {|{"cache_user_id":false,"cache_graphs":[],"cache_selected_graph":null}|}, - "catalog cache: field cache_user_id is missing or has an invalid type" - ; {|{"cache_user_id":"PRIVATE_USER","cache_graphs":false,"cache_selected_graph":null}|}, - "catalog cache: field cache_graphs is missing or has an invalid type" - ; {|{"cache_user_id":"PRIVATE_USER","cache_user_id":"PRIVATE_USER2","cache_graphs":[],"cache_selected_graph":null}|}, - "catalog cache: duplicate JSON field" - ] -;; -let snapshot_metadata_keeps_safe_field_cause () = - List.iter (fun ((source : string), expected) -> - let requested, _ = begin_snapshot_bootstrap () in - let failed = List.find_map (function - | Core.Run (Core.Request (ticket, Core.Fetch_snapshot_metadata _)) -> - Some (Core.step requested.next (Core.Runner_completed (Core.Completion (ticket, Ok source)))) - | _ -> None) requested.effects |> Option.get in - Alcotest.(check (option string)) "metadata parser publishes field cause without URL" - (Some expected) (Core.state failed.next).snapshot.last_error) - [ {|{"ok":true,"t":"PRIVATE_BODY","url":"https://user:PRIVATE_PASSWORD@example.com"}|}, - "snapshot metadata: field t is missing or has an invalid type" - ; {|{"ok":true,"t":1,"url":false}|}, - "snapshot metadata: field url is missing or has an invalid type" - ; {|{"ok":false,"t":1,"url":"https://example.com"}|}, - "snapshot metadata: server reported ok=false" - ] -;; -let scenarios = scenarios @ [ - Alcotest.test_case "cache decoder keeps safe field cause" `Quick cache_decoder_keeps_safe_field_cause; - Alcotest.test_case "snapshot metadata keeps safe field cause" `Quick snapshot_metadata_keeps_safe_field_cause] -;; - -(* The sync reducer owns batching: descriptors arrive through its public inspected - view, rather than injecting a rejected storage transition. *) -let submission_operation_batches operations expected_lengths () = - let descriptors = - List.mapi (fun index operation -> - { Overlay.mutation_id = - Logseq_db_types.Graph_types.Uuid.of_string - (Printf.sprintf "00000000-0000-4000-8000-%012d" (index + 1)) - |> Result.get_ok - ; fingerprint = token Overlay.Mutation_fingerprint.of_string - (Printf.sprintf "mutation-fingerprint:v1:batch-%d" index) - ; operation - ; state = Overlay.Queued - ; dependency_eligible = true - ; attempt_count = 0 - ; plaintext_bytes = 16 - ; protected_bytes = Some 16 - }) operations - in - let sync ordinal submissions = recovery_sync () ordinal submissions in - let core, connection = restore_recovery_core (sync 0 []) in - let inspecting = Core.step core Core.Local_outbox_changed in - let first_event = Core.Sync_inspected - { scope = connection.graph; sync = sync 1 descriptors } in - let first = Core.step inspecting.next first_event in - ignore (check_replay "operation batch" inspecting.next first_event first); - let rec drain (planned : Core.transition) ordinal checkpoint remaining = function - | [] -> - Alcotest.(check int) "no stranded mutations" 0 (List.length remaining); - require_no_new_submission planned.effects - | count :: counts -> - let request = recovery_request planned.effects in - let rec split n acc xs = match n, xs with - | 0, _ | _, [] -> List.rev acc, xs - | _, x :: tail -> split (n - 1) (x :: acc) tail in - let expected, rest = split count [] remaining in - Alcotest.(check bool) "ordered contiguous submission" - true (request.transition = Overlay.Submit_group - (List.map (fun (item : Overlay.submission_descriptor) -> item.mutation_id) expected)); - let batch_id = token Overlay.Submission_batch_id.of_string - (Printf.sprintf "submission-batch:v1:operation-%d" ordinal) in - let wires = List.map (fun (item : Overlay.submission_descriptor) -> - Overlay.submission_wire ~maximum_bytes:1024 ~mutation_id:item.mutation_id - ~operation:item.operation ~protected_transaction:"[]" |> Result.get_ok) expected in - let batch = Overlay.submission_batch ~maximum_wires:32 ~maximum_bytes:4096 - ~batch_id ~t_before:(recovery_cursor checkpoint) ~wires |> Result.get_ok in - let view state ordinal = recovery_sync () ~checkpoint ordinal - (List.map (fun (item : Overlay.submission_descriptor) -> {item with state}) expected @ rest) in - let submitted = Core.step planned.next - (recovery_commit request (view (Overlay.Submitted batch_id) (ordinal + 1)) (Some batch)) in - let through = checkpoint + count in - let acknowledged = Core.step submitted.next - (Core.Websocket_message (connection, Protocol.Server.Tx_batch_ok - {t = server_cursor_of_int through; checksum = Some "0123456789abcdef"})) in - let accept = recovery_request acknowledged.effects in - let accepted = Core.step acknowledged.next - (recovery_commit accept - (view (Overlay.Accepted_pending_authoritative batch_id) (ordinal + 2)) None) in - require_no_new_submission accepted.effects; - let pulled = Core.step accepted.next - (Core.Websocket_message (connection, Protocol.Server.Pull_ok - { t = server_cursor_of_int through; checksum = Some "0123456789abcdef" - ; txs = List.init count (fun i -> Protocol.Server. - {t = server_cursor_of_int (checkpoint + i + 1); tx = "[]"; outliner_op = None}) })) in - let applied = recovery_apply pulled.next connection - (recovery_sync () ~checkpoint:through (ordinal + 3) rest) in - drain applied (ordinal + 4) through rest counts - in - drain first 1 0 descriptors expected_lengths -;; - -let scenarios = scenarios @ - let open Overlay in - let normal = Append_journal_operation and delete = Delete_blocks_operation in - List.map (fun (name, operations, groups) -> - Alcotest.test_case name `Quick (submission_operation_batches operations groups)) - [ "delete first stays singleton", [delete; normal; normal], [1; 2] - ; "delete middle preserves normal batches", [normal; normal; delete; normal; normal], [2; 1; 2] - ; "delete last stays singleton", [normal; normal; delete], [2; 1] - ; "two deletes drain as singletons", [delete; delete], [1; 1] - ; "normal mutations retain batching", [Save_block_operation; Insert_blocks_operation; - Append_journal_operation; Create_journal_page_operation; Set_task_status_operation; - Clear_task_status_operation; Publish_asset_operation], [7] - ; "normal batch count cap remains", List.init 34 (fun _ -> normal), [32; 2] - ; "delete at batch cap remains singleton", List.init 32 (fun _ -> normal) @ [delete; normal], [32; 1; 1] - ] -;; - -let nonresident_scoped_outbox_notice () = - let core, connection = restore_recovery_core (recovery_sync () 0 []) in - let scope = connection.graph in - let changed = Core.step core (Core.Local_graph_outbox_changed scope) in - check_instructions "matching scoped notice" [Core.Delegate (Core.Inspect_sync scope)] changed.effects; - List.iter (fun stale -> - let event = Core.Local_graph_outbox_changed stale in - let ignored = Core.step core event in - check_instructions "stale scoped notice" [] ignored.effects; - check_observed "stale scoped notice" "state unchanged" (observe core) (observe ignored.next)) - [ {scope with graph_id = other_graph_id} - ; {scope with graph_generation = scope.graph_generation + 1} - ; {scope with account = {scope.account with account_generation = scope.account.account_generation + 1}} - ; {scope with account = {scope.account with lifecycle_generation = Int64.succ scope.account.lifecycle_generation}} ] -;; -let scenarios = scenarios @ [Alcotest.test_case "nonresident scoped outbox notice is fenced" - `Quick nonresident_scoped_outbox_notice] -;; - -let submission_keeps_dependency_eligibility () = - let make index operation dependency_eligible : Overlay.submission_descriptor = - { mutation_id = Logseq_db_types.Graph_types.Uuid.of_string - (Printf.sprintf "00000000-0000-4000-8000-%012d" index) |> Result.get_ok - ; fingerprint = token Overlay.Mutation_fingerprint.of_string - (Printf.sprintf "mutation-fingerprint:v1:dependency-%d" index) - ; operation; dependency_eligible; state = Overlay.Queued; attempt_count = 0 - ; plaintext_bytes = 16; protected_bytes = Some 16 } - in - let ineligible = make 1 Overlay.Delete_blocks_operation false in - let eligible = make 2 Overlay.Append_journal_operation true in - List.iter (fun (descriptors, expected) -> - let core, connection = restore_recovery_core (recovery_sync () 0 []) in - let inspecting = Core.step core Core.Local_outbox_changed in - let planned = Core.step inspecting.next (Core.Sync_inspected - {scope = connection.graph; sync = recovery_sync () 1 descriptors}) in - match expected with - | [] -> require_no_new_submission planned.effects - | ids -> Alcotest.(check bool) "ineligible descriptors are never submitted" - true ((recovery_request planned.effects).transition = Overlay.Submit_group ids)) - [ [ineligible], []; [ineligible; eligible], [eligible.mutation_id] ] -;; -let scenarios = scenarios @ [Alcotest.test_case "submission respects dependency eligibility" - `Quick submission_keeps_dependency_eligibility] -;; - -let superseded_outbox_request_recovers kind () = - let first=List.hd mutation_ids in - let descriptor state : Overlay.submission_descriptor = - {mutation_id=first;fingerprint=token Overlay.Mutation_fingerprint.of_string "mutation-fingerprint:v1:superseded"; - operation=Overlay.Append_journal_operation;state;dependency_eligible=true;attempt_count=0; - plaintext_bytes=16;protected_bytes=Some 16} in - let pending = match kind with - | `Submit | `Retry -> - let state=match kind with `Retry->Overlay.Submitted(token Overlay.Submission_batch_id.of_string "submission-batch:v1:superseded")|_->Overlay.Queued in - let core,connection=restore_recovery_core(recovery_sync() 80 []) in - let inspecting=Core.step core Core.Local_outbox_changed in - Core.step inspecting.next(Core.Sync_inspected{scope=connection.graph;sync=recovery_sync() 81 [descriptor state]}) - | `Accept | `Reject -> - let core,connection=submitted_core()in - let message=match kind with - |`Accept->Protocol.Server.Tx_batch_ok{t=server_cursor_of_int 3;checksum=Some "0123456789abcdef"} - |_->Protocol.Server.Tx_reject{reason=Db_transact_failed;t=Some(server_cursor_of_int 1); - checksum=Some "0123456789abcdef";success_tx_ids=[first];failed_tx_id=Some(List.nth mutation_ids 1); - missing_block_uuids=[];error_detail=None;data=None} in - Core.step core(Core.Websocket_message(connection,message)) in - let request=recovery_request pending.effects in - let fresh=recovery_sync() 82 [descriptor Overlay.Queued] in - let superseded request sync=Core.Outbox_transition_superseded{request;sync} in - let changed=Core.step pending.next(superseded request fresh) in - let retry=recovery_request changed.effects in - Alcotest.(check bool) "recovery advances only to observed fresh token" true - (Overlay.sync_token_equal retry.expected(Overlay.sync_view_token fresh)); - Alcotest.(check bool) "same graph and key retained" true (retry.scope=request.scope && retry.key=request.key); - (match kind with - |`Accept|`Reject->Alcotest.(check bool) "exact validated server outcome retained" true (retry.transition=request.transition) - |`Submit|`Retry->Alcotest.(check bool) "submission replans from current queued descriptors" true - (retry.transition=Overlay.Submit_group[first])); - check_instructions "superseded completion cannot replay after token advances" [] - (Core.step changed.next(superseded request fresh)).effects; - let same_token=Overlay.sync_view ~token:request.expected ~checkpoint:(recovery_cursor 0) ~submissions:[] in - check_instructions "unchanged token never retries" [] (Core.step pending.next(superseded request same_token)).effects; - List.iter(fun stale->check_instructions "wrong pending owner ignored" [] - (Core.step pending.next(superseded stale fresh)).effects) - [{request with scope={request.scope with graph_generation=request.scope.graph_generation+1}}; - {request with scope={request.scope with account={request.scope.account with lifecycle_generation=Int64.succ request.scope.account.lifecycle_generation}}}; - {request with expected=Overlay.sync_view_token fresh}; - {request with transition=Overlay.Submit_group[]}]; - let inspecting=Core.step pending.next Core.Local_outbox_changed in - let latest=recovery_sync() 83 [descriptor Overlay.Queued] in - let observed=Core.step inspecting.next(Core.Sync_inspected{scope=request.scope;sync=latest}) in - let recovered=Core.step observed.next(superseded request fresh) in - Alcotest.(check bool) "late superseded result cannot replace independently refreshed sync view" true - (Overlay.sync_token_equal(recovery_request recovered.effects).expected(Overlay.sync_view_token latest)); - let signed_out=Core.step pending.next(Core.Account_authenticated{user_id=None})in - check_instructions "retired account ignores superseded preparation" [] - (Core.step signed_out.next(superseded request fresh)).effects -;; -let scenarios=scenarios @ List.map(fun(name,kind)->Alcotest.test_case name `Quick(superseded_outbox_request_recovers kind)) - ["superseded submit replans and fences",`Submit;"superseded retry replans and fences",`Retry; - "superseded accept preserves outcome",`Accept;"superseded reject preserves partition",`Reject] -;; diff --git a/logseq_sync/test/crypto_contract.ml b/logseq_sync/test/crypto_contract.ml new file mode 100644 index 00000000..9e708508 --- /dev/null +++ b/logseq_sync/test/crypto_contract.ml @@ -0,0 +1,190 @@ +module D = Sync_dsl.Core +module C = Logseq_sync_crypto.Core + +let get = function + | Ok value -> value + | Error error -> Alcotest.fail (D.error_message error) +;; + +let identity account = + get + (D.identity + ~origin:"https://example.com" + ~account + ~graph:"g" + ~account_generation:0L + ~graph_generation:0L) +;; + +let limits = + get + (D.limits + ~maximum_items:8 + ~maximum_bytes:1024 + ~maximum_pending:8 + ~maximum_resources:8 + ~steps_per_turn:8 + ~maximum_tasks:8 + ~maximum_messages:8 + ~maximum_message_bytes:1024 + ~maximum_demands:8 + ~maximum_active_assets:2 + ~foreground_reserve:1 + ~foreground_demand_reserve:1 + ~foreground_descriptor_reserve_bytes:128 + ~asset_cache_budget_bytes:4096L + ~asset_maximum_file_bytes:1024 + ~asset_pending_budget_bytes:4096L + ~snapshot_maximum_artifact_bytes:4096L + ~snapshot_staging_budget_bytes:4096L + ~snapshot_scratch_budget_bytes:4096L) +;; + +let create encrypted = + C.create ~identity:(identity "a") ~limits ~encrypted ~backend:C.apple_backend +;; + +let run t operation = + C.run t ~identity:(identity "a") ~check_current:(fun () -> Ok ()) operation +;; + +let check_error expected = function + | Error error -> Alcotest.(check string) "error" expected (D.error_message error) + | Ok _ -> Alcotest.fail "Expected rejection" +;; + +let unencrypted () = + let t = create false in + Alcotest.(check string) "plain string" "hello" (get (run t (D.Decrypt_value "hello"))); + Alcotest.(check string) + "Transit string" + "hello" + (get (run t (D.Decrypt_value "\"hello\""))); + check_error + (D.error_message (D.Crypto_failure "Encryption requires a graph key")) + (run t (D.Encrypt_values (Rrbvec.of_list [ "secret" ]))) +;; + +let locked () = + let t = create true in + check_error (D.error_message D.Locked) (run t (D.Decrypt_value "hello")); + check_error + (D.error_message D.Stale_context) + (C.run + t + ~identity:(identity "a") + ~check_current:(fun () -> Error D.Stale_context) + (D.Decrypt_value "hello")) +;; + +let ownership () = + let t = create false in + check_error + (D.error_message (D.Identity_mismatch "Crypto owner")) + (C.run + t + ~identity:(identity "other") + ~check_current:(fun () -> Ok ()) + (D.Decrypt_value "hello")); + C.retire t; + check_error (D.error_message D.Stale_context) (run t (D.Decrypt_value "hello")) +;; + +let bounds () = + let t = create false in + check_error + (D.error_message (D.Capacity_exceeded "Crypto bytes")) + (run t (D.Decrypt_value (String.make 1025 'a'))); + check_error + (D.error_message (D.Crypto_failure "Protected value must be a string")) + (run t (D.Decrypt_value "42")) +;; + +let encrypted_roundtrip () = + let module Aes = Mirage_crypto.AES.GCM in + let backend : C.backend = + { unlock = (fun ~identity:_ ~wrapped_graph_key:_ -> Ok (String.make 32 'k')) + ; crypto = + { encrypt_aes_gcm = + (fun ~key ~plaintext -> + let iv = String.make 12 'i' in + Ok + (iv, Aes.authenticate_encrypt ~key:(Aes.of_secret key) ~nonce:iv plaintext)) + ; decrypt_aes_gcm = + (fun ~key ~iv ~ciphertext -> + match + Aes.authenticate_decrypt ~key:(Aes.of_secret key) ~nonce:iv ciphertext + with + | Some plaintext -> Ok plaintext + | None -> Error "Authentication failed") + } + } + in + let t = C.create ~identity:(identity "a") ~limits ~encrypted:true ~backend in + let input = get (D.unlock_input ~wrapped_graph_key:"wrapped") in + ignore (get (run t (D.Unlock input))); + let encrypted = + get (run t (D.Encrypt_values (Rrbvec.of_list [ "\"hello\""; "\"world\"" ]))) + |> Rrbvec.to_list + in + let wire (iv, ciphertext) = + Transit_native.Transit.Json.to_string + (Transit_core.Json.Array [ Binary iv; Binary ciphertext ]) + in + Alcotest.(check (list string)) + "ordered authenticated plaintexts" + [ "hello"; "world" ] + (List.map (fun encrypted -> get (run t (D.Decrypt_value (wire encrypted)))) encrypted); + let iv, ciphertext = List.hd encrypted in + let corrupted = Bytes.of_string ciphertext in + Bytes.set corrupted 0 (Char.chr (Char.code (Bytes.get corrupted 0) lxor 1)); + (match run t (D.Decrypt_value (wire (iv, Bytes.to_string corrupted))) with + | Error (D.Crypto_failure _) -> () + | _ -> Alcotest.fail "Authenticated ciphertext corruption must fail"); + check_error + (D.error_message (D.Capacity_exceeded "Crypto items")) + (run t (D.Encrypt_values (Rrbvec.of_list (List.init 9 (fun _ -> "value"))))); + C.retire t; + check_error + (D.error_message D.Stale_context) + (run t (D.Decrypt_value (wire (iv, ciphertext)))) +;; + +let unlock_revocation () = + let current = ref true in + let backend = + { C.apple_backend with + unlock = + (fun ~identity:_ ~wrapped_graph_key:_ -> + current := false; + Ok (String.make 32 'k')) + } + in + let owner = C.create ~identity:(identity "a") ~limits ~encrypted:true ~backend in + let input = get (D.unlock_input ~wrapped_graph_key:"wrapped") in + check_error + (D.error_message D.Stale_context) + (C.run + owner + ~identity:(identity "a") + ~check_current:(fun () -> if !current then Ok () else Error D.Stale_context) + (D.Unlock input)); + Alcotest.(check bool) + "revoked unlock never installs key" + true + (C.status owner = D.Key_locked) +;; + +let () = + Alcotest.run + "production crypto" + [ ( "scope" + , [ Alcotest.test_case "unencrypted" `Quick unencrypted + ; Alcotest.test_case "locked and revoked" `Quick locked + ; Alcotest.test_case "owner and retirement" `Quick ownership + ; Alcotest.test_case "budgets and shape" `Quick bounds + ; Alcotest.test_case "authenticated roundtrip" `Quick encrypted_roundtrip + ; Alcotest.test_case "unlock revocation" `Quick unlock_revocation + ] ) + ] +;; diff --git a/logseq_sync/test/dune b/logseq_sync/test/dune index 97575e3e..0f1f5c7b 100644 --- a/logseq_sync/test/dune +++ b/logseq_sync/test/dune @@ -1,55 +1,57 @@ (test - (name test_sync) - (modules - test_sync - core_contract - local_restore_failure_reconciliation - pure_reducer_bad_case_support - pure_reducer_bad_case_01 - pure_reducer_bad_case_02 - pure_reducer_bad_case_03 - pure_reducer_bad_case_04 - pure_reducer_bad_case_05 - pure_reducer_bad_case_06 - pure_reducer_bad_case_07 - pure_reducer_bad_case_08 - pure_reducer_bad_case_09 - pure_reducer_bad_case_10 - runner_contract - transport_contract - sync_protocol_contract) - (libraries - alcotest - eio - eio_main - eio.unix - logseq_db_types - logseq_overlay_db - logseq_sync.effect_runner - logseq_sync.pure_reducer - melange-transit-core - melange-transit-native - mtime - ptime.clock.os - unix - uri - x509 - yojson)) + (name crypto_contract) + (modules crypto_contract) + (libraries datascript_ocaml.types alcotest logseq_sync.crypto logseq_sync_sync_dsl_impl logseq_sync_execute_impl rrbvec mirage-crypto melange-transit-core melange-transit-native)) + +(test + (name execute_runtime_contract) + (modules execute_runtime_contract) + (libraries datascript_ocaml.types logseq_sync_sync_service logseq_sync_sync_service_impl alcotest eio eio_main datascript-ocaml-native logseq_db_storage execute logseq_sync_execute_impl sync_dsl logseq_sync_sync_dsl_impl logseq_overlay_db logseq_db_types rrbvec melange-transit-core melange-transit-native unix)) + +(test + (name execute_owner_contract) + (modules execute_owner_contract) + (libraries alcotest eio eio_main sync_dsl logseq_sync_sync_dsl_impl logseq_sync_execute_owners)) + +(test + (name service_lifecycle_contract) + (modules service_lifecycle_contract) + (libraries alcotest logseq_sync_service_lifecycle)) + +(test + (name sync_dsl_contract) + (modules sync_dsl_contract sync_dsl_interpreter sync_dsl_policy_contract) + (libraries alcotest sync_dsl logseq_sync_sync_dsl_impl logseq_overlay_db logseq_db_types rrbvec)) + +(test + (name indexed_program_contract) + (modules indexed_program_contract) + (libraries alcotest logseq_sync_indexed_program)) + +(test + (name test_wire_protocol) + (modules test_wire_protocol sync_protocol_contract) + (libraries alcotest logseq_db_types logseq_sync.protocol yojson)) + +(test + (name native_framing_contract) + (modules native_framing_contract tls_peer) + (libraries alcotest eio eio_main logseq_sync.native_transport ptime.clock.os unix uri x509 yojson)) (test (name test_asset_transfer) (modules test_asset_transfer) - (libraries alcotest logseq_db_types logseq_sync.pure_reducer uri)) + (libraries alcotest logseq_db_types logseq_sync.domain logseq_sync.asset_transfer uri)) (test (name test_asset_codec) (modules test_asset_codec) - (libraries alcotest logseq_db_types logseq_sync.effect_runner melange-transit-core melange-transit-native unix)) + (libraries alcotest logseq_db_types logseq_sync.native_crypto logseq_sync.assets melange-transit-core melange-transit-native unix)) (test (name test_asset_cache) (modules test_asset_cache) - (libraries alcotest logseq_sync.effect_runner logseq_sync.pure_reducer logseq_db_types unix uri)) + (libraries alcotest logseq_sync.native_crypto logseq_sync.assets logseq_sync.domain logseq_sync.asset_transfer logseq_db_types unix uri)) (test (name test_asset_descriptor) @@ -59,9 +61,9 @@ (executable (name asset_native_interop) (modules asset_native_interop) - (libraries logseq_db_types logseq_sync.effect_runner melange-transit-core melange-transit-native yojson)) + (libraries logseq_db_types logseq_sync.native_crypto logseq_sync.assets melange-transit-core melange-transit-native yojson)) (executable (name asset_adapter_memory) (modules asset_adapter_memory) - (libraries logseq_sync.effect_runner logseq_sync.pure_reducer logseq_db_types eio eio_main uri melange-transit-core melange-transit-native)) + (libraries datascript_ocaml.types execute sync_dsl logseq_sync.crypto logseq_sync.native_crypto logseq_sync.assets logseq_db_types eio eio_main uri melange-transit-core melange-transit-native)) diff --git a/logseq_sync/test/execute_owner_contract.ml b/logseq_sync/test/execute_owner_contract.ml new file mode 100644 index 00000000..0d141473 --- /dev/null +++ b/logseq_sync/test/execute_owner_contract.ml @@ -0,0 +1,259 @@ +module D = Sync_dsl.Core +module Binding = Logseq_sync_execute_owners.Owner_binding + +let require_ok = function + | Ok value -> value + | Error error -> Alcotest.fail (D.error_message error) +;; + +let check_error expected = function + | Error actual when actual = expected -> () + | Error actual -> Alcotest.fail (D.error_message actual) + | Ok _ -> Alcotest.fail "expected rejection" +;; + +let fixture + ?(prepare = fun () -> ()) + ?(commit = fun () -> ()) + ?(open_committed = fun () -> Ok 41) + () + = + let preparations = ref 0 in + let commits = ref 0 in + let cancellations = ref 0 in + let opens = ref 0 in + let backend : (int, int, unit, int) Binding.backend = + { same_input = ( = ) + ; prepare = + (fun inspection () -> + incr preparations; + Alcotest.(check int) "original inspection" 7 inspection; + prepare (); + Ok 8) + ; commit = + (fun prepared -> + incr commits; + Alcotest.(check int) "original preparation" 8 prepared; + commit (); + Ok 9) + ; cancel = + (fun prepared -> + Alcotest.(check int) "original cancellation" 8 prepared; + incr cancellations) + ; open_committed = + (fun inspection -> + incr opens; + Alcotest.(check int) "committed inspection" 9 inspection; + open_committed ()) + } + in + Binding.absent 7 backend, preparations, commits, cancellations, opens +;; + +let current () = Ok () + +let absent_rejects_reads () = + let owner, preparations, commits, _, opens = fixture () in + (match Binding.database owner with + | Error (D.Storage_failure _) -> () + | Error error -> Alcotest.fail (D.error_message error) + | Ok _ -> Alcotest.fail "absent mirror supplied a database"); + Alcotest.(check int) "no preparation" 0 !preparations; + Alcotest.(check int) "no commit" 0 !commits; + Alcotest.(check int) "no open" 0 !opens +;; + +let activated_once () = + let owner, preparations, commits, cancellations, opens = fixture () in + Alcotest.(check int) + "opened" + 41 + (require_ok (Binding.activate owner ~check_current:current ())); + Alcotest.(check int) + "same binding" + 41 + (require_ok (Binding.activate owner ~check_current:current ())); + Alcotest.(check int) "one preparation" 1 !preparations; + Alcotest.(check int) "one commit" 1 !commits; + Alcotest.(check int) "release staging" 1 !cancellations; + Alcotest.(check int) "one open" 1 !opens +;; + +let revoked_during_prepare () = + Eio.Switch.run (fun sw -> + let prepared, resolve_prepared = Eio.Promise.create () in + let resume, resolve_resume = Eio.Promise.create () in + let valid = ref true in + let check_current () = if !valid then Ok () else Error D.Stale_context in + let owner, _, commits, cancellations, opens = + fixture + ~prepare:(fun () -> + Eio.Promise.resolve resolve_prepared (); + Eio.Promise.await resume) + () + in + let result = + Eio.Fiber.fork_promise ~sw (fun () -> Binding.activate owner ~check_current ()) + in + Eio.Promise.await prepared; + valid := false; + Eio.Promise.resolve resolve_resume (); + check_error D.Stale_context (Eio.Promise.await_exn result); + Alcotest.(check int) "no stale commit" 0 !commits; + Alcotest.(check int) "staging released" 1 !cancellations; + Alcotest.(check int) "no open" 0 !opens) +;; + +let recover_open_without_recommit () = + let failed = ref true in + let owner, preparations, commits, cancellations, opens = + fixture + ~open_committed:(fun () -> + if !failed then Error (D.Storage_failure "open interrupted") else Ok 41) + () + in + check_error + (D.Storage_failure "open interrupted") + (Binding.activate owner ~check_current:current ()); + failed := false; + Alcotest.(check int) + "recover" + 41 + (require_ok (Binding.activate owner ~check_current:current ())); + Alcotest.(check int) "no replayed preparation" 1 !preparations; + Alcotest.(check int) "no replayed commit" 1 !commits; + Alcotest.(check int) "one staging release" 1 !cancellations; + Alcotest.(check int) "retry original open" 2 !opens +;; + +let revoked_during_open_keeps_binding () = + let valid = ref true in + let owner, _, commits, _, _ = + fixture + ~open_committed:(fun () -> + valid := false; + Ok 41) + () + in + let check_current () = if !valid then Ok () else Error D.Stale_context in + check_error D.Stale_context (Binding.activate owner ~check_current ()); + Alcotest.(check int) "committed once" 1 !commits; + Alcotest.(check int) + "original binding retained" + 41 + (require_ok (Binding.database owner)) +;; + +let queued_write_rechecks () = + Eio.Switch.run (fun sw -> + let owner = Binding.active 41 in + let entered, resolve_entered = Eio.Promise.create () in + let resume, resolve_resume = Eio.Promise.create () in + let valid = ref true in + let effects = ref 0 in + let first = + Eio.Fiber.fork_promise ~sw (fun () -> + Binding.with_database owner ~check_current:current (fun _ -> + Eio.Promise.resolve resolve_entered (); + Eio.Promise.await resume; + Ok ())) + in + Eio.Promise.await entered; + let second = + Eio.Fiber.fork_promise ~sw (fun () -> + Binding.with_database + owner + ~check_current:(fun () -> if !valid then Ok () else Error D.Stale_context) + (fun _ -> + incr effects; + Ok ())) + in + Eio.Fiber.yield (); + valid := false; + Eio.Promise.resolve resolve_resume (); + require_ok (Eio.Promise.await_exn first); + check_error D.Stale_context (Eio.Promise.await_exn second); + Alcotest.(check int) "rejected before mutation" 0 !effects) +;; + +let observation_during_open ~fails () = + Eio.Switch.run (fun sw -> + let entered, resolve_entered = Eio.Promise.create () in + let resume, resolve_resume = Eio.Promise.create () in + let owner, _, commits, _, opens = + fixture + ~open_committed:(fun () -> + Eio.Promise.resolve resolve_entered (); + Eio.Promise.await resume; + if fails then Error (D.Storage_failure "synthetic open failed") else Ok 41) + () + in + let activation = + Eio.Fiber.fork_promise ~sw (fun () -> + Binding.activate owner ~check_current:current ()) + in + Eio.Promise.await entered; + let observation = + Eio.Fiber.fork_promise ~sw (fun () -> Binding.database_if_open owner) + in + Eio.Fiber.yield (); + let premature = Eio.Promise.is_resolved observation in + Eio.Promise.resolve resolve_resume (); + let activated = Eio.Promise.await_exn activation in + let observed = Eio.Promise.await_exn observation in + Alcotest.(check bool) "observation waits for activation owner" false premature; + if fails + then ( + check_error (D.Storage_failure "synthetic open failed") activated; + check_error (D.Storage_failure "Committed mirror requires open recovery") observed) + else ( + Alcotest.(check int) "activation" 41 (require_ok activated); + Alcotest.(check (option int)) + "observed active owner" + (Some 41) + (require_ok observed)); + Alcotest.(check int) "one commit" 1 !commits; + Alcotest.(check int) "one open" 1 !opens) +;; + +let differing_activation_rejected () = + let preparations = ref 0 in + let backend : (int, int, string, int) Binding.backend = + { same_input = ( = ) + ; prepare = + (fun _ _ -> + incr preparations; + Ok 8) + ; commit = (fun _ -> Ok 9) + ; cancel = (fun _ -> ()) + ; open_committed = (fun _ -> Ok 41) + } + in + let owner = Binding.absent 7 backend in + ignore (require_ok (Binding.activate owner ~check_current:current "first")); + check_error D.Stale_context (Binding.activate owner ~check_current:current "second"); + Alcotest.(check int) "original commit retained" 1 !preparations +;; + +let () = + Eio_main.run (fun environment -> + Alcotest.run + "Execute owner binding" + [ ( "binding" + , List.map + (fun (name, test) -> + Alcotest.test_case name `Quick (fun () -> + Eio.Time.with_timeout_exn (Eio.Stdenv.clock environment) 5. test)) + [ "absent reads are local", absent_rejects_reads + ; "activation installs exactly once", activated_once + ; "revocation after preparation", revoked_during_prepare + ; "open recovery preserves commit", recover_open_without_recommit + ; "revoked open retains original binding", revoked_during_open_keeps_binding + ; "sole queue checks after suspension", queued_write_rechecks + ; "observation waits for committed open", observation_during_open ~fails:false + ; ( "failed committed open remains an error" + , observation_during_open ~fails:true ) + ; "different activation proof rejects", differing_activation_rejected + ] ) + ]) +;; diff --git a/logseq_sync/test/execute_runtime_contract.ml b/logseq_sync/test/execute_runtime_contract.ml new file mode 100644 index 00000000..5651ecc3 --- /dev/null +++ b/logseq_sync/test/execute_runtime_contract.ml @@ -0,0 +1,2715 @@ +module D = Sync_dsl.Core +module X = Execute.Core + +let get = function + | Ok x -> x + | Error e -> Alcotest.fail (D.error_message e) +;; + +let check b message = Alcotest.(check bool) message true b + +let identity = + get + (D.identity + ~origin:"https://example.com" + ~account:"a" + ~graph:"g" + ~account_generation:0L + ~graph_generation:0L) +;; + +let limits_with_items maximum_items = + get + (D.limits + ~maximum_items + ~maximum_bytes:1024 + ~maximum_pending:8 + ~maximum_resources:8 + ~steps_per_turn:8 + ~maximum_tasks:8 + ~maximum_messages:8 + ~maximum_message_bytes:1024 + ~maximum_demands:8 + ~maximum_active_assets:2 + ~foreground_reserve:1 + ~foreground_demand_reserve:1 + ~foreground_descriptor_reserve_bytes:128 + ~asset_cache_budget_bytes:4096L + ~asset_maximum_file_bytes:1024 + ~asset_pending_budget_bytes:4096L + ~snapshot_maximum_artifact_bytes:4096L + ~snapshot_staging_budget_bytes:4096L + ~snapshot_scratch_budget_bytes:4096L) +;; + +let limits = limits_with_items 8 + +module Worker = struct + type scope = unit + type imported_file = string + + let bind_runtime ~identity:_ ~database_owner:_ ~limits:_ ~cleanup_seconds:_ ~network:_ = + Ok () + ;; + + let with_scope ~identity:_ ~operation:_ f = f () + let crypto (type a) () (_ : a D.crypto) : (a, D.error) result = Error D.Locked + let unavailable () = Error (D.Invalid_input "Import not used by this fixture") + let prepare_import () _ ~current:_ = unavailable () + let publish_import_preview () _ = unavailable () + let persist_import () _ ~expected:_ = unavailable () + let inspect_import () _ = unavailable () + let apply_local_import () _ = unavailable () + let put_import () _ = unavailable () + let apply_import_metadata () _ = unavailable () + let await_import_publication () _ = unavailable () + let cancel_import () _ = unavailable () + let release_import_staging () _ = unavailable () + let recover_import_inputs () ~after:_ ~maximum_items:_ ~maximum_bytes:_ = unavailable () + let confirm_import_input () _ = unavailable () + let acknowledge_import_input () ~input:_ ~prepared:_ = unavailable () + let cancel () = Ok () + let retain_imported_file () _ = unavailable () + let imported_file_path x = x + let release_imported_file _ = Ok () + let fence ~identity:_ = Ok () + let await_quiescent ~identity:_ = Ok () +end + +let owner ~sw = + let module O = D.Overlay in + let deps = + match + O.Database.dependencies + ~epoch_ms:(fun () -> 0L) + ~monotonic_ns:(fun () -> 0L) + ~limits: + { O.Types.response_budget_bytes = 1024 + ; outbox_max_records = 8 + ; outbox_max_bytes = 4096 + ; change_max_items = 8 + ; change_max_bytes = 1024 + ; dispatcher_capacity = 8 + ; wire_batch_max_bytes = 1024 + } + with + | Ok x -> x + | Error _ -> failwith "dependencies" + in + let graph_id = + match + Logseq_db_types.Graph_types.Uuid.of_string "11111111-1111-4111-8111-111111111111" + with + | Ok x -> x + | Error _ -> failwith "uuid" + in + let inspection = + match + O.Database.inspect_mirror + ~application_support_directory:"/tmp/execute-runtime-absent-fixture" + ~graph_id + with + | Ok x -> x + | Error _ -> failwith "inspection" + in + get + (X.activation_owner + ~sw + ~dependencies:deps + ~inspection + ~graph_name:"Runtime fixture" + ~snapshot_source:(fun _ -> Error (D.Invalid_input "No snapshot in fixture"))) +;; + +let capabilities + ?dispatch + ?(close_network = fun _ -> Ok ()) + ?(restore = + fun ~scope:_ _ -> + Ok { D.keys = D.Unencrypted; checkpoint = None; recovery_pending = false }) + () + : X.capabilities + = + { validate_identity = (fun _ -> Ok ()) + ; validate_limits = (fun _ -> Ok ()) + ; websocket_liveness = X.Peer_close_only + ; network_deadlines = + { setup_seconds = 1. + ; tls_handshake_seconds = 1. + ; http_response_seconds = 1. + ; download_seconds = 1. + } + ; cleanup_seconds = 1. + ; acquire = (fun _ -> Ok ()) + ; fence_network = (fun _ -> Ok ()) + ; close_network + ; authorize_network = + (fun _ ~epoch -> + if epoch = 7L then Ok () else Error (D.Authentication_failure "epoch")) + ; validate_resource = (fun _ -> Ok ()) + ; restore + ; worker_owner = (module Worker) + ; dispatch = + (match dispatch with + | Some dispatch -> dispatch + | None -> + { database = + (fun ~owner:_ ~scope:_ ~identity:_ _ -> + Error (D.Storage_failure "No mirror")) + ; transport = (fun ~scope:_ ~identity:_ _ -> Error D.Network_closed) + ; crypto = (fun ~scope:_ ~identity:_ _ -> Error D.Locked) + ; assets = (fun ~scope:_ ~identity:_ _ -> Error (D.Invalid_input "No asset")) + }) + ; observe = + (fun _ -> + Ok + { D.identity + ; keys = D.Unencrypted + ; network = D.Closed + ; checkpoint = None + ; pending_submissions = 0 + ; pending_assets = 0 + ; cleanup_pending = 0 + ; last_failure = None + }) + ; fence = (fun _ -> Ok ()) + ; retire = (fun _ -> Ok ()) + ; release_file = (fun _ _ -> Ok ()) + ; release_lease = (fun _ _ -> Ok ()) + ; release_snapshot = (fun _ _ -> Ok ()) + } +;; + +let fixture + env + ~sw + ?(limits = limits) + ?database_owner + ?dispatch + ?close_network + ?restore + () + = + let owner = + match database_owner with + | Some owner -> owner + | None -> owner ~sw + in + let capabilities = capabilities ?dispatch ?close_network ?restore () in + let t, cold = + get + (X.create + ~sw + ~clock:(Eio.Stdenv.mono_clock env) + ~database_owner:owner + ~identity + ~limits + capabilities) + in + t, cold, owner +;; + +let restore = D.perform D.Restore ~success:(fun _ -> D.return ()) ~failure:D.abort + +let ready t cold = + match X.await (get (X.start t cold restore)) with + | X.Completed (next, ()) -> next + | X.Failed (_, e) -> Alcotest.fail (D.error_message e) +;; + +let run t context task = + X.await + (get + (X.start + t + context + (D.perform (D.Run_task task) ~success:D.return ~failure:D.abort))) +;; + +let task op = D.task_perform op ~success:D.task_return ~failure:D.task_fail + +let contexts env () = + Eio.Switch.run (fun sw -> + let t, cold, _ = fixture env ~sw () in + let next = ready t cold in + check (Result.is_error (X.start t cold restore)) "spent context rejects"; + let other, _, _ = fixture env ~sw () in + check (Result.is_error (X.start other next (D.return ()))) "foreign runtime rejects"; + let version = X.version next in + (match get (X.current t) with + | X.Context (_, c) -> + check (X.equal_version version (X.version c)) "same recovery version"); + let task = + D.with_scope (fun scope -> + D.task_bind + (D.spawn scope (D.task_return 42)) + (function + | Error e -> D.task_fail e + | Ok child -> + D.task_bind (D.join child) (function + | Ok n -> D.task_return n + | Error e -> D.task_fail e))) + in + match run t next task with + | X.Completed (_, 42) -> () + | _ -> Alcotest.fail "typed child result") +;; + +let network env () = + Eio.Switch.run (fun sw -> + let entered, enter = Eio.Promise.create () + and release, resolve = Eio.Promise.create () in + let closes = ref 0 in + let t, _, _ = + fixture + env + ~sw + ~close_network:(fun _ -> + incr closes; + ignore (Eio.Promise.try_resolve enter ()); + Eio.Promise.await release; + Ok ()) + () + in + get (X.open_network t ~authorization_epoch:7L); + let receipt = get (X.close_network t) in + let receipt2 = get (X.close_network t) in + check (receipt == receipt2) "same retirement receipt"; + check + (Result.is_error (X.open_network t ~authorization_epoch:7L)) + "cannot reopen before join"; + let joined = + Eio.Fiber.fork_promise ~sw (fun () -> X.await_network_retirement receipt) + in + Eio.Promise.await entered; + Eio.Promise.resolve resolve (); + get (Eio.Promise.await_exn joined); + get (X.open_network t ~authorization_epoch:7L); + check (!closes = 1) "one physical close"; + check + (Result.is_error (X.open_network t ~authorization_epoch:8L)) + "actual epoch checked") +;; + +let stop_and_outbox env () = + Eio.Switch.run (fun sw -> + let t, cold, owner = fixture env ~sw () in + let next = ready t cold in + get (X.notify_outbox_changed t ~owner ~identity); + let waiting = + D.with_scope (fun _ -> + D.task_bind (task D.Await_outbox_changed) (fun receipt -> + D.task_bind (task (D.Acknowledge_outbox_check receipt)) (fun () -> + task D.Next_input))) + in + let running = + get + (X.start + t + next + (D.perform (D.Run_task waiting) ~success:D.return ~failure:D.abort)) + in + Eio.Fiber.yield (); + get (X.request_stop t); + match X.await running with + | X.Completed (_, D.Stop_requested) -> () + | _ -> Alcotest.fail "stop input") +;; + +let cancellation env () = + Eio.Switch.run (fun sw -> + let t, cold, _ = fixture env ~sw () in + let next = ready t cold in + let running = + get + (X.start + t + next + (D.perform (D.Run_task (task D.Next_input)) ~success:D.return ~failure:D.abort)) + in + Eio.Fiber.yield (); + get (X.cancel t running); + get (X.cancel t running); + match X.await running with + | X.Failed (X.Context (D.Draining, _), D.Cancelled) -> () + | _ -> Alcotest.fail "cancel returns actual draining context") +;; + +let observation env () = + Eio.Switch.run (fun sw -> + let t, _, _ = fixture env ~sw () in + let first = get (X.observe_runtime t) in + get (X.open_network t ~authorization_epoch:7L); + let second = get (X.await_observation t ~after:first.revision) in + check + (second.state.network = D.Open { authorization_epoch = 7L }) + "actual gate overrides adapter"; + let other, _, _ = fixture env ~sw () in + check + (Result.is_error (X.await_observation other ~after:second.revision)) + "foreign revision rejects") +;; + +let rec remove_fixture path = + if Sys.is_directory path + then ( + Array.iter (fun name -> remove_fixture (Filename.concat path name)) (Sys.readdir path); + Unix.rmdir path) + else Sys.remove path +;; + +let with_snapshot_fixture run_fixture = + let root = Filename.temp_file "execute-activation-" "" in + Sys.remove root; + Unix.mkdir root 0o700; + Fun.protect + ~finally:(fun () -> remove_fixture root) + (fun () -> + let module O = D.Overlay in + let require = function + | Ok value -> value + | Error _ -> Alcotest.fail "real fixture setup failed" + in + let module Transit = Transit_core.Json in + let module Codec = Transit_native.Transit.Json in + let module Storage_codec = Logseq_db_storage.Logseq_sqlite_codec in + let attribute : Datascript.schema_attr = + { cardinality = One + ; unique = None + ; indexed = true + ; is_component = false + ; no_history = false + ; doc = None + ; value_type = None + ; tuple_attrs = None + ; tuple_types = None + } + in + let schema = + List.map + (fun name -> name, attribute) + [ "db/ident" + ; "block/uuid" + ; "block/title" + ; "block/parent" + ; "block/page" + ; "block/order" + ; "kv/value" + ] + in + let fact e a v : Datascript.datom = { e; a; v; tx = 536870913; added = true } in + let kv e name value = + [ fact e "db/ident" (Keyword name); fact e "kv/value" value ] + in + let facts = + List.concat + [ kv 1 "logseq.kv/graph-remote?" (Bool true) + ; kv 2 "logseq.kv/graph-uuid" (Uuid "11111111-1111-4111-8111-111111111111") + ; kv 3 "logseq.kv/db-type" (String "db") + ; kv + 4 + "logseq.kv/schema-version" + (Map [ Keyword "major", Int64 65L; Keyword "minor", Int64 33L ]) + ] + in + let stored = ref [] in + let storage : Datascript.storage = + { storage_store = + (fun entries -> + List.iter + (fun (key, value) -> + stored := (key, value) :: List.remove_assoc key !stored) + entries) + ; storage_restore = (fun key -> List.assoc_opt key !stored) + ; storage_list_addresses = (fun () -> List.map fst !stored) + ; storage_delete = + (fun keys -> + stored := List.filter (fun (key, _) -> not (List.mem key keys)) !stored) + } + in + ignore (Datascript.store ~storage (Datascript.init_db ~schema facts)); + let entries = + require (Storage_codec.encode_physical_batch !stored) + |> List.sort (fun (left : Storage_codec.physical_entry) right -> + Int.compare (int_of_string left.address) (int_of_string right.address)) + in + let rows = + List.map + (fun (entry : Storage_codec.physical_entry) -> + Transit.Array + [ Transit.Int (int_of_string entry.address) + ; Transit.String entry.content + ; (match entry.addresses with + | [] -> Transit.Null + | addresses -> + Transit.String + (Codec.to_string + ~mode:Codec.Verbose + (Transit.Array + (List.map + (fun address -> Transit.Int (int_of_string address)) + addresses)))) + ]) + entries + in + let payload = Codec.to_string ~mode:Codec.Verbose (Transit.Array rows) in + let path = Filename.concat root "snapshot.transit" in + let output = open_out_bin path in + Fun.protect + ~finally:(fun () -> close_out_noerr output) + (fun () -> + output_binary_int output (String.length payload); + output_string output payload); + let dependencies = + require + (O.Database.dependencies + ~epoch_ms:(fun () -> 0L) + ~monotonic_ns:(fun () -> 0L) + ~limits: + { O.Types.response_budget_bytes = 1048576 + ; outbox_max_records = 8 + ; outbox_max_bytes = 1048576 + ; change_max_items = 8 + ; change_max_bytes = 1048576 + ; dispatcher_capacity = 8 + ; wire_batch_max_bytes = 1048576 + }) + in + let graph_id = + require + (Logseq_db_types.Graph_types.Uuid.of_string + "11111111-1111-4111-8111-111111111111") + in + let inspect support = + require + (O.Database.inspect_mirror ~application_support_directory:support ~graph_id) + in + let through = require (Logseq_db_types.Server_cursor.of_int64 0L) in + let baseline = Filename.concat root "baseline" in + Unix.mkdir baseline 0o700; + let activation = function + | Error (O.Types.Snapshot_parse_error message) -> Alcotest.fail message + | result -> + get (Result.map_error (fun error -> D.Activation_failure error) result) + in + let prepared = + activation + (O.Database.prepare_snapshot_activation + dependencies + (inspect baseline) + ~path + ~applied_server_cursor:through + ~expected_checksum:None + ~expected_rows:(List.length rows)) + in + let committed = + Fun.protect + ~finally:(fun () -> O.Database.cancel_snapshot_activation prepared) + (fun () -> + check + (require (O.Database.next_snapshot_unprotection_batch prepared) = None) + "unencrypted fixture"; + require (O.Database.commit_snapshot_activation prepared)) + in + let checksum = + match O.Database.mirror_presence committed with + | O.Types.Available { checksum = Some checksum; _ } -> + O.Types.Checksum.to_string checksum + | _ -> Alcotest.fail "baseline checksum missing" + in + let target = Filename.concat root "target" in + Unix.mkdir target 0o700; + run_fixture + ~dependencies + ~inspection:(inspect target) + ~path + ~rows + ~checksum + ~inspect_target:(fun () -> inspect target)) +;; + +let real_activation env () = + with_snapshot_fixture + (fun ~dependencies ~inspection ~path ~rows ~checksum ~inspect_target -> + let module O = D.Overlay in + let require = Result.get_ok in + let through = require (Logseq_db_types.Server_cursor.of_int64 0L) in + Eio.Switch.run (fun sw -> + let database_owner = + get + (X.activation_owner + ~sw + ~dependencies + ~inspection + ~graph_name:"Real runtime activation" + ~snapshot_source:(fun _ -> + Ok { X.path; expected_rows = List.length rows })) + in + check + (Result.is_error (X.database database_owner)) + "mirror absent before activation"; + let snapshot = + get + (D.snapshot + ~resource:(get (D.resource ~owner:identity ~token:"real-snapshot")) + ~through + ~checksum + ~encrypted:false) + in + let dispatch : X.dispatch = + { database = + (fun ~owner:_ ~scope:_ ~identity:_ _ -> + Alcotest.fail "database must use real owner") + ; transport = + (fun (type a) + ~scope + ~identity:_ + (operation : a D.transport) + : (a, D.error) result -> + match operation with + | D.Fetch_snapshot _ -> + Result.map + (fun () -> snapshot) + (X.handoff scope (Rrbvec.of_list [ D.Snapshot_reference snapshot ])) + | _ -> Error D.Network_closed) + ; crypto = (fun ~scope:_ ~identity:_ _ -> Error D.Locked) + ; assets = (fun ~scope:_ ~identity:_ _ -> Error (D.Invalid_input "No asset")) + } + in + let t, cold, _ = + fixture env ~sw ~limits:(limits_with_items 1) ~database_owner ~dispatch () + in + let next = ready t cold in + get (X.open_network t ~authorization_epoch:7L); + let program = + D.perform + (D.Transport (D.Fetch_snapshot through)) + ~failure:D.abort + ~success:(fun snapshot -> + D.perform + (D.Database (D.Activate_snapshot snapshot)) + ~failure:D.abort + ~success:D.return) + in + let next = + match X.await (get (X.start t next program)) with + | X.Completed (next, D.Activated _) -> next + | X.Completed (_, D.Activation_deferred message) -> Alcotest.fail message + | X.Failed (_, error) -> Alcotest.fail (D.error_message error) + in + let original = get (X.database database_owner) in + let next, view = + match + X.await + (get + (X.start + t + next + (D.perform (D.Database D.Inspect) ~failure:D.abort ~success:D.return))) + with + | X.Completed (next, view) -> next, view + | X.Failed (_, error) -> Alcotest.fail (D.error_message error) + in + check + (original == get (X.database database_owner)) + "same activated owner serves reads"; + ignore view; + for index = 1 to 2 do + let uuid suffix = + require + (Logseq_db_types.Graph_types.Uuid.of_string + (Printf.sprintf "22222222-2222-4222-8222-%012d" suffix)) + in + let page = uuid (index + 10) in + let snapshot = require (O.Database.current_snapshot original) in + let expected = + Fun.protect + ~finally:(fun () -> O.Database.release_snapshot snapshot) + (fun () -> + match require (O.Database.get_pages snapshot [ page ]) with + | [ O.Types.Missing_page { revision; _ } ] -> + require + (O.Database.write_precondition + ~blocks:[] + ~pages:[ page, revision ] + ~scopes:[]) + | _ -> Alcotest.fail "new journal page must be absent") + in + ignore + (require + (O.Database.commit_local + original + ~expected + (O.Types.Create_journal_page + { mutation_id = uuid index + ; page + ; title = Printf.sprintf "2026-01-%02d" index + ; journal_day = 20260100 + index + }))) + done; + (match run t next (task (D.Task_database D.Inspect)) with + | X.Failed (_, D.Invalid_input "Sync view items") -> () + | X.Failed (_, error) -> Alcotest.fail (D.error_message error) + | X.Completed _ -> Alcotest.fail "oversized durable view must fail permanently"); + match O.Database.mirror_presence (inspect_target ()) with + | O.Types.Available { checksum = Some actual; _ } -> + Alcotest.(check string) + "durable checksum" + checksum + (O.Types.Checksum.to_string actual) + | _ -> Alcotest.fail "real mirror was not committed")) +;; + +let nested_unlock env () = + Eio.Switch.run (fun sw -> + let t, cold, _ = fixture env ~sw () in + let next = ready t cold in + let consume = + D.with_scope (fun _ -> + D.task_bind (task D.Next_input) (function + | D.Unlock_requested receipt -> + D.task_perform + (D.Consume_unlock receipt) + ~success:(fun _ -> D.task_return ()) + ~failure:(function + | D.Locked -> D.task_return () + | e -> D.task_fail e) + | _ -> D.task_fail (D.Invalid_input "Expected unlock"))) + in + let running = + get + (X.start + t + next + (D.perform (D.Run_task consume) ~success:D.return ~failure:D.abort)) + in + Eio.Fiber.yield (); + ignore + (get + (X.submit_unlock + t + ~identity + ~valid_for:10. + (get (D.unlock_input ~wrapped_graph_key:"wrapped")))); + match X.await running with + | X.Completed (_, ()) -> () + | X.Failed (_, e) -> Alcotest.fail (D.error_message e)) +;; + +let cancelled_subtree env () = + Eio.Switch.run (fun sw -> + let entered = ref false + and joined = ref false in + let never, _ = Eio.Promise.create () in + let dispatch : X.dispatch = + { database = (fun ~owner:_ ~scope:_ ~identity:_ _ -> Error D.Stale_context) + ; transport = (fun ~scope:_ ~identity:_ _ -> Error D.Network_closed) + ; crypto = (fun ~scope:_ ~identity:_ _ -> Error D.Locked) + ; assets = + (fun (type a) + ~scope:_ + ~identity:_ + (operation : a D.assets) + : (a, D.error) result -> + match operation with + | D.Prune_cache -> + entered := true; + Fun.protect + ~finally:(fun () -> joined := true) + (fun () -> + Eio.Promise.await never; + Error D.Cancelled) + | _ -> Error D.Stale_context) + } + in + let t, cold, _ = fixture env ~sw ~dispatch () in + let parent = + D.with_scope (fun scope -> + D.task_bind + (D.spawn scope (task (D.Task_assets D.Prune_cache))) + (function + | Error e -> D.task_fail e + | Ok _ -> task (D.Sleep 3600.))) + in + let rec wait_entered () = + if !entered then D.task_return () else D.task_bind (task D.Yield) wait_entered + in + let body = + D.with_scope (fun scope -> + D.task_bind (D.spawn scope parent) (function + | Error e -> D.task_fail e + | Ok child -> + D.task_bind (wait_entered ()) (fun () -> + D.task_bind (D.cancel_child child) (fun () -> + D.task_bind (D.join child) (fun _ -> D.task_return !joined))))) + in + match run t (ready t cold) body with + | X.Completed (_, joined) -> + check joined "grandchild physically joined before child result" + | X.Failed (_, e) -> Alcotest.fail (D.error_message e)) +;; + +let ready_child_not_starved env () = + Eio.Switch.run (fun sw -> + let t, cold, _ = fixture env ~sw () in + let rec select child turns = + if turns = 4 + then D.task_return false + else ( + ignore + (get + (X.submit_unlock + t + ~identity + ~valid_for:10. + (get (D.unlock_input ~wrapped_graph_key:"ready")))); + D.task_bind + (task (D.Await_input_or_child (Rrbvec.of_list [ child ]))) + (function + | D.Child_ready _ -> D.task_bind (D.join child) (fun _ -> D.task_return true) + | D.Input _ -> select child (turns + 1))) + in + let body = + D.with_scope (fun scope -> + D.task_bind + (D.spawn scope (D.task_return ())) + (function + | Error e -> D.task_fail e + | Ok child -> D.task_bind (task D.Yield) (fun () -> select child 0))) + in + match run t (ready t cold) body with + | X.Completed (_, handled) -> + check handled "ready child handled within four input turns" + | X.Failed (_, e) -> Alcotest.fail (D.error_message e)) +;; + +let demand_change = + let module A = Logseq_db_types.Asset_descriptor in + let uuid = + get + (Result.map_error + (fun message -> D.Invalid_input message) + (Logseq_db_types.Graph_types.Uuid.of_string + "11111111-1111-4111-8111-111111111111")) + in + let asset = + get + (Result.map_error + (fun message -> D.Invalid_input message) + (A.create + ~uuid + ~source:(A.External "https://example.com/asset") + ~current_checksum:None + ~size:None + ~dimensions:None)) + in + D.Demand { id = get (D.demand_id "consumer"); asset; priority = D.Foreground } +;; + +let demand_loop = + D.with_scope (fun _ -> + let rec loop () = + D.task_bind (task D.Next_input) (function + | D.Demand_changed (receipt, _) -> + D.task_bind (task (D.Publish_demand (receipt, D.Demand_queued))) loop + | D.Stop_requested -> D.task_return () + | _ -> loop ()) + in + loop ()) +;; + +let held_delivery_retirement env () = + Eio.Switch.run (fun sw -> + let t, cold, _ = fixture env ~sw () in + let next = ready t cold in + let running = + get + (X.start + t + next + (D.perform (D.Run_task demand_loop) ~success:D.return ~failure:D.abort)) + in + Eio.Fiber.yield (); + ignore (get (X.update_demand t demand_change)); + let delivery = get (X.receive_demand t) in + get (X.request_stop t); + let next = + match X.await running with + | X.Completed (next, ()) -> next + | X.Failed (_, e) -> Alcotest.fail (D.error_message e) + in + let closing = + get + (X.start + t + next + (D.perform (D.begin_drain D.Ready_context) ~failure:D.abort ~success:(fun () -> + D.perform D.finish_retirement ~failure:D.abort ~success:D.return))) + in + let finished = ref false in + let result = + Eio.Fiber.fork_promise ~sw (fun () -> + let result = X.await closing in + finished := true; + result) + in + for _ = 1 to 4 do + Eio.Fiber.yield () + done; + let waited = not !finished in + get (X.release_demand t delivery.delivery); + get (X.release_demand t delivery.delivery); + check waited "retirement waits for consumer release"; + match Eio.Promise.await_exn result with + | X.Completed (_, ()) -> () + | X.Failed (_, e) -> Alcotest.fail (D.error_message e)) +;; + +let callback_failed_join_retains_owner env () = + Eio.Switch.run (fun sw -> + let weak = Weak.create 1 in + let dispatch : X.dispatch = + { database = (fun ~owner:_ ~scope:_ ~identity:_ _ -> Error D.Stale_context) + ; transport = (fun ~scope:_ ~identity:_ _ -> Error D.Network_closed) + ; crypto = (fun ~scope:_ ~identity:_ _ -> Error D.Locked) + ; assets = + (fun ~scope ~identity:_ _ -> + let owner = ref 17 in + Weak.set weak 0 (Some owner); + X.Callback_bridge.run + ~scope + ~resources:(fun _ -> Rrbvec.of_list []) + { start = (fun ~deliver -> deliver (Error D.Locked)) + ; stop = (fun () -> Error (D.Storage_failure "stop failed")) + ; await_stopped = + (fun ~deadline:_ -> + ignore (Sys.opaque_identity !owner); + Error (D.Storage_failure "join failed")) + }) + } + in + let t, cold, _ = fixture env ~sw ~dispatch () in + (match run t (ready t cold) (task (D.Task_assets D.Prune_cache)) with + | X.Failed (_, D.Cleanup_failure _) -> () + | _ -> Alcotest.fail "cleanup failure must force draining"); + Gc.full_major (); + check (Weak.check weak 0) "failed producer owner stays reachable until physical join") +;; + +let released_metadata_bounded env () = + Eio.Switch.run (fun sw -> + let scope = ref None in + let t, cold, _ = + fixture + env + ~sw + ~restore:(fun ~scope:current _ -> + scope := Some current; + Ok { D.keys = D.Unencrypted; checkpoint = None; recovery_pending = false }) + () + in + let context = ref (ready t cold) in + let through = + get + (Result.map_error + (fun _ -> D.Invalid_input "cursor") + (Logseq_db_types.Server_cursor.of_int64 0L)) + in + let batch offset = + for index = 1 to 1000 do + let snapshot = + get + (D.snapshot + ~resource: + (get + (D.resource ~owner:identity ~token:(string_of_int (offset + index)))) + ~through + ~checksum:"snapshot" + ~encrypted:false) + in + get + (X.handoff + (Option.get !scope) + (Rrbvec.of_list [ D.Snapshot_reference snapshot ])); + let program = + D.perform (D.release_snapshot snapshot) ~failure:D.abort ~success:D.return + in + match X.await (get (X.start t !context program)) with + | X.Completed (next, ()) -> context := next + | X.Failed (_, error) -> Alcotest.fail (D.error_message error) + done + in + batch 0; + Gc.full_major (); + let before = (Gc.stat ()).live_words in + batch 1000; + Gc.full_major (); + let growth = (Gc.stat ()).live_words - before in + check + (growth < 5000) + (Printf.sprintf "released resource metadata bounded: grew %d words" growth)) +;; + +let callback_capacity env () = + Eio.Switch.run (fun sw -> + let starts = ref 0 + and stops = ref 0 + and joins = ref 0 + and rejected = ref false in + let dispatch : X.dispatch = + { database = (fun ~owner:_ ~scope:_ ~identity:_ _ -> Error D.Locked) + ; transport = (fun ~scope:_ ~identity:_ _ -> Error D.Network_closed) + ; crypto = (fun ~scope:_ ~identity:_ _ -> Error D.Locked) + ; assets = + (fun (type a) + ~scope + ~identity:_ + (operation : a D.assets) + : (a, D.error) result -> + match operation with + | D.Prune_cache -> + let rec reserve depth = + X.Callback_bridge.run + ~scope + ~resources:(fun _ -> Rrbvec.empty) + { start = + (fun ~deliver -> + incr starts; + let result = reserve (depth + 1) in + (if depth = limits.maximum_resources + then + rejected + := match result with + | Error (D.Capacity_exceeded _) -> true + | _ -> false); + deliver (Error D.Locked)) + ; stop = + (fun () -> + incr stops; + Ok ()) + ; await_stopped = + (fun ~deadline:_ -> + incr joins; + Ok ()) + } + in + reserve 1 + | _ -> Error D.Stale_context) + } + in + let t, cold, _ = fixture env ~sw ~dispatch () in + ignore (run t (ready t cold) (task (D.Task_assets D.Prune_cache))); + check !rejected "producer capacity rejects before native start"; + Alcotest.(check int) "bounded starts" limits.maximum_resources !starts; + Alcotest.(check int) "all admitted producers stopped" !starts !stops; + Alcotest.(check int) "all admitted producers joined" !starts !joins) +;; + +let callback_terminal_capacity_rejection env () = + let releases = ref 0 + and joined = ref false + and rejected = ref false in + Eio.Switch.run (fun sw -> + let cursor = + get + (Result.map_error + (fun _ -> D.Invalid_input "cursor") + (Logseq_db_types.Server_cursor.of_int64 0L)) + in + let reference index = + D.Snapshot_reference + (get + (D.snapshot + ~resource:(get (D.resource ~owner:identity ~token:(string_of_int index))) + ~through:cursor + ~checksum:"capacity" + ~encrypted:false)) + in + let dispatch : X.dispatch = + { database = (fun ~owner:_ ~scope:_ ~identity:_ _ -> Error D.Locked) + ; transport = (fun ~scope:_ ~identity:_ _ -> Error D.Network_closed) + ; crypto = (fun ~scope:_ ~identity:_ _ -> Error D.Locked) + ; assets = + (fun (type a) + ~scope + ~identity:_ + (operation : a D.assets) + : (a, D.error) result -> + match operation with + | D.Prune_cache -> + X.Callback_bridge.run + ~scope + ~resources:(fun _ -> Rrbvec.of_list [ reference 8; reference 9 ]) + { start = + (fun ~deliver -> + (rejected + := match + deliver + (Ok + { D.resident_bytes = 0L + ; pending_bytes = 0L + ; retained_bytes = 0L + ; cleanup_pending = 0 + ; over_budget = false + }) + with + | Error (D.Capacity_exceeded _) -> true + | _ -> false); + Ok ()) + ; stop = (fun () -> Ok ()) + ; await_stopped = + (fun ~deadline:_ -> + joined := true; + Ok ()) + } + | _ -> Error D.Stale_context) + } + in + let caps = + capabilities + ~dispatch + ~restore:(fun ~scope _ -> + get (X.handoff scope (Rrbvec.of_list (List.init 7 reference))); + Ok { D.keys = D.Unencrypted; checkpoint = None; recovery_pending = false }) + () + in + let caps = + { caps with + release_snapshot = + (fun _ _ -> + incr releases; + Ok ()) + } + in + let t, cold = + get + (X.create + ~sw + ~clock:(Eio.Stdenv.mono_clock env) + ~database_owner:(owner ~sw) + ~identity + ~limits + caps) + in + match run t (ready t cold) (task (D.Task_assets D.Prune_cache)) with + | X.Failed (_, D.Capacity_exceeded _) -> () + | _ -> Alcotest.fail "resource rejection must resolve terminal without hanging"); + check !rejected "callback receives rejected handoff ACK"; + check !joined "rejected terminal still physically joins producer"; + Alcotest.(check int) "no partially accepted resources" 7 !releases +;; + +let terminal_cancel_rejected env () = + Eio.Switch.run (fun sw -> + let t, cold, _ = fixture env ~sw () in + let running = get (X.start t cold restore) in + ignore (X.await running); + check (X.cancel t running = Error D.Stale_context) "terminal cancellation rejects"; + match get (X.current t) with + | X.Context (D.Ready, next) -> + (match X.await (get (X.start t next (D.return ()))) with + | X.Completed _ -> () + | X.Failed (_, error) -> Alcotest.fail (D.error_message error)) + | _ -> Alcotest.fail "terminal rejection must preserve ready context") +;; + +let delivery_metadata_bounded env () = + Eio.Switch.run (fun sw -> + let t, cold, _ = fixture env ~sw () in + let running = + get + (X.start + t + (ready t cold) + (D.perform (D.Run_task demand_loop) ~success:D.return ~failure:D.abort)) + in + Eio.Fiber.yield (); + let batch () = + for _ = 1 to 1000 do + ignore (get (X.update_demand t demand_change)); + let output = get (X.receive_demand t) in + get (X.release_demand t output.delivery); + get (X.release_demand t output.delivery); + check + (get (X.demand_status t output.delivery) = X.Revoked) + "released delivery stays revoked" + done + in + batch (); + Gc.full_major (); + let before = (Gc.stat ()).live_words in + batch (); + Gc.full_major (); + let growth = (Gc.stat ()).live_words - before in + check + (growth < 5000) + (Printf.sprintf "released delivery metadata bounded: grew %d words" growth); + get (X.request_stop t); + match X.await running with + | X.Completed _ -> () + | X.Failed (_, error) -> Alcotest.fail (D.error_message error)) +;; + +module Service = Logseq_sync_sync_service.Core + +let service_config = + let retry = + get + (D.retry_policy + ~initial_seconds:0.01 + ~maximum_seconds:0.02 + ~multiplier:2. + ~maximum_attempts:0) + in + { Service.identity + ; limits + ; network = D.Closed + ; policy = + get (D.policy_config ~sync_retry:retry ~asset_retry:retry ~no_progress_seconds:0.01) + } +;; + +let service_exhausted_network_recovery ?terminal maximum_attempts env () = + with_snapshot_fixture + (fun ~dependencies ~inspection ~path ~rows ~checksum ~inspect_target:_ -> + Eio.Time.with_timeout_exn (Eio.Stdenv.clock env) 5. (fun () -> + Eio.Switch.run (fun sw -> + let module Types = Logseq_overlay_db.Types in + let cursor value = Result.get_ok (Types.Server_cursor.of_int64 value) in + let target = cursor 1L in + let database_owner = + get + (X.activation_owner + ~sw + ~dependencies + ~inspection + ~graph_name:"Recovery fixture" + ~snapshot_source:(fun _ -> + Ok { X.path; expected_rows = List.length rows })) + in + let checkpoint () = + match X.database database_owner with + | Error _ -> None + | Ok database -> + let view = Result.get_ok (D.Overlay.Database.inspect_sync database) in + Some (Types.sync_view_checkpoint view) + in + let snapshot = + get + (D.snapshot + ~resource:(get (D.resource ~owner:identity ~token:"recovery-snapshot")) + ~through:target + ~checksum + ~encrypted:false) + in + let attempts = ref 0 + and repaired = ref false in + let authorized = ref [] + and events = ref [] + and latest = ref None in + let closed, release_close = Eio.Promise.create () in + let hold_close = ref true in + let expected_checksum = Result.get_ok (Types.Checksum.of_string checksum) in + let never, _ = Eio.Promise.create () in + let rec transport + : type value. X.effect_scope -> value D.transport -> (value, D.error) result + = + fun scope operation -> + match operation with + | D.Connect -> + incr attempts; + if not !repaired + then + Error + (Option.value + terminal + ~default:(D.Network_failure "Synthetic offline")) + else ( + let connection = + get (D.connection ~owner:identity ~token:"recovered" ~generation:1L) + in + Result.map + (fun () -> connection) + (X.handoff + scope + (Rrbvec.singleton (D.Connection_reference connection)))) + | D.On_connection (_, operation) -> transport scope operation + | D.Head -> + Ok + { D.through = target + ; snapshot_available = true + ; expected_checksum = Some expected_checksum + } + | D.Fetch_snapshot _ -> + Result.map + (fun () -> snapshot) + (X.handoff scope (Rrbvec.singleton (D.Snapshot_reference snapshot))) + | D.Receive _ -> + let () = Eio.Promise.await never in + Error D.Cancelled + | D.Retire_connection _ -> Ok () + | _ -> Error (D.Protocol_failure "Unexpected recovery transport") + in + let dispatch : X.dispatch = + { (capabilities ()).dispatch with + transport = (fun ~scope ~identity:_ operation -> transport scope operation) + ; database = + (fun ~owner:_ ~scope:_ ~identity:_ _ -> + Alcotest.fail "recovery must use the physical database owner") + } + in + let caps = + capabilities + ~dispatch + ~restore:(fun ~scope:_ _ -> + Ok + { D.keys = D.Unencrypted; checkpoint = None; recovery_pending = false }) + () + in + let caps = + { caps with + authorize_network = + (fun _ ~epoch -> + (match !authorized with + | previous :: _ when previous = epoch -> () + | previous :: _ when previous > epoch -> + Alcotest.fail "stale authorization epoch" + | _ -> + authorized := epoch :: !authorized; + events := "open" :: !events); + Ok ()) + ; fence_network = + (fun _ -> + events := "fence" :: !events; + Ok ()) + ; close_network = + (fun _ -> + if !hold_close then Eio.Promise.await closed; + events := "joined" :: !events; + Ok ()) + ; observe = + (fun _ -> + Ok + { D.identity + ; keys = D.Unencrypted + ; network = D.Closed + ; checkpoint = checkpoint () + ; pending_submissions = 0 + ; pending_assets = 0 + ; cleanup_pending = 0 + ; last_failure = None + }) + } + in + let retry = + get + (D.retry_policy + ~initial_seconds:0.001 + ~maximum_seconds:0.001 + ~multiplier:1. + ~maximum_attempts) + in + let config = + { service_config with + policy = + get + (D.policy_config + ~sync_retry:retry + ~asset_retry:retry + ~no_progress_seconds:0.01) + } + in + let service = + get + (Service.start + ~sw + ~config + ~dependencies: + { clock = Eio.Stdenv.mono_clock env + ; database_owner + ; capabilities = caps + ; bind_ingress = (fun _ -> ()) + } + ~on_change:(fun status -> latest := Some status.Service.observation)) + in + Fun.protect + ~finally:(fun () -> + hold_close := false; + ignore (Eio.Promise.try_resolve release_close ()); + Service.close service) + (fun () -> + let wait predicate = + let remaining = ref 1000 in + while (not (predicate ())) && !remaining > 0 do + decr remaining; + Eio.Time.sleep (Eio.Stdenv.clock env) 0.001 + done; + check + (predicate ()) + (Printf.sprintf + "service settles after %d attempts (%s)" + !attempts + (match !latest with + | Some { X.status = D.Service_blocked error; _ } -> + D.error_message error + | _ -> "pending")) + in + get (Service.set_network service (D.Open { authorization_epoch = 7L })); + wait (fun () -> + Option.exists + (fun observation -> + match observation.X.status with + | D.Service_blocked error -> + error + = Option.value + terminal + ~default:(D.Network_failure "Synthetic offline") + | _ -> false) + !latest); + Alcotest.(check int) + "maximum attempts exhausted exactly" + (maximum_attempts + 1) + !attempts; + Eio.Time.sleep (Eio.Stdenv.clock env) 0.02; + Alcotest.(check int) + "no remaining automatic retry" + (maximum_attempts + 1) + !attempts; + check (checkpoint () = None) "offline checkpoint does not advance"; + repaired := true; + get (Service.set_network service D.Closed); + let reopening = + Eio.Fiber.fork_promise ~sw (fun () -> + Service.set_network service (D.Open { authorization_epoch = 8L })) + in + for _ = 1 to 32 do + Eio.Fiber.yield () + done; + (match terminal with + | None -> () + | Some cause -> + wait (fun () -> + Option.exists + (fun observation -> + observation.X.status = D.Service_waiting_network + && observation.state.last_failure = Some cause) + !latest)); + let waited_for_join = not (Eio.Promise.is_resolved reopening) in + let fenced_epochs = !authorized in + hold_close := false; + Eio.Promise.resolve release_close (); + check waited_for_join "fresh Open waits for physical network join"; + Alcotest.(check (list int64)) + "old epoch remains fenced during join" + [ 7L ] + fenced_epochs; + get (Eio.Promise.await_exn reopening); + (match terminal with + | Some cause -> + wait (fun () -> + Option.exists + (fun observation -> + observation.X.status = D.Service_blocked cause + && observation.state.last_failure = Some cause) + !latest); + Eio.Time.sleep (Eio.Stdenv.clock env) 0.02; + Alcotest.(check int) "terminal recovery never reconnects" 1 !attempts; + check + (checkpoint () = None) + "terminal recovery never advances checkpoint" + | None -> + wait (fun () -> + Option.exists + (fun observation -> observation.X.status = D.Service_online) + !latest); + check + (checkpoint () = Some target) + "recovery commits actual authoritative checkpoint"; + check + (Option.exists + (fun observation -> observation.X.state.checkpoint = Some target) + !latest) + "Current observation carries actual recovered checkpoint"; + Alcotest.(check int) + "explicit recovery resumes one fresh connection" + (maximum_attempts + 2) + !attempts); + check + (List.rev !events = [ "open"; "fence"; "joined"; "open" ]) + "physical fence and join precede fresh authorization")))) +;; + +let service_receipts_and_consumer ?(withdraw = false) env () = + Eio.Switch.run (fun sw -> + let unlocked = ref 0 + and released = ref [] in + let asset = + match demand_change with + | D.Demand demand -> demand.asset + | _ -> assert false + in + let file = + get (D.asset_file (get (D.resource ~owner:identity ~token:"service-file")) asset) + in + let lease = + get (D.asset_lease (get (D.resource ~owner:identity ~token:"service-lease")) file) + in + let dispatch : X.dispatch = + { database = + (fun ~owner:_ ~scope:_ ~identity:_ _ -> Error (D.Storage_failure "No mirror")) + ; transport = (fun ~scope:_ ~identity:_ _ -> Error D.Network_closed) + ; crypto = + (fun (type a) + ~scope:_ + ~identity:_ + (operation : a D.crypto) + : (a, D.error) result -> + match operation with + | D.Unlock _ -> + incr unlocked; + Ok D.Unencrypted + | _ -> Error D.Locked) + ; assets = + (fun (type a) + ~scope + ~identity:_ + (operation : a D.assets) + : (a, D.error) result -> + match operation with + | D.Lookup _ -> + Result.map + (fun () -> Some file) + (X.handoff scope (Rrbvec.of_list [ D.File_reference file ])) + | D.Retain _ -> + Result.map + (fun () -> lease) + (X.handoff scope (Rrbvec.of_list [ D.Lease_reference lease ])) + | _ -> Error D.Stale_context) + } + in + let caps = + capabilities + ~dispatch + ~restore:(fun ~scope:_ _ -> + Ok { D.keys = D.Key_locked; checkpoint = None; recovery_pending = false }) + () + in + let caps = + { caps with + release_file = + (fun _ _ -> + released := !released @ [ "file" ]; + Ok ()) + ; release_lease = + (fun _ _ -> + released := !released @ [ "lease" ]; + Ok ()) + } + in + let ingress, bind = Eio.Promise.create () in + let statuses = ref [] in + let service = + get + (Service.start + ~sw + ~config:service_config + ~dependencies: + { clock = Eio.Stdenv.mono_clock env + ; database_owner = owner ~sw + ; capabilities = caps + ; bind_ingress = + (fun value -> + check + (Result.is_error + (value.submit_unlock + ~identity + ~valid_for:10. + (get (D.unlock_input ~wrapped_graph_key:"too-early")))) + "binder precedes policy input scope"; + Eio.Promise.resolve bind value) + } + ~on_change:(fun status -> statuses := status.lifecycle :: !statuses)) + in + let ingress = Eio.Promise.await ingress in + for _ = 1 to 12 do + Eio.Fiber.yield () + done; + ignore + (get + (ingress.submit_unlock + ~identity + ~valid_for:10. + (get (D.unlock_input ~wrapped_graph_key:"wrapped")))); + ignore (get (ingress.update_demand demand_change)); + let rec available () = + let output = get (ingress.receive_demand ()) in + match output.availability with + | D.Demand_available _ -> output + | _ -> + get (ingress.release_demand output.delivery); + available () + in + let output = Eio.Time.with_timeout_exn (Eio.Stdenv.clock env) 2. available in + check (!unlocked = 1) "actual policy consumed exact unlock receipt once"; + check + (get (ingress.demand_status output.delivery) = X.Current) + "completed delivery remains current until revocation"; + if withdraw + then ( + let demand_id = + match demand_change with + | D.Demand demand -> demand.id + | _ -> assert false + in + ignore (get (ingress.update_demand (D.Withdraw demand_id))); + check + (get (ingress.demand_status output.delivery) = X.Revoked) + "withdrawal revokes completed delivery"); + let closing = Eio.Fiber.fork_promise ~sw (fun () -> Service.close service) in + for _ = 1 to 12 do + Eio.Fiber.yield () + done; + check (not (Eio.Promise.is_resolved closing)) "service close waits for consumer lease"; + check (!released = []) "revocation preserves consumer bytes"; + get (ingress.release_demand output.delivery); + get (ingress.release_demand output.delivery); + Eio.Promise.await_exn closing; + Alcotest.(check (list string)) + "physical lease before file once" + [ "lease"; "file" ] + !released; + check (List.mem Service.Closed !statuses) "closed callback follows retirement") +;; + +let service_callback_failure env () = + let retired = ref false in + let callback_error = Failure "service callback regression" in + let failed = + try + Eio.Switch.run (fun sw -> + let caps = capabilities () in + let caps = + { caps with + retire = + (fun _ -> + retired := true; + Ok ()) + } + in + ignore + (get + (Service.start + ~sw + ~config:service_config + ~dependencies: + { clock = Eio.Stdenv.mono_clock env + ; database_owner = owner ~sw + ; capabilities = caps + ; bind_ingress = (fun _ -> ()) + } + ~on_change:(fun _ -> raise callback_error))); + let never, _ = Eio.Promise.create () in + (Eio.Promise.await never : unit)); + false + with + | _ -> true + in + check failed "callback failure reaches service switch"; + check !retired "callback failure still retires original owners" +;; + +let ( let@ ) = D.task_bind +let ( let* ) = Result.bind + +let notification_capacity env () = + Eio.Switch.run (fun sw -> + let t, cold, _ = fixture env ~sw () in + let notification = D.Policy_failure (D.Network_failure "caller-owned") in + let body = + D.with_scope (fun scope -> + let@ first = task (D.Notifications scope) in + let@ second = task (D.Notifications scope) in + let rec fill count = + if count = limits.maximum_messages + then D.task_return () + else + let@ () = + task (D.Notify ((if count mod 2 = 0 then first else second), notification)) + in + fill (count + 1) + in + let@ () = fill 0 in + let@ rejected = + D.task_perform + (D.Notify (first, notification)) + ~success:(fun () -> D.task_return false) + ~failure:(fun error -> + D.task_return + (match error with + | D.Capacity_exceeded _ -> true + | _ -> false)) + in + check rejected "notification count is shared across channels"; + let@ received = task (D.Next_notification second) in + check (received = notification) "rejection preserves queued payload"; + let@ () = task (D.Notify (first, notification)) in + D.task_return ()) + in + match run t (ready t cold) body with + | X.Completed _ -> () + | X.Failed (_, error) -> Alcotest.fail (D.error_message error)) +;; + +let notification_descriptor_bytes env () = + Eio.Switch.run (fun sw -> + let t, cold, _ = fixture env ~sw () in + let payload = String.make 800 'x' in + let notification = D.Policy_failure (D.Network_failure payload) in + let body = + D.with_scope (fun scope -> + let@ first = task (D.Notifications scope) in + let@ second = task (D.Notifications scope) in + let@ () = task (D.Notify (first, notification)) in + let rejected = X.update_demand t demand_change in + check + (match rejected with + | Error (D.Capacity_exceeded _) -> true + | _ -> false) + "descriptors share notification byte budget"; + let@ rejected = + D.task_perform + (D.Notify (second, notification)) + ~success:(fun () -> D.task_return false) + ~failure:(fun error -> + D.task_return + (match error with + | D.Capacity_exceeded _ -> true + | _ -> false)) + in + check rejected "notification bytes are shared across channels"; + let@ received = task (D.Next_notification first) in + check + (received = notification && String.length payload = 800) + "byte rejection preserves caller and queued payload"; + ignore (get (X.update_demand t demand_change)); + let@ rejected = + D.task_perform + (D.Notify (second, notification)) + ~success:(fun () -> D.task_return false) + ~failure:(fun error -> + D.task_return + (match error with + | D.Capacity_exceeded _ -> true + | _ -> false)) + in + check rejected "notification admission includes descriptors"; + D.task_return ()) + in + match run t (ready t cold) body with + | X.Completed _ -> () + | X.Failed (_, error) -> Alcotest.fail (D.error_message error)) +;; + +let active_asset_capacity env () = + Eio.Switch.run (fun sw -> + let module A = Logseq_db_types.Asset_descriptor in + let asset = + Result.get_ok + (A.create + ~uuid: + (Result.get_ok + (Logseq_db_types.Graph_types.Uuid.of_string + "11111111-1111-4111-8111-111111111111")) + ~source: + (A.Managed + (Some + (Result.get_ok + (A.version ~checksum:(String.make 64 'a') ~file_type:"png")))) + ~current_checksum:None + ~size:(Some 1L) + ~dimensions:None) + in + let entered = ref 0 in + let release, resolve = Eio.Promise.create () in + let dispatch : X.dispatch = + { (capabilities ()).dispatch with + assets = + (fun (type a) ~scope:_ ~identity:_ (op : a D.assets) : (a, D.error) result -> + match op with + | D.Download _ -> + incr entered; + Eio.Promise.await release; + Ok D.Waiting_remote + | D.Upload _ -> + incr entered; + if !entered <= 2 then Eio.Promise.await release; + Ok () + | _ -> Error D.Stale_context) + } + in + let t, cold, _ = fixture env ~sw ~dispatch () in + get (X.open_network t ~authorization_epoch:7L); + let rec until n = + if !entered >= n + then D.task_return () + else D.task_bind (task D.Yield) (fun () -> until n) + in + let body = + D.with_scope (fun scope -> + let@ first = D.spawn scope (task (D.Task_assets (D.Download asset))) in + let@ () = until 1 in + let@ second = D.spawn scope (task (D.Task_assets (D.Upload asset))) in + let@ () = until 2 in + let@ rejected = + D.task_perform + (D.Task_assets (D.Upload asset)) + ~success:(fun () -> D.task_return false) + ~failure:(fun e -> + D.task_return + (match e with + | D.Capacity_exceeded _ -> true + | _ -> false)) + in + Eio.Promise.resolve resolve (); + check rejected "third transfer rejects before dispatch"; + check (!entered = 2) "rejection starts no transfer"; + let@ _ = D.join (get first) in + let@ _ = D.join (get second) in + task (D.Task_assets (D.Upload asset))) + in + match run t (ready t cold) body with + | X.Completed _ -> check (!entered = 3) "joined work returns capacity after rejection" + | X.Failed (_, e) -> Alcotest.fail (D.error_message e)) +;; + +let active_asset_cancel_join env () = + Eio.Switch.run (fun sw -> + let asset = + match demand_change with + | D.Demand d -> d.asset + | D.Withdraw _ -> assert false + in + let entered = ref 0 + and stopping = ref 0 in + let joined, resolve_join = Eio.Promise.create () in + let dispatch : X.dispatch = + { (capabilities ()).dispatch with + assets = + (fun (type a) ~scope ~identity:_ (op : a D.assets) : (a, D.error) result -> + match op with + | D.Upload _ -> + X.Callback_bridge.run + ~scope + ~resources:(fun () -> Rrbvec.empty) + { start = + (fun ~deliver -> + incr entered; + if !entered > 2 then deliver (Ok ()) else Ok ()) + ; stop = + (fun () -> + incr stopping; + Ok ()) + ; await_stopped = + (fun ~deadline:_ -> + if !entered <= 2 then Eio.Promise.await joined; + Ok ()) + } + | _ -> Error D.Stale_context) + } + in + let t, cold, _ = fixture env ~sw ~dispatch () in + get (X.open_network t ~authorization_epoch:7L); + let rec until counter n = + if !counter >= n + then D.task_return () + else D.task_bind (task D.Yield) (fun () -> until counter n) + in + let body = + D.with_scope (fun scope -> + let@ first = D.spawn scope (task (D.Task_assets (D.Upload asset))) in + let@ second = D.spawn scope (task (D.Task_assets (D.Upload asset))) in + let@ () = until entered 2 in + let@ () = D.cancel_child (get first) in + let@ () = until stopping 1 in + let@ rejected = + D.task_perform + (D.Task_assets (D.Upload asset)) + ~success:(fun () -> D.task_return false) + ~failure:(fun e -> + D.task_return + (match e with + | D.Capacity_exceeded _ -> true + | _ -> false)) + in + Eio.Promise.resolve resolve_join (); + check rejected "cancel retains active capacity until physical join"; + check (!entered = 2) "cancel does not start a third producer"; + let@ _ = D.join (get first) in + let@ () = D.cancel_child (get second) in + let@ _ = D.join (get second) in + D.task_return ()) + in + match run t (ready t cold) body with + | X.Completed _ -> () + | X.Failed (_, e) -> Alcotest.fail (D.error_message e)) +;; + +let active_asset_foreground env () = + Eio.Switch.run (fun sw -> + let t, cold, _ = fixture env ~sw () in + let module A = Logseq_db_types.Asset_descriptor in + let base = + match demand_change with + | D.Demand d -> d.asset + | _ -> assert false + in + let asset checksum = + Result.get_ok + (A.create + ~uuid:base.uuid + ~source: + (A.Managed + (Some + (Result.get_ok + (A.version ~checksum:(String.make 64 checksum) ~file_type:"png")))) + ~current_checksum:None + ~size:(Some 1L) + ~dimensions:None) + in + let change id selected priority = + D.Demand { id = get (D.demand_id id); asset = selected; priority } + in + let rec next () = + let@ input = task D.Next_input in + match input with + | D.Demand_changed (receipt, _) -> D.task_return receipt + | _ -> next () + in + let body = + D.with_scope (fun scope -> + ignore (get (X.update_demand t (change "bg" (asset 'a') D.Background))); + let@ background = next () in + let@ first = D.spawn_asset scope background (task (D.Sleep 3600.)) in + ignore (get first); + ignore (get (X.update_demand t (change "fg" (asset 'b') D.Foreground))); + let@ foreground = next () in + let@ second = D.spawn_asset scope foreground (task (D.Sleep 3600.)) in + ignore (get second); + ignore (get (X.update_demand t (change "extra" (asset 'c') D.Foreground))); + let@ extra = next () in + let@ rejected = D.spawn_asset scope extra (D.task_return ()) in + let cap = + match rejected with + | Error (D.Capacity_exceeded _) -> true + | _ -> false + in + let@ () = task (D.Finish_demand extra) in + check + (match X.update_demand t (change "shared" (asset 'b') D.Foreground) with + | Error (D.Capacity_exceeded _) -> true + | _ -> false) + "finished interest remains descriptor charged until withdrawal"; + ignore (get (X.update_demand t (D.Withdraw (get (D.demand_id "extra"))))); + let@ withdrawn = next () in + let@ () = task (D.Finish_demand withdrawn) in + ignore (get (X.update_demand t (change "shared" (asset 'b') D.Foreground))); + let@ shared = next () in + let@ shared_child = D.spawn_asset scope shared (D.task_return ()) in + let demotion = X.update_demand t (change "fg" (asset 'b') D.Background) in + let shared_demotion = + X.update_demand t (change "shared" (asset 'b') D.Background) + in + let@ _ = D.join (get shared_child) in + let@ () = D.cancel_child (get first) in + let@ _ = D.join (get first) in + let@ () = D.cancel_child (get second) in + let@ _ = D.join (get second) in + D.task_return (cap, Result.is_ok demotion, shared_demotion)) + in + match run t (ready t cold) body with + | X.Completed (_, (cap, demoted, last_demotion)) -> + check cap "asset child reserves before allocation"; + check demoted "shared foreground interest keeps shared work foreground"; + check + (match last_demotion with + | Error (D.Capacity_exceeded _) -> true + | _ -> false) + "last foreground demotion cannot borrow reserved work" + | X.Failed (_, e) -> Alcotest.fail (D.error_message e)) +;; + +let active_asset_shared_download env () = + Eio.Switch.run (fun sw -> + let selected = + match demand_change with + | D.Demand d -> d.asset + | _ -> assert false + in + let started = ref 0 in + let release, resolve = Eio.Promise.create () in + let dispatch : X.dispatch = + { (capabilities ()).dispatch with + assets = + (fun (type a) ~scope:_ ~identity:_ (op : a D.assets) : (a, D.error) result -> + match op with + | D.Download _ -> + incr started; + Eio.Promise.await release; + Ok D.Waiting_remote + | _ -> Error D.Stale_context) + } + in + let t, cold, _ = fixture env ~sw ~dispatch () in + get (X.open_network t ~authorization_epoch:7L); + let rec next () = + let@ input = task D.Next_input in + match input with + | D.Demand_changed (receipt, _) -> D.task_return receipt + | _ -> next () + in + let body = + D.with_scope (fun scope -> + ignore (get (X.update_demand t demand_change)); + let@ first_receipt = next () in + let@ first = + D.spawn_asset scope first_receipt (task (D.Task_assets (D.Download selected))) + in + ignore + (get + (X.update_demand + t + (D.Demand + { id = get (D.demand_id "shared") + ; asset = selected + ; priority = D.Foreground + }))); + let@ second_receipt = next () in + let@ second = + D.spawn_asset scope second_receipt (task (D.Task_assets (D.Download selected))) + in + let@ () = task D.Yield in + let@ () = task D.Yield in + let count = !started in + let@ () = D.cancel_child (get first) in + let@ () = task D.Yield in + Eio.Promise.resolve resolve (); + let@ _ = D.join (get first) in + let@ result = D.join (get second) in + D.task_return (count, result)) + in + match run t (ready t cold) body with + | X.Completed (_, (count, result)) -> + Alcotest.(check int) "shared demand admits one physical download" 1 count; + check (result = Ok D.Waiting_remote) "cancelling one interest preserves the other" + | X.Failed (_, error) -> Alcotest.fail (D.error_message error)) +;; + +let active_asset_available_owner env () = + Eio.Switch.run (fun sw -> + let selected = + match demand_change with + | D.Demand demand -> demand.asset + | _ -> assert false + in + let original = + get + (D.asset_file + (get (D.resource ~owner:identity ~token:"shared-original")) + selected) + in + let entered = ref false in + let release, resolve = Eio.Promise.create () in + let dispatch : X.dispatch = + { (capabilities ()).dispatch with + assets = + (fun (type a) + ~scope + ~identity:_ + (operation : a D.assets) + : (a, D.error) result -> + match operation with + | D.Download _ -> + entered := true; + Eio.Promise.await release; + let result = + X.handoff scope (Rrbvec.of_list [ D.File_reference original ]) + in + Result.map (fun () -> D.Available original) result + | D.Lookup _ -> Ok None + | _ -> Error D.Stale_context) + } + in + let t, cold, _ = fixture env ~sw ~dispatch () in + get (X.open_network t ~authorization_epoch:7L); + let rec next () = + let@ input = task D.Next_input in + match input with + | D.Demand_changed (receipt, _) -> D.task_return receipt + | _ -> next () + in + let rec until () = + if !entered + then D.task_return () + else + let@ () = task D.Yield in + until () + in + let body = + D.with_scope (fun scope -> + ignore (get (X.update_demand t demand_change)); + let@ receipt = next () in + let@ first = + D.spawn_asset scope receipt (task (D.Task_assets (D.Download selected))) + in + let@ () = until () in + let@ second = + D.spawn_asset scope receipt (task (D.Task_assets (D.Download selected))) + in + let@ () = task D.Yield in + let@ () = task D.Yield in + let@ () = D.cancel_child (get first) in + let@ () = task D.Yield in + Eio.Promise.resolve resolve (); + let@ _ = D.join (get first) in + let@ result = D.join (get second) in + check + (result = Ok (D.Available original)) + "surviving waiter owns original available file"; + task (D.Task_assets (D.Release_file original))) + in + match run t (ready t cold) body with + | X.Completed _ -> () + | X.Failed (_, error) -> Alcotest.fail (D.error_message error)) +;; + +let active_asset_demotion env () = + Eio.Switch.run (fun sw -> + let t, cold, _ = fixture env ~sw () in + let selected = + match demand_change with + | D.Demand demand -> demand.asset + | _ -> assert false + in + let rec next () = + let@ input = task D.Next_input in + match input with + | D.Demand_changed (receipt, _) -> D.task_return receipt + | _ -> next () + in + let body = + D.with_scope (fun scope -> + ignore (get (X.update_demand t demand_change)); + let@ receipt = next () in + let rec fill count children = + if count = limits.maximum_pending - 3 + then D.task_return children + else + let@ child = D.spawn_asset scope receipt (task (D.Sleep 3600.)) in + fill (count + 1) (get child :: children) + in + let@ children = fill 0 [] in + let demoted = + X.update_demand + t + (D.Demand + { id = + (match demand_change with + | D.Demand demand -> demand.id + | _ -> assert false) + ; asset = selected + ; priority = D.Background + }) + in + check + (match demoted with + | Error (D.Capacity_exceeded _) -> true + | _ -> false) + "demotion cannot inherit foreground child reservations"; + let rec join = function + | [] -> D.task_return () + | child :: rest -> + let@ () = D.cancel_child child in + let@ _ = D.join child in + join rest + in + let@ () = join children in + ignore + (get + (X.update_demand + t + (D.Demand + { id = + (match demand_change with + | D.Demand demand -> demand.id + | _ -> assert false) + ; asset = selected + ; priority = D.Background + }))); + D.task_return ()) + in + match run t (ready t cold) body with + | X.Completed _ -> () + | X.Failed (_, error) -> Alcotest.fail (D.error_message error)) +;; + +let active_asset_mixed_direction env () = + Eio.Switch.run (fun sw -> + let selected = + match demand_change with + | D.Demand demand -> demand.asset + | _ -> assert false + in + let entered = ref 0 in + let release, resolve = Eio.Promise.create () in + let dispatch : X.dispatch = + { (capabilities ()).dispatch with + assets = + (fun (type a) + ~scope:_ + ~identity:_ + (operation : a D.assets) + : (a, D.error) result -> + match operation with + | D.Download _ -> + incr entered; + Eio.Promise.await release; + Ok D.Waiting_remote + | D.Upload _ -> + incr entered; + Ok () + | _ -> Error D.Stale_context) + } + in + let t, cold, _ = fixture env ~sw ~dispatch () in + get (X.open_network t ~authorization_epoch:7L); + let rec next () = + let@ input = task D.Next_input in + match input with + | D.Demand_changed (receipt, _) -> D.task_return receipt + | _ -> next () + in + let rec until () = + if !entered > 0 + then D.task_return () + else + let@ () = task D.Yield in + until () + in + let body = + D.with_scope (fun scope -> + ignore (get (X.update_demand t demand_change)); + let@ receipt = next () in + let@ first = + D.spawn_asset scope receipt (task (D.Task_assets (D.Download selected))) + in + let@ () = until () in + let module A = Logseq_db_types.Asset_descriptor in + let other = + Result.get_ok + (A.create + ~uuid:selected.uuid + ~source:selected.source + ~current_checksum:(Some (String.make 64 'b')) + ~size:selected.size + ~dimensions:selected.dimensions) + in + ignore + (get + (X.update_demand + t + (D.Demand + { id = get (D.demand_id "other") + ; asset = other + ; priority = D.Foreground + }))); + let@ other_receipt = next () in + let@ blocker = D.spawn_asset scope other_receipt (task (D.Sleep 3600.)) in + let@ second = + D.spawn_asset scope receipt (task (D.Task_assets (D.Upload selected))) + in + let@ result = D.join (get second) in + let rejected = + match result with + | Error (D.Capacity_exceeded _) -> true + | _ -> false + in + Eio.Promise.resolve resolve (); + let@ _ = D.join (get first) in + check rejected "shared asset cannot silently run both transfer directions"; + check (!entered = 1) "mixed-direction rejection starts no producer"; + let@ () = D.cancel_child (get blocker) in + let@ _ = D.join (get blocker) in + task (D.Task_assets (D.Upload selected))) + in + match run t (ready t cold) body with + | X.Completed _ -> () + | X.Failed (_, error) -> Alcotest.fail (D.error_message error)) +;; + +let active_asset_resource_rejection env () = + Eio.Switch.run (fun sw -> + let asset = + match demand_change with + | D.Demand demand -> demand.asset + | _ -> assert false + in + let called = ref false in + let dispatch : X.dispatch = + { (capabilities ()).dispatch with + assets = + (fun (type a) ~scope:_ ~identity:_ (op : a D.assets) : (a, D.error) result -> + match op with + | D.Upload _ -> + called := true; + Ok () + | _ -> Error D.Stale_context) + } + in + let restore ~scope _ = + let references = + List.init limits.maximum_resources (fun index -> + D.File_reference + (get + (D.asset_file + (get (D.resource ~owner:identity ~token:(string_of_int index))) + asset))) + in + get (X.handoff scope (Rrbvec.of_list references)); + Ok { D.keys = D.Unencrypted; checkpoint = None; recovery_pending = false } + in + let t, cold, _ = fixture env ~sw ~dispatch ~restore () in + get (X.open_network t ~authorization_epoch:7L); + let result = run t (ready t cold) (task (D.Task_assets (D.Upload asset))) in + check + (match result with + | X.Failed (_, D.Capacity_exceeded _) -> true + | _ -> false) + "asset reserves cleanup capacity before dispatch"; + check (not !called) "resource rejection does not allocate or start") +;; + +let policy_demand_lifecycle env () = + Eio.Switch.run (fun sw -> + let dispatch : X.dispatch = + { (capabilities ()).dispatch with + assets = + (fun (type a) ~scope:_ ~identity:_ (op : a D.assets) : (a, D.error) result -> + match op with + | D.Lookup _ -> Ok None + | _ -> Error D.Stale_context) + } + in + let t, cold, _ = fixture env ~sw ~dispatch () in + let running = + get (X.start t cold (D.sync_policy service_config.policy ~initial_network:D.Closed)) + in + let selected = + match demand_change with + | D.Demand d -> d.asset + | _ -> assert false + in + let batch offset = + for index = 1 to 1000 do + let id = get (D.demand_id (string_of_int (offset + index))) in + ignore + (get + (X.update_demand + t + (D.Demand { id; asset = selected; priority = D.Foreground }))); + let rec terminal () = + let output = get (X.receive_demand t) in + get (X.release_demand t output.delivery); + match output.availability with + | D.Demand_missing_local -> () + | _ -> terminal () + in + terminal (); + ignore (get (X.update_demand t (D.Withdraw id))); + let output = get (X.receive_demand t) in + check (output.availability = D.Demand_revoked) "withdrawal delivered"; + get (X.release_demand t output.delivery); + for _ = 1 to 8 do + Eio.Fiber.yield () + done + done + in + Eio.Fiber.yield (); + batch 0; + Gc.full_major (); + let before = (Gc.stat ()).live_words in + batch 1000; + Gc.full_major (); + let growth = (Gc.stat ()).live_words - before in + check + (growth < 5000) + (Printf.sprintf "actual runtime and policy retained growth: %d words" growth); + get (X.request_stop t); + match X.await running with + | X.Completed _ -> () + | X.Failed (_, e) -> Alcotest.fail (D.error_message e)) +;; + +let policy_download_capacity env () = + Eio.Switch.run (fun sw -> + let selected = + match demand_change with + | D.Demand d -> d.asset + | _ -> assert false + in + let version = + Result.get_ok + (Logseq_db_types.Asset_descriptor.version + ~checksum:(String.make 64 'a') + ~file_type:"png") + in + let selected = + Result.get_ok + (Logseq_db_types.Asset_descriptor.create + ~uuid:selected.uuid + ~source:(Logseq_db_types.Asset_descriptor.Managed (Some version)) + ~current_checksum:None + ~size:None + ~dimensions:None) + in + let file token = + get (D.asset_file (get (D.resource ~owner:identity ~token)) selected) + in + let original = file "capacity-original" in + let lease = + get + (D.asset_lease + (get (D.resource ~owner:identity ~token:"capacity-lease")) + original) + in + let lookups = ref 0 + and downloads = ref 0 in + let dispatch : X.dispatch = + { (capabilities ()).dispatch with + assets = + (fun (type a) ~scope ~identity:_ (op : a D.assets) : (a, D.error) result -> + match op with + | D.Lookup _ -> + incr lookups; + if !lookups = 1 + then + Result.map + (fun () -> Some original) + (X.handoff scope (Rrbvec.of_list [ D.File_reference original ])) + else Ok None + | D.Retain _ -> + Result.map + (fun () -> lease) + (X.handoff scope (Rrbvec.of_list [ D.Lease_reference lease ])) + | D.Download _ -> + incr downloads; + Ok D.Waiting_remote + | _ -> Error D.Stale_context) + } + in + let restore ~scope _ = + get + (X.handoff + scope + (Rrbvec.of_list + (List.init 5 (fun index -> + D.File_reference (file ("held-" ^ string_of_int index)))))); + Ok { D.keys = D.Unencrypted; checkpoint = None; recovery_pending = false } + in + let t, cold, _ = fixture env ~sw ~dispatch ~restore () in + let running = + get + (X.start + t + cold + (D.sync_policy + service_config.policy + ~initial_network:(D.Open { authorization_epoch = 7L }))) + in + Eio.Fiber.yield (); + ignore + (get + (X.update_demand + t + (D.Demand + { id = get (D.demand_id "capacity-first") + ; asset = selected + ; priority = D.Foreground + }))); + let rec available () = + let output = get (X.receive_demand t) in + match output.availability with + | D.Demand_available _ -> output + | _ -> + get (X.release_demand t output.delivery); + available () + in + let held = available () in + let id = get (D.demand_id "capacity-second") in + ignore + (get + (X.update_demand t (D.Demand { id; asset = selected; priority = D.Foreground }))); + let queued = get (X.receive_demand t) in + check (queued.availability = D.Demand_queued) "second demand accepted"; + get (X.release_demand t queued.delivery); + for _ = 1 to 32 do + Eio.Fiber.yield () + done; + let waiting_lookups = !lookups in + for _ = 1 to 32 do + Eio.Fiber.yield () + done; + let after_wait_lookups = !lookups in + let before_release_downloads = !downloads in + get (X.release_demand t held.delivery); + Alcotest.(check int) + "capacity wait does not spin through lookups" + waiting_lookups + after_wait_lookups; + Alcotest.(check int) "no dispatch before capacity" 0 before_release_downloads; + let resumed = get (X.receive_demand t) in + get (X.release_demand t resumed.delivery); + check + (resumed.availability = D.Demand_waiting_remote) + "accepted receipt resumes without fake failure"; + Alcotest.(check int) "one real attempt after release" 1 !downloads; + get (X.request_stop t); + match X.await running with + | X.Completed _ -> () + | X.Failed (_, e) -> Alcotest.fail (D.error_message e)) +;; + +let resource_demotion env () = + Eio.Switch.run (fun sw -> + let selected = + match demand_change with + | D.Demand demand -> demand.asset + | _ -> assert false + in + let files = + List.init limits.maximum_resources (fun index -> + get + (D.asset_file + (get (D.resource ~owner:identity ~token:("demotion-" ^ string_of_int index))) + selected)) + in + let restore ~scope _ = + get + (X.handoff + scope + (Rrbvec.of_list (List.map (fun file -> D.File_reference file) files))); + Ok { D.keys = D.Unencrypted; checkpoint = None; recovery_pending = false } + in + let t, cold, _ = fixture env ~sw ~restore () in + let body = + D.with_scope (fun _ -> + ignore (get (X.update_demand t demand_change)); + let background = + match demand_change with + | D.Demand demand -> D.Demand { demand with priority = D.Background } + | D.Withdraw _ -> assert false + in + let rejected = X.update_demand t background in + check + (match rejected with + | Error (D.Capacity_exceeded _) -> true + | _ -> false) + "demotion includes retained resources"; + let@ () = task (D.Task_assets (D.Release_file (List.hd files))) in + ignore (get (X.update_demand t background)); + D.task_return ()) + in + match run t (ready t cold) body with + | X.Completed _ -> () + | X.Failed (_, error) -> Alcotest.fail (D.error_message error)) +;; + +let rejected_handoff_preserves_caller env () = + Eio.Switch.run (fun sw -> + let selected = + match demand_change with + | D.Demand demand -> demand.asset + | _ -> assert false + in + let file token = + get (D.asset_file (get (D.resource ~owner:identity ~token)) selected) + in + let held = + List.init (limits.maximum_resources - 1) (fun index -> + file ("held-" ^ string_of_int index)) + in + let first = file "caller-first" + and second = file "caller-second" in + let calls = ref 0 in + let dispatch : X.dispatch = + { (capabilities ()).dispatch with + assets = + (fun (type a) + ~scope + ~identity:_ + (operation : a D.assets) + : (a, D.error) result -> + match operation with + | D.Prune_cache -> + incr calls; + let* () = + if !calls = 1 + then ( + let rejected = + X.handoff + scope + (Rrbvec.of_list [ D.File_reference first; D.File_reference second ]) + in + check + (match rejected with + | Error (D.Capacity_exceeded _) -> true + | _ -> false) + "oversized handoff is atomic"; + X.handoff scope (Rrbvec.of_list [ D.File_reference first ])) + else X.handoff scope (Rrbvec.of_list [ D.File_reference second ]) + in + Ok + { D.resident_bytes = 0L + ; pending_bytes = 0L + ; retained_bytes = 0L + ; cleanup_pending = 0 + ; over_budget = false + } + | _ -> Error D.Stale_context) + } + in + let restore ~scope _ = + get + (X.handoff + scope + (Rrbvec.of_list (List.map (fun file -> D.File_reference file) held))); + Ok { D.keys = D.Unencrypted; checkpoint = None; recovery_pending = false } + in + let t, cold, _ = fixture env ~sw ~dispatch ~restore () in + let body = + D.with_scope (fun _ -> + let@ _ = task (D.Task_assets D.Prune_cache) in + let@ () = task (D.Task_assets (D.Release_file first)) in + let@ _ = task (D.Task_assets D.Prune_cache) in + task (D.Task_assets (D.Release_file second))) + in + match run t (ready t cold) body with + | X.Completed _ -> + check (!calls = 2) "rejected caller reference remains transferable and releasable" + | X.Failed (_, error) -> Alcotest.fail (D.error_message error)) +;; + +let active_asset_pending_slots env () = + Eio.Switch.run (fun sw -> + let t, cold, _ = fixture env ~sw () in + let rec next () = + let@ input = task D.Next_input in + match input with + | D.Demand_changed (receipt, _) -> D.task_return receipt + | _ -> next () + in + let body = + D.with_scope (fun scope -> + ignore (get (X.update_demand t demand_change)); + let@ receipt = next () in + let rec fill count children = + if count = limits.maximum_pending - 3 + then D.task_return children + else + let@ child = D.spawn_asset scope receipt (D.task_return ()) in + fill (count + 1) (get child :: children) + in + let@ children = fill 0 [] in + let@ () = task D.Yield in + let@ rejected = D.spawn_asset scope receipt (D.task_return ()) in + let first = List.hd children in + let@ _ = D.join first in + let@ accepted = D.spawn_asset scope receipt (D.task_return ()) in + D.task_return (rejected, Result.is_ok accepted)) + in + match run t (ready t cold) body with + | X.Completed (_, (rejected, accepted)) -> + check + (match rejected with + | Error (D.Capacity_exceeded _) -> true + | _ -> false) + "asset tasks preserve three pending control slots even after completion"; + check accepted "physical join returns pending reservation" + | X.Failed (_, error) -> Alcotest.fail (D.error_message error)) +;; + +let () = + Eio_main.run (fun env -> + Alcotest.run + "Execute runtime" + [ ( "public boundary" + , List.map + (fun (name, test) -> Alcotest.test_case name `Quick (test env)) + [ "notification aggregate count", notification_capacity + ; "notification descriptor aggregate bytes", notification_descriptor_bytes + ; "active asset capacity", active_asset_capacity + ; "active asset cancel join", active_asset_cancel_join + ; "active asset foreground", active_asset_foreground + ; "active asset shared download", active_asset_shared_download + ; "active asset available ownership", active_asset_available_owner + ; "active asset demotion", active_asset_demotion + ; "active asset mixed direction", active_asset_mixed_direction + ; "active asset resource rejection", active_asset_resource_rejection + ; "policy demand lifecycle metadata", policy_demand_lifecycle + ; ( "policy download capacity resumes" + , fun env () -> + Eio.Time.with_timeout_exn + (Eio.Stdenv.clock env) + 5. + (policy_download_capacity env) ) + ; "resource demotion", resource_demotion + ; "rejected handoff preserves caller", rejected_handoff_preserves_caller + ; "active asset pending slots", active_asset_pending_slots + ; "contexts and typed child", contexts + ; "network join controls", network + ; "stop and durable outbox", stop_and_outbox + ; "cancellation", cancellation + ; "observations", observation + ; "real snapshot activation and read", real_activation + ; "nested unlock consumer", nested_unlock + ; "cancel joins grandchildren", cancelled_subtree + ; "ready child fairness", ready_child_not_starved + ; "held delivery retirement", held_delivery_retirement + ; "failed callback join retains owner", callback_failed_join_retains_owner + ; "released metadata remains bounded", released_metadata_bounded + ; ( "actual service policy receipts and consumer close" + , service_receipts_and_consumer ~withdraw:false ) + ; ( "actual service withdrawal and consumer close" + , service_receipts_and_consumer ~withdraw:true ) + ; "actual service zero retries recovery", service_exhausted_network_recovery 0 + ; ( "actual service exhausted network recovery" + , service_exhausted_network_recovery 5 ) + ; ( "actual service terminal protocol recovery held retirement" + , service_exhausted_network_recovery + ~terminal:(D.Protocol_failure "Original protocol diagnostic") + 0 ) + ; ( "actual service terminal storage recovery held retirement" + , service_exhausted_network_recovery + ~terminal:(D.Storage_failure "Original storage diagnostic") + 0 ) + ; ( "actual service terminal crypto recovery held retirement" + , service_exhausted_network_recovery + ~terminal:(D.Crypto_failure "Original crypto diagnostic") + 0 ) + ; ( "actual service terminal integrity recovery held retirement" + , service_exhausted_network_recovery + ~terminal: + (D.Authoritative_failure + Logseq_overlay_db.Types.Authoritative_checksum_mismatch) + 0 ) + ; ( "actual service authentication is not network recovery" + , service_exhausted_network_recovery + ~terminal:(D.Authentication_failure "Original auth diagnostic") + 0 ) + ; "actual service callback failure cleanup", service_callback_failure + ; "callback capacity before start", callback_capacity + ; "callback terminal capacity rejection", callback_terminal_capacity_rejection + ; "terminal cancellation rejects", terminal_cancel_rejected + ; "released delivery metadata remains bounded", delivery_metadata_bounded + ] ) + ]) +;; diff --git a/logseq_sync/test/indexed_program_contract.ml b/logseq_sync/test/indexed_program_contract.ml new file mode 100644 index 00000000..6ff80156 --- /dev/null +++ b/logseq_sync/test/indexed_program_contract.ml @@ -0,0 +1,140 @@ +module Effects = struct + type error = string + type cold + type ready + + type (_, _, _, _) operation = + | Restore : (cold, ready, cold, unit) operation + | Persist : string -> (ready, ready, ready, string) operation + | Send : string -> (ready, ready, ready, unit) operation +end + +module Program = Logseq_sync_indexed_program.Make (Effects) +open Effects + +let check = Alcotest.(check (list string)) + +let workflow () = + Program.perform + Restore + ~success:(fun () -> + Program.bind + (Program.perform + (Persist "exact-wire") + ~success:Program.return + ~failure:Program.abort) + (fun wire -> + Program.perform (Send wire) ~success:Program.return ~failure:Program.abort)) + ~failure:Program.abort +;; + +let run ~persist_error program = + let rec loop + : type before after value. + string list + -> (before, after, value) Program.t + -> string list * (value, string) result + = + fun trace program -> + match Program.view program with + | Program.Return value -> List.rev trace, Ok value + | Program.Abort error -> List.rev trace, Error error + | Program.Perform (operation, success, failure) -> + (match operation with + | Restore -> loop ("restore" :: trace) (success ()) + | Persist wire -> + let trace = ("persist:" ^ wire) :: trace in + (match persist_error with + | None -> loop trace (success wire) + | Some error -> loop trace (failure error)) + | Send wire -> loop (("send:" ^ wire) :: trace) (success ())) + in + loop [] program +;; + +let test_persist_before_send () = + let trace, result = run ~persist_error:None (workflow ()) in + check "ordered effects" [ "restore"; "persist:exact-wire"; "send:exact-wire" ] trace; + Alcotest.(check bool) "completed" true (result = Ok ()) +;; + +let test_persist_failure () = + let trace, result = run ~persist_error:(Some "disk full") (workflow ()) in + check "no send on failed persistence" [ "restore"; "persist:exact-wire" ] trace; + Alcotest.(check (result unit string)) "original cause" (Error "disk full") result +;; + +let test_bind_failure () = + let invoked = ref false in + let program = + Program.bind (Program.abort "original") (fun () -> + invoked := true; + Program.return ()) + in + let _, result = run ~persist_error:None program in + Alcotest.(check bool) "continuation not run" false !invoked; + Alcotest.(check (result unit string)) "abort preserved" (Error "original") result +;; + +let test_association () = + let first = + Program.perform (Persist "one") ~success:Program.return ~failure:Program.abort + in + let second wire = + Program.perform + (Persist (wire ^ "/two")) + ~success:Program.return + ~failure:Program.abort + in + let third wire = + Program.perform (Send wire) ~success:Program.return ~failure:Program.abort + in + let left = Program.bind (Program.bind first second) third in + let right = Program.bind first (fun value -> Program.bind (second value) third) in + let left_trace, left_result = run ~persist_error:None left in + let right_trace, right_result = run ~persist_error:None right in + check "association preserves effects" left_trace right_trace; + Alcotest.(check (result unit string)) + "association preserves result" + left_result + right_result +;; + +let test_deep_left_bind () = + let program = ref (Program.return 0) in + for _ = 1 to 100_000 do + program := Program.bind !program (fun value -> Program.return (value + 1)) + done; + match Program.view !program with + | Program.Return value -> Alcotest.(check int) "stack safe normalization" 100_000 value + | Program.Abort error -> Alcotest.fail error + | Program.Perform _ -> Alcotest.fail "unexpected effect" +;; + +let test_lazy_continuations () = + let called = ref false in + let program = + Program.bind (Program.return ()) (fun () -> + called := true; + Program.return ()) + in + Alcotest.(check bool) "construction is lazy" false !called; + ignore (Program.view program); + Alcotest.(check bool) "view runs pure continuation" true !called +;; + +let () = + Alcotest.run + "indexed program" + [ ( "composition" + , List.map + (fun (name, test) -> Alcotest.test_case name `Quick test) + [ "persist before send", test_persist_before_send + ; "failed persistence", test_persist_failure + ; "abort bypasses bind", test_bind_failure + ; "association", test_association + ; "deep left bind", test_deep_left_bind + ; "lazy construction", test_lazy_continuations + ] ) + ] +;; diff --git a/logseq_sync/test/local_restore_failure_reconciliation.ml b/logseq_sync/test/local_restore_failure_reconciliation.ml deleted file mode 100644 index 7df42457..00000000 --- a/logseq_sync/test/local_restore_failure_reconciliation.ml +++ /dev/null @@ -1,172 +0,0 @@ -module Core = Logseq_sync_pure_reducer.Core - -let initial () = - let limits = - Core.limits - ~maximum_response_bytes:Logseq_db_types.Limits.maximum_response_bytes - ~maximum_artifact_bytes:(1024 * 1024) - ~submission_batch_size:32 - |> Result.get_ok - in - Core.config ~managed_sync_origin:(Uri.of_string "https://sync.example.test") ~limits - |> Result.get_ok - |> Core.initial - |> Result.get_ok -;; - -let graph : Core.graph = - { graph_id = - Logseq_db_types.Graph_types.Uuid.of_string "11111111-1111-4111-8111-111111111111" - |> Result.get_ok - ; name = "Cached encrypted journal" - ; schema = { major = 1; minor = 0; exact = true } - ; encrypted = true - } -;; - -let require_instruction label select effects = - match List.find_map select effects with - | Some value -> value - | None -> Alcotest.fail ("Missing " ^ label) -;; - -let complete core ticket result = - Core.step core (Core.Runner_completed (Core.Completion (ticket, result))) -;; - -let check_preparation_failure label core = - let snapshot = (Core.state core).snapshot in - Alcotest.(check bool) - (label ^ ": E2EE preparation failure") - true - (snapshot.startup.failure = Some Core.During_e2ee); - Alcotest.(check (option string)) - (label ^ ": diagnostic") - (Some "synthetic key preparation offline") - snapshot.last_error; - Alcotest.(check bool) - (label ^ ": selected graph") - true - (snapshot.selected_graph = Some graph.graph_id); - Alcotest.(check bool) - (label ^ ": failed sync phase") - true - (snapshot.sync_phase = Core.Failed) -;; - -let check_catalog_only label effects = - Alcotest.(check bool) - (label ^ ": no implicit graph opening or recovery") - true - (List.for_all - (function - | Core.Publish _ - | Run (Request (_, Fetch_catalog _)) - | Run (Request (_, Save_catalog _)) -> true - | Run _ | Delegate _ -> false) - effects) -;; - -(* Explicit online recovery can fail before authentication reconciliation; the - existing error must survive catalog reconciliation rather than leave loading. *) -let preserves_local_restore_failure_during_reconciliation () = - let user_id = "cached-account" in - let restoring = Core.step (initial ()) (Core.Restore_local_account { user_id }) in - let load_ticket = - require_instruction - "cached catalog load" - (function - | Core.Run (Core.Request (ticket, Core.Load_catalog _)) -> - Some (ticket : Core.catalog_cache option Core.effect_ticket) - | Run _ | Delegate _ | Publish _ -> None) - restoring.effects - in - let cache = - Core.catalog_cache ~user_id ~graphs:[ graph ] ~selected_graph:(Some graph.graph_id) - in - let restored = complete restoring.next load_ticket (Ok (Some cache)) in - let scope = Core.admitted_graph_scope restored.next |> Option.get in - let inspected = - Core.step - restored.next - (Core.Mirror_inspected (Core.Mirror_available { graph; scope })) - in - let key_ticket = - require_instruction - "cached graph key load" - (function - | Core.Run (Core.Request (ticket, Core.Load_and_unlock_graph_key _)) -> - Some (ticket : Core.graph_key_handle Core.effect_ticket) - | Run _ | Delegate _ | Publish _ -> None) - inspected.effects - in - let failed = - complete - inspected.next - key_ticket - (Error (Core.Effect_failed "wrappedGraphKeyUnavailable")) - in - let permission=Core.step failed.next (Core.Sign_in_requested (Core_contract.startup_scope failed.next)) in - let permission=Core.step permission.next (Core.Account_authenticated {user_id=Some user_id}) in - let recovery=permission in - let graph_key_ticket = require_instruction "explicit E2EE recovery" - (function Core.Run (Core.Request (ticket, Core.Fetch_e2ee_graph_key _)) -> Some (ticket : string Core.effect_ticket) - | _ -> None) recovery.effects in - let failed = complete recovery.next graph_key_ticket - (Error (Core.Effect_failed "synthetic key preparation offline")) in - check_preparation_failure "remote preparation failed" failed.next; - let authenticated = - Core.step failed.next (Core.Account_authenticated { user_id = Some user_id }) - in - let startup = (Core.state authenticated.next).snapshot.startup in - Alcotest.(check bool) "same account authenticated" true startup.authenticated; - Alcotest.(check bool) "catalog reconciliation started" true startup.catalog_loading; - check_preparation_failure "same-account authentication" authenticated.next; - check_catalog_only "authentication" authenticated.effects; - let fetch_ticket = - require_instruction - "catalog reconciliation request" - (function - | Core.Run (Core.Request (ticket, Core.Fetch_catalog account)) -> - Alcotest.(check string) "reconciles the same account" user_id account.user_id; - Some (ticket : Core.graph list Core.effect_ticket) - | Run _ | Delegate _ | Publish _ -> None) - authenticated.effects - in - let refreshed = complete authenticated.next fetch_ticket (Ok [ graph ]) in - let snapshot = (Core.state refreshed.next).snapshot in - Alcotest.(check bool) "catalog loading finished" false snapshot.startup.catalog_loading; - Alcotest.(check bool) "catalog refreshed" true (snapshot.catalog = [ graph ]); - check_preparation_failure "catalog success" refreshed.next; - check_catalog_only "catalog success" refreshed.effects; - let recovery = Core.step refreshed.next (Core.Online_recovery_requested (Core_contract.timeline_scope refreshed.next)) in - let recovery_scope = - require_instruction - "explicit E2EE recovery request" - (function - | Core.Run (Core.Request (_, Core.Fetch_e2ee_graph_key scope)) -> Some scope - | Run _ | Delegate _ | Publish _ -> None) - recovery.effects - in - Alcotest.(check bool) "recovery retains the graph scope" true (recovery_scope = scope); - let snapshot = (Core.state recovery.next).snapshot in - Alcotest.(check bool) - "explicit recovery clears failure" - true - (snapshot.startup.failure = None); - Alcotest.(check (option string)) - "explicit recovery clears diagnostic" - None - snapshot.last_error; - Alcotest.(check bool) - "recovery leaves failed sync phase" - true - (snapshot.sync_phase = Core.Offline) -;; - -let scenario = - Alcotest.test_case - "E2EE preparation failure survives same-account reconciliation" - `Quick - preserves_local_restore_failure_during_reconciliation -;; diff --git a/logseq_sync/test/native_framing_contract.ml b/logseq_sync/test/native_framing_contract.ml new file mode 100644 index 00000000..365c4c88 --- /dev/null +++ b/logseq_sync/test/native_framing_contract.ml @@ -0,0 +1,349 @@ +module Native = Logseq_sync_native_transport +module Ws = Native.Websocket_eio +module Http = Native.Http +module Peer = Tls_peer + +let with_peer = Peer.with_peer +let wait = Peer.wait +let read_file = Peer.read_file +let case = Peer.case + +let authenticator ~support = + let certificate = + X509.Certificate.decode_pem (read_file (Filename.concat support "cert.pem")) + |> Result.get_ok + in + X509.Authenticator.chain_of_trust + ~time:(fun () -> Ptime.of_float_s (Unix.gettimeofday ())) + [ certificate ] +;; + +let response ?(status = 200) ?(headers = "Content-Length: 1\r\n") body = + Printf.sprintf + "HTTP/1.1 %d Test\r\n\ + Content-Type: application/transit+json\r\n\ + X-Snapshot-Row-Count: 1\r\n\ + %s\r\n\ + %s" + status + headers + body +;; + +let check_retired report = + let rows = Yojson.Basic.Util.to_list report in + List.iter + (fun row -> + Alcotest.(check bool) + "peer observed local resource retirement" + true + Yojson.Basic.Util.(row |> member "closed" |> to_bool)) + rows +;; + +let captured_request_headers row = + let encoded = Yojson.Basic.Util.(row |> member "request" |> to_string) in + let request = + String.init + (String.length encoded / 2) + (fun n -> Char.chr (int_of_string ("0x" ^ String.sub encoded (2 * n) 2))) + in + let rec headers = function + | [] | "" :: _ -> [] + | line :: rest -> line :: headers rest + in + String.split_on_char '\n' request + |> List.tl + |> List.map (fun line -> String.lowercase_ascii (String.trim line)) + |> headers +;; + +let check_default_connection report = + Yojson.Basic.Util.to_list report + |> List.iter (fun row -> + Alcotest.(check bool) + "ordinary request omits Connection" + false + (List.exists + (String.starts_with ~prefix:"connection:") + (captured_request_headers row))) +;; + +let http_test ~download ~success ?(segment = 16384) ?(eof = false) wire () = + let report = + with_peer + [ case ~segment ~eof wire ] + (fun ~environment ~sw ~support ~origin -> + let request : Http.request = + { operation = Get + ; uri = origin + ; headers = [] + ; maximum_response_bytes = 1024 + ; expected_content_type = Snapshot_artifact + } + in + let authenticator = authenticator ~support in + let network = Eio.Stdenv.net environment in + let clock = + Native.Transport_time.monotonic (Eio.Stdenv.mono_clock environment) + in + let result = + if download + then + Native.Http_eio.download + ~sw + ~authenticator + ~network + ~clock + ~request + ~destination:(Filename.concat support "artifact") + ~maximum_bytes:1024 + ~on_progress:(fun _ -> ()) + else Native.Http_eio.perform ~sw ~authenticator ~network ~clock request + in + Alcotest.(check bool) "response outcome" success (Result.is_ok result); + if success + then ( + let response = Result.get_ok result in + Alcotest.(check int) "response status" 200 response.status; + Alcotest.(check string) + "exact payload" + "x" + (if download + then read_file (Filename.concat support "artifact") + else response.body))) + in + check_retired report; + check_default_connection report +;; + +let frame ?(fin = true) opcode payload = + let size = String.length payload in + let prefix = + if size < 126 + then String.make 1 (Char.chr size) + else if size <= 65535 + then + String.init 3 (function + | 0 -> Char.chr 126 + | 1 -> Char.chr (size lsr 8) + | _ -> Char.chr (size land 255)) + else + String.init 9 (function + | 0 -> Char.chr 127 + | i -> + Char.chr + (Int64.to_int + (Int64.logand + 255L + (Int64.shift_right_logical (Int64.of_int size) ((8 - i) * 8))))) + in + String.make 1 (Char.chr (opcode lor if fin then 128 else 0)) ^ prefix ^ payload +;; + +let ws_test + ?(extra = "") + ?(segment = 16384) + ?(maximum_frame_bytes = 1024) + ?expected_messages + ?expected_pong + ~messages + ~terminal + wire + () + = + let report = + with_peer + [ case ~ws:true ~extra ~segment wire ] + (fun ~environment ~sw ~support ~origin -> + let received = ref [] + and closed = ref [] + and opened = ref false in + let uri = Uri.with_scheme origin (Some "wss") in + let result = + Ws.connect + ~setup_seconds:5. + ~tls_handshake_seconds:5. + ~http_response_seconds:5. + ~sw + ~authenticator:(authenticator ~support) + ~network:(Eio.Stdenv.net environment) + ~clock:(Native.Transport_time.monotonic (Eio.Stdenv.mono_clock environment)) + ~liveness:Ws.Disabled + ~uri + ~token:"synthetic" + ~maximum_frame_bytes + ~on_message:(fun message -> + Alcotest.(check bool) "open precedes data" true !opened; + received := !received @ [ message ]) + ~on_close:(fun reason -> closed := reason :: !closed) + in + match result with + | Error _ when terminal -> + Alcotest.(check int) "no application data" 0 (List.length !received) + | Error _ -> Alcotest.fail "WebSocket connection failed" + | Ok socket -> + (match Ws.activate socket ~on_open:(fun () -> opened := true) with + | Error _ when terminal -> () + | Error _ -> Alcotest.fail "WebSocket activation failed" + | Ok () -> + wait (Eio.Stdenv.clock environment) (fun () -> + if terminal then !closed <> [] else List.length !received >= messages); + Alcotest.(check int) + "ordered application message count" + messages + (List.length !received); + Option.iter + (fun expected -> + Alcotest.(check (list string)) + "exact application messages" + expected + !received) + expected_messages; + if not terminal then Ws.close socket); + Ws.await_retired socket; + Alcotest.(check int) "one terminal notification" 1 (List.length !closed)) + in + check_retired report; + if not terminal + then ( + let frames = + Yojson.Basic.Util.(report |> to_list |> List.hd |> member "frames" |> to_list) + in + Alcotest.(check bool) + "peer received masked Close" + true + (List.exists + (fun frame -> + Yojson.Basic.Util.(frame |> member "opcode" |> to_int) = 8 + && String.length Yojson.Basic.Util.(frame |> member "mask" |> to_string) = 8) + frames); + Option.iter + (fun payload -> + Alcotest.(check bool) + "peer received masked Pong with exact ping payload" + true + (List.exists + (fun frame -> + Yojson.Basic.Util.(frame |> member "opcode" |> to_int) = 10 + && String.length Yojson.Basic.Util.(frame |> member "mask" |> to_string) + = 8 + && Yojson.Basic.Util.(frame |> member "payload" |> to_string) + = Peer.hex payload) + frames)) + expected_pong) +;; + +let http_cases = + [ "complete length", true, false, response "x" + ; "truncated length", false, true, response ~headers:"Content-Length: 2\r\n" "x" + ; "close delimited", true, true, response ~headers:"" "x" + ; ( "plain chunked" + , true + , false + , response ~headers:"Transfer-Encoding: chunked\r\n" "1\r\nx\r\n0\r\n\r\n" ) + ; ( "truncated chunk" + , false + , true + , response ~headers:"Transfer-Encoding: chunked\r\n" "2\r\nx" ) + ; ( "malformed chunk after payload preserves failure" + , false + , false + , response ~headers:"Transfer-Encoding: chunked\r\n" "1\r\nx\r\nZ\r\n" ) + ; ( "bad content type" + , false + , false + , "HTTP/1.1 200 OK\r\nContent-Type: text/html\r\nContent-Length: 1\r\n\r\nx" ) + ] +;; + +let ws_cases = + let message = {|{"type":"changed","t":42}|} in + [ ( "coalesced fragmented message" + , 1 + , false + , frame ~fin:false 1 (String.sub message 0 3) + ^ frame 0 (String.sub message 3 (String.length message - 3)) ) + ; "ordered multiple messages", 2, false, frame 1 message ^ frame 1 message + ; ( "interleaved ping" + , 1 + , false + , frame ~fin:false 1 (String.sub message 0 3) + ^ frame 9 "probe" + ^ frame 0 (String.sub message 3 (String.length message - 3)) ) + ; "peer close", 0, true, frame 8 "\x03\xe8" + ] +;; + +let scenarios = + List.concat_map + (fun download -> + List.map + (fun (name, success, eof, wire) -> + Alcotest.test_case + ((if download then "download " else "HTTP ") ^ name) + `Quick + (http_test ~download ~success ~eof wire)) + http_cases) + [ false; true ] + @ List.map + (fun (name, messages, terminal, wire) -> + Alcotest.test_case + ("WebSocket " ^ name) + `Quick + (ws_test ~messages ~terminal wire)) + ws_cases + @ [ Alcotest.test_case + "HTTP segmented supported chunked response" + `Quick + (http_test + ~download:false + ~success:true + ~segment:1 + (response ~headers:"Transfer-Encoding: chunked\r\n" "1\r\nx\r\n0\r\n\r\n")) + ; Alcotest.test_case + "WebSocket segmented Upgrade and frame" + `Quick + (ws_test + ~messages:1 + ~terminal:false + ~segment:1 + (frame 1 {|{"type":"changed","t":42}|})) + ; Alcotest.test_case + "WebSocket bounds retained incomplete Upgrade input" + `Quick + (ws_test + ~messages:0 + ~terminal:true + ~extra:("X-Incomplete: " ^ String.make 20000 'x') + "") + ] + @ List.map + (fun size -> + let message = {|{"type":"changed","t":42}|} in + let payload = String.make (size - String.length message) ' ' ^ message in + Alcotest.test_case + (Printf.sprintf "WebSocket minimal length boundary %d" size) + `Quick + (ws_test + ~maximum_frame_bytes:65536 + ~expected_messages:[ payload ] + ~messages:1 + ~terminal:false + (frame 1 payload))) + [ 125; 126; 65535; 65536 ] + @ [ Alcotest.test_case + "WebSocket UTF8 codepoint split across fragments" + `Quick + (ws_test + ~expected_messages:[ "{\"type\":\"error\",\"message\":\"\xe4\xb8\xad\"}" ] + ~expected_pong:"probe" + ~messages:1 + ~terminal:false + (frame ~fin:false 1 "{\"type\":\"error\",\"message\":\"\xe4" + ^ frame 9 "probe" + ^ frame 0 "\xb8\xad\"}")) + ] +;; + +let () = Alcotest.run "native framing regressions" [ "transport", scenarios ] diff --git a/logseq_sync/test/production_transport/dune b/logseq_sync/test/production_transport/dune new file mode 100644 index 00000000..ff8597b9 --- /dev/null +++ b/logseq_sync/test/production_transport/dune @@ -0,0 +1,10 @@ +(test + (name transport_owner_contract) + (modules transport_owner_contract) + (libraries datascript_ocaml.types alcotest eio eio_main logseq_sync.transport logseq_sync_sync_dsl_impl logseq_sync_execute_impl logseq_overlay_db logseq_db_types logseq_sync.protocol rrbvec uri x509)) + +(test + (name snapshot_owner_contract) + (modules snapshot_owner_contract tls_peer) + (deps snapshot_rows.json) + (libraries datascript_ocaml.types mirage-crypto logseq_sync.crypto logseq_sync_execute_owners alcotest eio eio_main logseq_sync.transport logseq_sync.native_transport logseq_journal.sync_lifecycle logseq_sync.sync_service logseq_sync_sync_service_impl logseq_sync.worker_owner logseq_sync.protocol logseq_db_storage logseq_overlay_db.impl datascript-ocaml-native sqlite3 mtime.clock.os logseq_sync_sync_dsl_impl logseq_sync_execute_impl logseq_overlay_db logseq_db_types melange-transit-core melange-transit-native rrbvec unix uri yojson)) diff --git a/logseq_sync/test/production_transport/snapshot_owner_contract.ml b/logseq_sync/test/production_transport/snapshot_owner_contract.ml new file mode 100644 index 00000000..bbe39bac --- /dev/null +++ b/logseq_sync/test/production_transport/snapshot_owner_contract.ml @@ -0,0 +1,1866 @@ +module D = Sync_dsl.Core +module S = Logseq_sync_transport.Snapshot +module C = Logseq_sync_transport.Http_client +module H = Logseq_sync_native_transport.Http +module N = Logseq_sync_native_transport.Http_eio + +let get = function + | Ok value -> value + | Error error -> Alcotest.fail (D.error_message error) +;; + +let identity = + get + (D.identity + ~origin:"https://example.com" + ~account:"synthetic" + ~graph:"60000000-0000-4000-8000-000000000001" + ~account_generation:0L + ~graph_generation:0L) +;; + +let cursor = Result.get_ok (Logseq_overlay_db.Types.Server_cursor.of_int64 3L) + +let limits_with_items maximum_items = + get + (D.limits + ~maximum_items + ~maximum_bytes:1048576 + ~maximum_pending:8 + ~maximum_resources:8 + ~steps_per_turn:8 + ~maximum_tasks:8 + ~maximum_messages:4 + ~maximum_message_bytes:1048576 + ~maximum_demands:8 + ~maximum_active_assets:2 + ~foreground_reserve:1 + ~foreground_demand_reserve:1 + ~foreground_descriptor_reserve_bytes:128 + ~asset_cache_budget_bytes:4096L + ~asset_maximum_file_bytes:1024 + ~asset_pending_budget_bytes:4096L + ~snapshot_maximum_artifact_bytes:8388608L + ~snapshot_staging_budget_bytes:33554432L + ~snapshot_scratch_budget_bytes:33554432L) +;; + +let limits = limits_with_items 1024 + +let scope : C.scope = + { check_current = (fun () -> Ok ()); admit_network = (fun () -> Ok ()) } +;; + +let response ?(status = 200) ?(headers = []) body : N.response = { status; headers; body } + +let rec remove path = + match Unix.lstat path with + | { st_kind = Unix.S_DIR; _ } -> + Array.iter (fun name -> remove (Filename.concat path name)) (Sys.readdir path); + Unix.rmdir path + | _ -> Unix.unlink path + | exception Unix.Unix_error (Unix.ENOENT, _, _) -> () +;; + +let with_directory callback = + let path = Filename.temp_file ~temp_dir:(Sys.getcwd ()) "snapshot-owner-" "" in + Sys.remove path; + Unix.mkdir path 0o700; + Fun.protect ~finally:(fun () -> remove path) (fun () -> callback path) +;; + +let fixture () = + let open Yojson.Safe.Util in + let rec root path = + if Sys.file_exists (Filename.concat path "dune-project") + then path + else root (Filename.dirname path) + in + let rows = + Yojson.Safe.from_file + (Filename.concat + (root (Sys.getcwd ())) + "logseq_sync/test/production_transport/snapshot_rows.json") + |> member "rows" + |> to_list + in + let module V = Transit_core.Json in + let module Codec = Transit_native.Transit.Json in + let rows = + List.map + (fun row -> + V.Array + [ V.Int (row |> member "addr" |> to_int) + ; V.String (row |> member "content" |> to_string) + ; (match row |> member "addresses" with + | `Null -> V.Null + | value -> V.String (to_string value)) + ]) + rows + in + let payload = Codec.to_string ~mode:Codec.Verbose (V.Array rows) in + let prefix = Bytes.create 4 in + for index = 0 to 3 do + Bytes.set + prefix + index + (Char.chr ((String.length payload lsr ((3 - index) * 8)) land 255)) + done; + Bytes.to_string prefix ^ payload, List.length rows +;; + +let write path wire = + let channel = open_out_bin path in + Fun.protect + ~finally:(fun () -> close_out_noerr channel) + (fun () -> output_string channel wire) +;; + +let expect predicate = function + | Error error when predicate error -> () + | Error error -> Alcotest.fail (D.error_message error) + | Ok _ -> Alcotest.fail "Expected failure" +;; + +let protocol = function + | D.Protocol_failure _ -> true + | _ -> false +;; + +let with_owner + ?(encrypted = false) + ?(metadata = "{\"ok\":true,\"url\":\"https://cdn.example.com/snapshot\",\"t\":3}") + ?wire + ?rows + callback + = + Eio_main.run (fun _ -> + with_directory (fun root -> + let original, count = fixture () in + let wire = Option.value wire ~default:original + and rows = Option.value rows ~default:count in + let requests = ref [] + and downloads = ref 0 + and tokens = ref [] in + let backend : C.backend = + { perform = + (fun request -> + requests := request :: !requests; + Ok (response metadata)) + ; download = + (fun ~request ~destination ~maximum_bytes -> + incr downloads; + requests := request :: !requests; + Alcotest.(check bool) "bounded stream" true (maximum_bytes > 0); + write destination wire; + Ok (response ~headers:[ "x-snapshot-row-count", string_of_int rows ] "")) + } + in + let http = + C.create ~backend ~token:(fun ~epoch ~refresh -> + tokens := (epoch, refresh) :: !tokens; + Ok "synthetic-bearer") + in + let owner = + get + (S.create + ~verify:Logseq_overlay_db.Database.inspect_snapshot_storage + ~root + ~identity + ~limits + ~encrypted + ~http) + in + Fun.protect + ~finally:(fun () -> get (S.close owner)) + (fun () -> callback root owner requests downloads tokens))) +;; + +let encryption_failure () = + with_owner ~encrypted:true (fun root owner _ _ _ -> + expect + protocol + (S.acquire + owner + ~scope + ~epoch:7L + ~handoff:(fun _ -> Alcotest.fail "Invalid encryption must not hand off") + cursor); + Alcotest.(check int) + "encryption failure cleans staging" + 0 + (Array.length (Sys.readdir root))) +;; + +let acquire owner handoff = S.acquire owner ~scope ~epoch:7L ~handoff cursor + +let success () = + with_owner (fun root owner requests downloads tokens -> + let handed = ref [] in + let snapshot = + get + (acquire owner (fun refs -> + handed := Rrbvec.to_list refs; + Ok ())) + in + let source = get (S.source owner snapshot) in + Alcotest.(check bool) "retained actual file" true (Sys.file_exists source.path); + Alcotest.(check int) "single download" 1 !downloads; + Alcotest.(check int) "no preliminary pull" 2 (List.length !requests); + Alcotest.(check int) "only metadata authenticates" 1 (List.length !tokens); + let artifact = List.hd !requests in + Alcotest.(check bool) + "no CDN bearer" + false + (List.mem_assoc "authorization" artifact.H.headers); + Alcotest.(check int) "original handoff" 1 (List.length !handed); + Alcotest.(check int) + "canonical checksum length" + 28 + (String.length snapshot.D.checksum); + let forged_resource = + get (D.resource ~owner:identity ~token:snapshot.resource.token) + in + let forged = + get + (D.snapshot + ~resource:forged_resource + ~through:cursor + ~checksum:snapshot.checksum + ~encrypted:false) + in + expect + (function + | D.Stale_context -> true + | _ -> false) + (S.source owner forged); + get (S.release owner ~identity snapshot); + get (S.release owner ~identity snapshot); + Alcotest.(check bool) "physical release" false (Sys.file_exists source.path); + Alcotest.(check int) "all owned staging gone" 0 (Array.length (Sys.readdir root))) +;; + +let metadata_failure () = + List.iter + (fun metadata -> + with_owner ~metadata (fun root owner _ downloads _ -> + expect + protocol + (acquire owner (fun _ -> Alcotest.fail "invalid metadata handed off")); + Alcotest.(check int) "no download" 0 !downloads; + Alcotest.(check int) "no artifacts" 0 (Array.length (Sys.readdir root)))) + [ "{\"ok\":true,\"url\":\"https://cdn.example.com/x\"}" + ; "{\"ok\":true,\"url\":\"https://cdn.example.com/x\",\"t\":-1}" + ; "{\"ok\":true,\"url\":\"https://cdn.example.com/x\",\"t\":4}" + ; "{\"ok\":true,\"url\":\"http://cdn.example.com/x\",\"t\":3}" + ] +;; + +let framing_failure () = + with_owner ~wire:"\000\000\000\010broken" ~rows:2 (fun root owner _ _ _ -> + expect + protocol + (acquire owner (fun _ -> Alcotest.fail "malformed artifact handed off")); + Alcotest.(check int) "failed framing cleaned" 0 (Array.length (Sys.readdir root))) +;; + +let row_failure () = + with_owner ~rows:1 (fun root owner _ _ _ -> + expect protocol (acquire owner (fun _ -> Alcotest.fail "wrong count handed off")); + Alcotest.(check int) "failed rows cleaned" 0 (Array.length (Sys.readdir root))) +;; + +let handoff_failure () = + with_owner (fun root owner _ _ _ -> + expect + (function + | D.Stale_context -> true + | _ -> false) + (acquire owner (fun _ -> Error D.Stale_context)); + Alcotest.(check int) "failed handoff cleaned" 0 (Array.length (Sys.readdir root))) +;; + +let auth () = + Eio_main.run (fun _ -> + let calls = ref 0 + and tokens = ref [] in + let backend : C.backend = + { perform = + (fun _ -> + incr calls; + Ok (response ~status:(if !calls = 1 then 401 else 200) "ok")) + ; download = (fun ~request:_ ~destination:_ ~maximum_bytes:_ -> assert false) + } + in + let client = + C.create ~backend ~token:(fun ~epoch:_ ~refresh -> + tokens := refresh :: !tokens; + Ok "synthetic") + in + ignore + (get + (C.perform client ~scope ~epoch:7L ~authenticated:true (fun token -> + H.catalog ~base_url:(Uri.of_string identity.origin) ~token:(Option.get token)))); + Alcotest.(check (list bool)) "one checked refresh" [ true; false ] !tokens; + Alcotest.(check int) "two attempts" 2 !calls) +;; + +let http_status_policy () = + List.iter + (fun (status, network) -> + let calls = ref 0 in + let backend : C.backend = + { perform = + (fun _ -> + incr calls; + Ok (response ~status "")) + ; download = (fun ~request:_ ~destination:_ ~maximum_bytes:_ -> assert false) + } + in + let client = + C.create ~backend ~token:(fun ~epoch:_ ~refresh:_ -> Ok "synthetic") + in + let result = + C.perform client ~scope ~epoch:1L ~authenticated:true (fun _ -> + H.artifact + ~base_url:(Uri.of_string identity.origin) + ~uri:(Uri.of_string identity.origin) + ~token:None) + in + expect + (function + | D.Network_failure _ -> network + | D.Protocol_failure _ -> not network + | _ -> false) + result; + Alcotest.(check int) "no hidden refresh or retry" 1 !calls) + [ 400, false; 422, false; 429, false; 500, false; 503, false ] +;; + +let native_client + ~environment + ~sw + ?(http_response_seconds = 1.) + ?(download_seconds = 3.) + () + = + let deadlines : Execute.Core.network_deadlines = + { setup_seconds = 1. + ; tls_handshake_seconds = 1. + ; http_response_seconds + ; download_seconds + } + in + C.create + ~backend: + (C.native_backend + ~sw + ~clock:(Eio.Stdenv.mono_clock environment) + ~network:(Eio.Stdenv.net environment) + ~authenticator:(fun ?ip:_ ~host:_ _ -> Ok None) + ~deadlines) + ~token:(fun ~epoch:_ ~refresh:_ -> Ok "synthetic-bearer") +;; + +let wire_response headers body = "HTTP/1.1 200 OK\r\n" ^ headers ^ "\r\n" ^ body + +let check_retired report = + let open Yojson.Basic.Util in + report + |> to_list + |> List.iter (fun row -> + Alcotest.(check bool) "physical peer close" true (row |> member "closed" |> to_bool)) +;; + +let native_snapshot layers () = + let wire, rows = fixture () in + let metadata = "{\"ok\":true,\"url\":\"{{ORIGIN}}/snapshot\",\"t\":3}" in + let metadata = + wire_response + "Content-Type: application/json\r\nContent-Length: {{LENGTH}}\r\n" + metadata + in + let artifact = + wire_response + (Printf.sprintf "Content-Length: {{LENGTH}}\r\nX-Snapshot-Row-Count: %d\r\n" rows) + wire + in + let artifact = + match Tls_peer.case artifact with + | `Assoc fields -> `Assoc (("gzip_layers", `Int layers) :: fields) + | _ -> assert false + in + let report = + Tls_peer.with_peer + [ Tls_peer.case metadata; artifact ] + (fun ~environment ~sw ~support ~origin -> + let http = native_client ~environment ~sw () in + let identity = + get + (D.identity + ~origin:(Uri.to_string origin) + ~account:"synthetic" + ~graph:identity.graph + ~account_generation:0L + ~graph_generation:0L) + in + let staging = Filename.concat support "staging" in + Unix.mkdir staging 0o700; + let owner = + get + (S.create + ~verify:Logseq_overlay_db.Database.inspect_snapshot_storage + ~root:staging + ~identity + ~limits + ~encrypted:false + ~http) + in + Fun.protect + ~finally:(fun () -> get (S.close owner)) + (fun () -> + let result = acquire owner (fun _ -> Ok ()) in + if layers > 2 + then expect protocol result + else ( + let snapshot = get result in + let source = get (S.source owner snapshot) in + Alcotest.(check string) + "actual native artifact" + wire + (Tls_peer.read_file source.path); + get (S.release owner ~identity snapshot)); + Alcotest.(check int) + "native staging cleaned" + 0 + (Array.length (Sys.readdir staging)))) + in + check_retired report +;; + +let remote_graph_fixture ?(protected_titles = []) () = + with_directory (fun root -> + let module Storage = Logseq_db_storage.Logseq_sqlite_storage in + let attribute : Datascript.schema_attr = + { cardinality = One + ; unique = None + ; indexed = true + ; is_component = false + ; no_history = false + ; doc = None + ; value_type = None + ; tuple_attrs = None + ; tuple_types = None + } + in + let schema = + List.map + (fun name -> + let attribute = + match name with + | "db/ident" | "block/uuid" -> { attribute with unique = Some Identity } + | "block/parent" | "block/page" -> + { attribute with value_type = Some RefType } + | "block/tags" -> + { attribute with cardinality = Many; value_type = Some RefType } + | _ -> attribute + in + name, attribute) + [ "db/ident" + ; "block/uuid" + ; "block/title" + ; "block/name" + ; "block/journal-day" + ; "block/link" + ; "block/refs" + ; "block/tags" + ; "block/parent" + ; "block/page" + ; "block/order" + ; "kv/value" + ] + in + let fact e a v : Datascript.datom = { e; a; v; tx = 536870913; added = true } in + let kv e name value = [ fact e "db/ident" (Keyword name); fact e "kv/value" value ] in + let database = + Datascript.init_db + ~schema + (List.concat + [ kv 1 "logseq.kv/graph-remote?" (Bool true) + ; kv 2 "logseq.kv/graph-uuid" (Uuid identity.graph) + ; kv 3 "logseq.kv/db-type" (String "db") + ; kv + 4 + "logseq.kv/schema-version" + (Map [ Keyword "major", Int64 65L; Keyword "minor", Int64 33L ]) + ; (if protected_titles = [] + then [] + else kv 5 "logseq.kv/graph-rtc-e2ee?" (Bool true)) + ; [ fact 6 "db/ident" (Keyword "logseq.class/Journal") ] + ; List.concat + (List.mapi + (fun index title -> + let entity = 100 + index in + [ fact + entity + "block/uuid" + (Uuid (Printf.sprintf "70000000-0000-4000-8000-%012d" index)) + ; fact entity "block/title" (String title) + ]) + protected_titles) + ]) + in + let path = Filename.concat root "db.sqlite" in + let sqlite = Sqlite3.db_open path in + Sqlite3.Rc.check + (Sqlite3.exec + sqlite + "CREATE TABLE kvs (addr INTEGER PRIMARY KEY, content TEXT, addresses JSON)"); + ignore (Sqlite3.db_close sqlite); + let connection = Result.get_ok (Storage.open_database path) in + let callbacks = Storage.connection_callbacks connection in + Fun.protect + ~finally:(fun () -> ignore (Storage.close callbacks)) + (fun () -> + Result.get_ok (callbacks.begin_staging ()); + ignore (Datascript.store ~storage:callbacks.storage database); + let batch = Result.get_ok (callbacks.finish_staging None []) in + Result.get_ok (Storage.commit_batch callbacks batch)); + let checksum = + Result.get_ok + (Logseq_overlay_db.Database.inspect_snapshot_storage + ~check_current:(fun () -> ()) + ~scratch_bytes:33554432L + ~encrypted:(protected_titles <> []) + ~path) + in + let sqlite = Sqlite3.db_open path in + let rows = ref [] in + Fun.protect + ~finally:(fun () -> ignore (Sqlite3.db_close sqlite)) + (fun () -> + Sqlite3.Rc.check + (Sqlite3.exec + sqlite + ~cb:(fun values _ -> + let open Transit_core.Json in + rows + := Array + [ Int (int_of_string (Option.get values.(0))) + ; String (Option.get values.(1)) + ; (match values.(2) with + | None -> Null + | Some value -> String value) + ] + :: !rows) + "SELECT addr, content, addresses FROM kvs ORDER BY addr")); + let module Codec = Transit_native.Transit.Json in + let payload = + Codec.to_string ~mode:Codec.Verbose (Transit_core.Json.Array (List.rev !rows)) + in + let prefix = + Bytes.init 4 (fun index -> + Char.chr ((String.length payload lsr ((3 - index) * 8)) land 255)) + in + Bytes.to_string prefix ^ payload, List.length !rows, checksum) +;; + +let protected_append_echo database ~path ~protect ~unprotect ~duplicate = + let module Database = Logseq_overlay_db.Database in + let module Types = Logseq_overlay_db.Types in + let module Graph = Logseq_db_types.Graph_types in + let module J = Transit_core.Json in + let module Codec = Transit_native.Transit.Json in + let ok result = + match result with + | Ok value -> value + | Error _ -> Alcotest.fail "Synthetic protected append operation failed" + in + let uuid ordinal = + ok (Graph.Uuid.of_string (Printf.sprintf "90000000-0000-4000-8000-%012d" ordinal)) + in + let page = uuid 7790 + and root = uuid 7791 + and child = uuid 7792 in + let expected = ok (Database.write_precondition ~blocks:[] ~pages:[] ~scopes:[]) in + let intent = + ok + (Database.persist_local_intent + database + ~namespace:(String.make 64 'a') + ~payload:"Synthetic AES append" + ~expected + (Types.Append_journal + { mutation_id = uuid 7793 + ; page + ; title = "2026-11-02" + ; journal_day = 20261102 + ; root = + { uuid = root + ; title = "Synthetic AES root" + ; task_status = None + ; children = + Rrbvec.singleton + { Types.uuid = child + ; title = "Synthetic AES child" + ; task_status = None + ; children = Rrbvec.empty + } + } + })) + in + let local = ok (Database.commit_local_intent database intent) in + let local = Database.intent_commit_outcome local in + let view () = ok (Database.inspect_sync database) in + let transition change = + ok + (Database.begin_outbox_transition + database + ~expected:(Types.sync_view_token (view ())) + change) + in + let prepared, request = transition (Types.Submit_group [ local.mutation_id ]) in + let request = Option.get request in + let protected = request, get (protect request) in + let submitted = + ok (Database.apply_outbox_transition database prepared ~encrypted:(Some protected)) + |> fun commit -> Option.get commit.Types.submission_batch + in + let wires = Types.submission_batch_wires submitted in + Alcotest.(check int) "one atomic AES transaction" 1 (List.length wires); + let operations = + Types.submission_wire_protected_transaction (List.hd wires) + |> Codec.of_string + |> function + | J.Array operations -> List.rev operations + | _ -> Alcotest.fail "Synthetic append wire is not an array" + in + let through = ok (Types.Server_cursor.of_int64 4L) in + let barrier = + { Types.through + ; checksum = ok (Types.Checksum.of_string "checksum:v1:0000000000000000") + } + in + let prepared, crypto = + transition + (Types.Accept_group { batch_id = Types.submission_batch_id submitted; barrier }) + in + Alcotest.(check bool) "acceptance needs no crypto" true (crypto = None); + ignore (ok (Database.apply_outbox_transition database prepared ~encrypted:None)); + let state () = + let sync = view () in + let snapshot = ok (Database.current_snapshot database) in + Fun.protect + ~finally:(fun () -> Database.release_snapshot snapshot) + (fun () -> + ( Types.sync_view_checkpoint sync + , Types.sync_view_submissions sync + , Database.snapshot_version snapshot + , ok (Database.get_outline_blocks snapshot [ root; child ]) )) + in + let apply operations = + let wire = Codec.to_string ~mode:Codec.Verbose (J.Array operations) in + let batch = + ok + (Types.authoritative_batch + ~maximum_count:16 + ~maximum_bytes:1048576 + ~transactions: + [ Types.authoritative_transaction + ~cursor:through + ~transaction: + (ok (Types.encoded_transaction_of_string ~maximum_bytes:1048576 wire)) + ] + ~through + ~checksum:None) + in + let prepared, request = + ok + (Database.begin_authoritative + database + ~expected:(Types.sync_view_token (view ())) + batch) + in + let request = Option.get request in + let decrypted = request, get (unprotect request) in + let result = + Database.apply_authoritative database prepared ~decrypted:(Some decrypted) + in + (match result with + | Error (Types.Authoritative_decode_failed message) -> + Alcotest.failf "Synthetic echo decode: %s" message + | _ -> ()); + result + in + let durable_state () = + let sqlite = Sqlite3.db_open ~mode:`READONLY path in + Fun.protect + ~finally:(fun () -> ignore (Sqlite3.db_close sqlite)) + (fun () -> + let tables = ref [] in + Sqlite3.Rc.check + (Sqlite3.exec + sqlite + ~cb:(fun row _ -> tables := Option.get row.(0) :: !tables) + "SELECT name FROM sqlite_master WHERE type='table' ORDER BY name"); + List.map + (fun table -> + let rows = ref [] in + Sqlite3.Rc.check + (Sqlite3.exec + sqlite + ~cb:(fun row _ -> rows := Array.to_list row :: !rows) + ("SELECT * FROM \"" + ^ String.concat "\"\"" (String.split_on_char '"' table) + ^ "\"")); + table, List.sort compare !rows) + !tables) + in + let before = state () + and durable_before = durable_state () in + let reject label operations = + (match apply operations with + | Error (Types.Authoritative_integrity_failure _) -> () + | _ -> Alcotest.fail ("AES echo must reject " ^ label)); + Alcotest.(check bool) + (label ^ " leaves logical view unchanged") + true + (state () = before); + Alcotest.(check bool) + (label ^ " leaves every durable table unchanged") + true + (durable_state () = durable_before) + in + if not duplicate + then ( + reject + "changed UUID" + (List.map + (function + | J.Array [ op; entity; J.Keyword "block/uuid"; J.Uuid value ] + when value = Graph.Uuid.to_string child -> + J.Array + [ op + ; entity + ; J.Keyword "block/uuid" + ; J.Uuid (Graph.Uuid.to_string (uuid 7794)) + ] + | operation -> operation) + operations); + reject + "changed reference" + (List.map + (function + | J.Array [ op; entity; J.Keyword "block/parent"; _ ] -> + J.Array + [ op + ; entity + ; J.Keyword "block/parent" + ; J.Array [ J.Keyword "block/uuid"; J.Uuid (Graph.Uuid.to_string child) ] + ] + | operation -> operation) + operations); + reject + "changed ident reference" + (List.map + (function + | J.Array [ op; entity; J.Keyword "block/tags"; _ ] -> + J.Array + [ op + ; entity + ; J.Keyword "block/tags" + ; J.Array [ J.Keyword "db/ident"; J.Keyword "logseq.kv/graph-remote?" ] + ] + | operation -> operation) + operations); + let title = + List.find + (function + | J.Array [ _; _; J.Keyword "block/title"; _ ] -> true + | _ -> false) + operations + in + reject "missing fact" (List.filter (fun operation -> operation <> title) operations); + let entity = + match title with + | J.Array [ _; entity; _; _ ] -> entity + | _ -> assert false + in + reject + "extra fact" + (operations + @ [ J.Array [ J.Keyword "db/add"; entity; J.Keyword "block/updated-at"; J.Int 123 ] + ]); + let repeated = + match title with + | J.Array [ op; _; attr; value ] -> + J.Array [ op; J.String "synthetic-extra-owner"; attr; value ] + | _ -> assert false + in + reject "changed multiplicity" (operations @ [ repeated ])); + let operations = if duplicate then operations @ operations else operations in + (match ok (apply operations) with + | Database.Authoritative_applied _ -> () + | _ -> Alcotest.fail "AES reordered append was deferred"); + Alcotest.(check int64) + "AES echo advances checkpoint" + 4L + (Types.Server_cursor.to_int64 (Types.sync_view_checkpoint (view ()))); + Alcotest.(check int) + "AES echo retires accepted owner" + 0 + (List.length (Types.sync_view_submissions (view ()))); + let actual = + ok + (Database.inspect_snapshot_storage + ~check_current:(fun () -> ()) + ~scratch_bytes:33554432L + ~encrypted:true + ~path) + in + Alcotest.(check (option bool)) + "AES echo actual durable checksum" + (Some true) + (ok (Database.check_checkpoint_checksum database ~through ~expected:actual)); + let wrong = ok (Types.Checksum.of_string "checksum:v1:ffffffffffffffff") in + Alcotest.(check (option bool)) + "AES echo wrong checksum rejected" + (Some false) + (ok (Database.check_checkpoint_checksum database ~through ~expected:wrong)); + let snapshot = ok (Database.current_snapshot database) in + Fun.protect + ~finally:(fun () -> Database.release_snapshot snapshot) + (fun () -> + match ok (Database.get_outline_blocks snapshot [ root; child ]) with + | [ Types.Present_block { value = first; _ } + ; Types.Present_block { value = second; _ } + ] -> + Alcotest.(check bool) + "AES echo stable parent/page references" + true + (first.block.parent = page + && second.block.parent = root + && first.block.page = page + && second.block.page = page); + Alcotest.(check string) + "AES root authenticated title" + "Synthetic AES root" + first.block.title; + Alcotest.(check string) + "AES child authenticated title" + "Synthetic AES child" + second.block.title + | _ -> Alcotest.fail "AES echo lost the complete tree") +;; + +let encrypted_snapshot_activation ?echo () = + let module Crypto = Logseq_sync_crypto.Core in + let module Owner = Logseq_sync_execute_owners.Database_owner in + let module Database = Logseq_overlay_db.Database in + let module Types = Logseq_overlay_db.Types in + let module Aes = Mirage_crypto.AES.GCM in + let backend : Crypto.backend = + { unlock = (fun ~identity:_ ~wrapped_graph_key:_ -> Ok (String.make 32 'k')) + ; crypto = + { encrypt_aes_gcm = + (fun ~key ~plaintext -> + let iv = String.make 12 'i' in + Ok + (iv, Aes.authenticate_encrypt ~key:(Aes.of_secret key) ~nonce:iv plaintext)) + ; decrypt_aes_gcm = + (fun ~key ~iv ~ciphertext -> + match + Aes.authenticate_decrypt ~key:(Aes.of_secret key) ~nonce:iv ciphertext + with + | Some plaintext -> Ok plaintext + | None -> Error "Synthetic authentication failure") + } + } + in + let crypto_limits = limits_with_items 2 in + let crypto = Crypto.create ~identity ~limits:crypto_limits ~encrypted:true ~backend in + let run operation = + Crypto.run crypto ~identity ~check_current:(fun () -> Ok ()) operation + in + Fun.protect + ~finally:(fun () -> Crypto.retire crypto) + (fun () -> + ignore (get (run (D.Unlock (get (D.unlock_input ~wrapped_graph_key:"synthetic"))))); + let titles = + List.init 5 (fun index -> Printf.sprintf "Synthetic title %d" index) + in + let protected_titles = + List.map + (fun title -> + let plaintext = + Transit_native.Transit.Json.to_string (Transit_core.Json.String title) + in + let iv, ciphertext = + List.hd + (Rrbvec.to_list + (get (run (D.Encrypt_values (Rrbvec.singleton plaintext))))) + in + Transit_native.Transit.Json.to_string + (Transit_core.Json.Array [ Binary iv; Binary ciphertext ])) + titles + in + let wire, rows, _ = remote_graph_fixture ~protected_titles () in + with_owner ~encrypted:true ~wire ~rows (fun root owner _ _ _ -> + Eio.Switch.run (fun sw -> + let snapshot = get (acquire owner (fun _ -> Ok ())) in + let source = get (S.source owner snapshot) in + let dependencies = + Result.get_ok + (Database.dependencies + ~epoch_ms:(fun () -> 0L) + ~monotonic_ns:Mtime_clock.elapsed_ns + ~limits: + { Types.response_budget_bytes = 1048576 + ; outbox_max_records = 32 + ; outbox_max_bytes = 4194304 + ; change_max_items = 1024 + ; change_max_bytes = 1048576 + ; dispatcher_capacity = 32 + ; wire_batch_max_bytes = 1048576 + }) + in + let inspection = + Result.get_ok + (Database.inspect_mirror + ~application_support_directory:root + ~graph_id: + (Result.get_ok + (Logseq_db_types.Graph_types.Uuid.of_string identity.graph))) + in + let binding = + Owner.activation_owner + ~sw + ~dependencies + ~inspection + ~graph_name:"Encrypted synthetic" + in + let batches = ref [] in + let unprotect request = + batches + := List.length (Database.unprotection_ciphertexts request) :: !batches; + run (D.Unprotect request) + in + (match + Owner.activate_snapshot + binding + ~check_current:(fun () -> Ok ()) + { snapshot + ; path = source.path + ; expected_rows = source.expected_rows + ; maximum_crypto_items = 1024 + ; maximum_crypto_bytes = crypto_limits.maximum_bytes + ; unprotect + } + with + | Error (D.Capacity_exceeded "Crypto items") -> () + | _ -> + Alcotest.fail + "unbounded snapshot must reproduce the crypto capacity rejection"); + Alcotest.(check bool) + "rejected activation stays absent" + true + (get (Owner.database_if_open binding) = None); + batches := []; + ignore + (get + (Owner.activate_snapshot + binding + ~check_current:(fun () -> Ok ()) + { snapshot + ; path = source.path + ; expected_rows = source.expected_rows + ; maximum_crypto_items = crypto_limits.maximum_items + ; maximum_crypto_bytes = crypto_limits.maximum_bytes + ; unprotect = + (fun request -> + batches + := List.length (Database.unprotection_ciphertexts request) + :: !batches; + run (D.Unprotect request)) + })); + let database = get (Owner.database binding) in + Fun.protect + ~finally:(fun () -> ignore (Database.close database)) + (fun () -> + Alcotest.(check (list int)) "bounded crypto batches" [ 1; 2; 2 ] !batches; + let expected = + Result.get_ok (Types.Checksum.of_string snapshot.checksum) + in + Alcotest.(check (option bool)) + "actual authoritative checksum" + (Some true) + (Result.get_ok + (Database.check_checkpoint_checksum + database + ~through:cursor + ~expected)); + let wrong = + Result.get_ok (Types.Checksum.of_string "checksum:v1:ffffffffffffffff") + in + Alcotest.(check (option bool)) + "same cursor rejects wrong checksum" + (Some false) + (Result.get_ok + (Database.check_checkpoint_checksum + database + ~through:cursor + ~expected:wrong)); + Alcotest.(check (option bool)) + "different cursor has no checksum comparison" + None + (Result.get_ok + (Database.check_checkpoint_checksum + database + ~through:(Result.get_ok (Types.Server_cursor.of_int64 4L)) + ~expected:wrong)); + let view = Result.get_ok (Database.inspect_sync database) in + Alcotest.(check int64) + "activated checkpoint" + 3L + (Types.Server_cursor.to_int64 (Types.sync_view_checkpoint view)); + Option.iter + (fun duplicate -> + let echo_crypto = + Crypto.create ~identity ~limits ~encrypted:true ~backend + in + let run operation = + Crypto.run + echo_crypto + ~identity + ~check_current:(fun () -> Ok ()) + operation + in + Fun.protect + ~finally:(fun () -> Crypto.retire echo_crypto) + (fun () -> + ignore + (get + (run + (D.Unlock + (get (D.unlock_input ~wrapped_graph_key:"synthetic"))))); + protected_append_echo + database + ~path: + (Filename.concat + root + ("logseq-db-worker/synced-graphs/" + ^ identity.graph + ^ "/db.sqlite")) + ~protect:(fun request -> run (D.Protect request)) + ~unprotect:(fun request -> run (D.Unprotect request)) + ~duplicate)) + echo)))) +;; + +let native_service_bootstrap ?(checksum_mode = `Valid) () = + let module App = Journal_sync_lifecycle.Core in + let module X = Execute.Core in + let module Database = Logseq_overlay_db.Database in + let module Types = Logseq_overlay_db.Types in + let wire, rows, checksum = remote_graph_fixture () in + let metadata = + wire_response + "Content-Type: application/json\r\nContent-Length: {{LENGTH}}\r\n" + "{\"ok\":true,\"url\":\"{{ORIGIN}}/snapshot\",\"t\":3}" + in + let artifact = + wire_response + (Printf.sprintf "Content-Length: {{LENGTH}}\r\nX-Snapshot-Row-Count: %d\r\n" rows) + wire + in + let module Protocol = Logseq_sync_protocol.Sync_protocol in + let wire_checksum = Types.Checksum.to_string checksum in + let wire_checksum = String.sub wire_checksum (String.length "checksum:v1:") 16 in + let hello = + Result.get_ok + (Protocol.encode_server_message + (Protocol.Server.Hello + { t = cursor + ; checksum = + (match checksum_mode with + | `Valid -> Some wire_checksum + | `Invalid -> + Some + (if wire_checksum = "ffffffffffffffff" + then "0000000000000000" + else "ffffffffffffffff") + | `Missing -> None) + })) + in + let frame = + String.make 1 (Char.chr 0x81) ^ String.make 1 (Char.chr (String.length hello)) ^ hello + in + let socket = + match Tls_peer.case ~ws:true "" with + | `Assoc fields -> + `Assoc + (("parallel", `Bool true) + :: ("lifetime", `Int 10) + :: ("reply_text", `String (Tls_peer.hex frame)) + :: fields) + | _ -> assert false + in + let report = + Tls_peer.with_peer + [ socket; Tls_peer.case ~eof:true metadata; Tls_peer.case ~eof:true artifact ] + (fun ~environment ~sw ~support ~origin -> + let identity = + get + (D.identity + ~origin:(Uri.to_string origin) + ~account:"synthetic" + ~graph:identity.graph + ~account_generation:0L + ~graph_generation:0L) + in + let root = Filename.concat support "application" in + Unix.mkdir root 0o700; + let dependencies = + Result.get_ok + (Database.dependencies + ~epoch_ms:(fun () -> 0L) + ~monotonic_ns:Mtime_clock.elapsed_ns + ~limits: + { Types.response_budget_bytes = 1048576 + ; outbox_max_records = 32 + ; outbox_max_bytes = 4194304 + ; change_max_items = 1024 + ; change_max_bytes = 1048576 + ; dispatcher_capacity = 32 + ; wire_batch_max_bytes = 1048576 + }) + in + let sqlite = Sqlite3.db_open ":memory:" in + Fun.protect + ~finally:(fun () -> + Alcotest.(check bool) + "App upload store released" + true + (Sqlite3.db_close sqlite)) + (fun () -> + Result.get_ok + (Logseq_db_storage.Asset_upload_store.initialize_database sqlite); + let retry = + get + (D.retry_policy + ~initial_seconds:0.01 + ~maximum_seconds:0.01 + ~multiplier:1. + ~maximum_attempts:0) + in + let config : Logseq_sync_sync_service.Core.config = + { identity + ; limits + ; network = D.Open { authorization_epoch = 1L } + ; policy = + get + (D.policy_config + ~sync_retry:retry + ~asset_retry:retry + ~no_progress_seconds:0.01) + } + in + let errors = ref [] in + let observed_checkpoint = ref None in + let returned = ref false in + let service = + get + (App.start_owned_native + ~sw + ~clock:(Eio.Stdenv.mono_clock environment) + ~network:(Eio.Stdenv.net environment) + ~authenticator:(fun ?ip:_ ~host:_ _ -> Ok None) + ~config + ~deadlines: + { X.setup_seconds = 2. + ; tls_handshake_seconds = 2. + ; http_response_seconds = 2. + ; download_seconds = 2. + } + ~liveness:X.Peer_close_only + ~cleanup_seconds:2. + ~root + ~dependencies + ~graph_name:"Synthetic native bootstrap" + ~upload_store:(Logseq_sync_worker_owner.Core.store sqlite) + ~namespace:(String.make 64 'a') + ~encrypted:false + ~token:(fun ~epoch:_ ~refresh:_ -> Ok "synthetic-fixture-token") + ~on_change:(fun status -> + Alcotest.(check bool) + "startup callback follows return" + true + !returned; + let observation = + status.Logseq_sync_sync_service.Core.observation + in + observed_checkpoint := observation.state.checkpoint; + match observation.status with + | D.Service_blocked error -> errors := error :: !errors + | _ -> ()) + ~on_error:(fun error -> errors := error :: !errors)) + in + returned := true; + Fun.protect + ~finally:(fun () -> App.close service) + (fun () -> + if checksum_mode = `Invalid + then ( + Tls_peer.wait (Eio.Stdenv.clock environment) (fun () -> + !errors <> [] + || Option.is_some + (get (X.database_if_open (App.database_owner service)))); + Alcotest.(check bool) + "checksum mismatch blocks before activation" + true + (!errors <> []); + Alcotest.(check bool) + "invalid snapshot remains absent" + true + (get (X.database_if_open (App.database_owner service)) = None)) + else ( + Tls_peer.wait (Eio.Stdenv.clock environment) (fun () -> + (match !errors with + | D.Activation_failure (Types.Snapshot_parse_error message) :: _ + -> Alcotest.fail ("Activation parse: " ^ message) + | D.Activation_failure Types.Snapshot_limit_exceeded :: _ -> + Alcotest.fail "Activation limit exceeded" + | D.Activation_failure + (Types.Snapshot_commit_persistence_failed message) + :: _ -> Alcotest.fail ("Activation persistence: " ^ message) + | error :: _ -> Alcotest.fail (D.error_message error) + | [] -> ()); + Option.is_some !observed_checkpoint); + let database = get (X.database (App.database_owner service)) in + let view = Result.get_ok (Database.inspect_sync database) in + Alcotest.(check int64) + "native service activates exact server checkpoint" + 3L + (Types.sync_view_checkpoint view |> Types.Server_cursor.to_int64); + Alcotest.(check int) + "bootstrap subscription errors" + 0 + (List.length !errors))))) + in + check_retired report +;; + +let native_http ~download ~success ~eof wire () = + let report = + Tls_peer.with_peer + [ Tls_peer.case ~eof wire ] + (fun ~environment ~sw ~support ~origin -> + let http = native_client ~environment ~sw () in + let destination = Filename.concat support "artifact" in + let request token = + H. + { operation = Get + ; uri = origin + ; headers = H.authorization (Option.get token) + ; maximum_response_bytes = 16 + ; expected_content_type = Asset_binary + } + in + let result = + if download + then + C.download + http + ~scope + ~epoch:1L + ~authenticated:true + ~destination + ~maximum_bytes:16 + request + else C.perform http ~scope ~epoch:1L ~authenticated:true request + in + Alcotest.(check bool) "HTTP outcome" success (Result.is_ok result); + if not success + then + expect + (function + | D.Protocol_failure _ -> true + | _ -> false) + result; + if download + then ( + Alcotest.(check bool) + "publish only complete body" + success + (Sys.file_exists destination); + Alcotest.(check bool) + "no partial files" + false + (Array.exists + (fun name -> Filename.check_suffix name ".partial") + (Sys.readdir support)))) + in + check_retired report +;; + +let native_assets () = + let report = + Tls_peer.with_peer + [ Tls_peer.case (wire_response "Content-Length: 4\r\n" "wire") + ; Tls_peer.case (wire_response "Content-Length: 0\r\n" "") + ] + (fun ~environment ~sw ~support:_ ~origin -> + let http = native_client ~environment ~sw () in + let identity = + get + (D.identity + ~origin:(Uri.to_string origin) + ~account:"synthetic" + ~graph:identity.graph + ~account_generation:0L + ~graph_generation:0L) + in + let asset = + Result.get_ok + (Logseq_db_types.Graph_types.Uuid.of_string + "11111111-1111-4111-8111-111111111111") + in + let version = + Result.get_ok + (Logseq_db_types.Asset_descriptor.version + ~checksum:(String.make 64 'a') + ~file_type:"png") + in + Alcotest.(check (option string)) + "GET preserves protected wire" + (Some "wire") + (get + (C.get_asset + http + ~scope + ~epoch:1L + ~identity + ~asset + ~version + ~maximum_wire_bytes:16)); + get + (C.put_asset + http + ~scope + ~epoch:1L + ~identity + ~asset + ~version + ~wire:"protected-wire")) + in + check_retired report; + let open Yojson.Basic.Util in + let requests = + report |> to_list |> List.map (fun row -> row |> member "request" |> to_string) + in + let decode hex = + String.init + (String.length hex / 2) + (fun index -> Char.chr (int_of_string ("0x" ^ String.sub hex (index * 2) 2))) + in + let put = decode (List.nth requests 1) in + Alcotest.(check bool) + "production PUT method" + true + (String.starts_with ~prefix:"PUT /assets/" put); + Alcotest.(check bool) + "PUT exact frozen wire" + true + (String.ends_with ~suffix:"protected-wire" put) +;; + +let native_cancel () = + let case = + match Tls_peer.case (wire_response "Content-Length: 1\r\n" "x") with + | `Assoc fields -> `Assoc (("body_delay", `Float 0.2) :: fields) + | _ -> assert false + in + let report = + Tls_peer.with_peer [ case ] (fun ~environment ~sw ~support ~origin -> + let http = native_client ~environment ~sw ~download_seconds:0.05 () in + let destination = Filename.concat support "artifact" in + expect + (function + | D.Network_failure _ -> true + | _ -> false) + (C.download + http + ~scope + ~epoch:1L + ~authenticated:false + ~destination + ~maximum_bytes:16 + (fun _ -> H.artifact ~base_url:origin ~uri:origin ~token:None)); + Alcotest.(check bool) + "deadline leaves no artifact" + false + (Sys.file_exists destination)) + in + check_retired report +;; + +let native_response_deadline () = + let case = + match Tls_peer.case (wire_response "Content-Length: 1\r\n" "x") with + | `Assoc fields -> `Assoc (("response_delay", `Float 0.2) :: fields) + | _ -> assert false + in + let report = + Tls_peer.with_peer [ case ] (fun ~environment ~sw ~support:_ ~origin -> + let http = native_client ~environment ~sw ~http_response_seconds:0.05 () in + expect + (function + | D.Network_failure _ -> true + | _ -> false) + (C.perform http ~scope ~epoch:1L ~authenticated:false (fun _ -> + H.artifact ~base_url:origin ~uri:origin ~token:None))) + in + check_retired report +;; + +let scaled_mono clock scale = + let started = Eio.Time.Mono.now clock |> Mtime.to_uint64_ns in + let module Clock = struct + type t = unit + type time = Mtime.t + + let now () = + let elapsed = Int64.sub (Mtime.to_uint64_ns (Eio.Time.Mono.now clock)) started in + Mtime.of_uint64_ns (Int64.of_float (Int64.to_float elapsed *. scale)) + ;; + + let sleep_until () deadline = + let elapsed = Int64.to_float (Mtime.to_uint64_ns deadline) /. scale in + Eio.Time.Mono.sleep_until + clock + (Mtime.of_uint64_ns (Int64.add started (Int64.of_float elapsed))) + ;; + end + in + Eio.Resource.T ((), Eio.Time.Pi.clock (module Clock)) +;; + +let native_phase_budget phase ~expires () = + let delay = if phase = "body_delay" then 1.6 else 0.4 in + let fields = + match Tls_peer.case (wire_response "Content-Length: 1\r\n" "x") with + | `Assoc fields -> fields + | _ -> assert false + in + let cases = + if expires && phase = "tls_delay" + then ( + let case = `Assoc (("stall_tls", `Bool true) :: fields) in + [ case; case ]) + else [ `Assoc ((phase, `Float delay) :: fields) ] + in + let report = + Tls_peer.with_peer cases (fun ~environment ~sw ~support ~origin -> + let deadlines : Execute.Core.network_deadlines = + { setup_seconds = 60. + ; tls_handshake_seconds = 60. + ; http_response_seconds = 60. + ; download_seconds = 600. + } + in + let deadlines = + if not expires + then deadlines + else ( + match phase with + | "tls_delay" -> { deadlines with tls_handshake_seconds = 3. } + | "response_delay" -> { deadlines with http_response_seconds = 3. } + | _ -> { deadlines with download_seconds = 3. }) + in + let http = + C.create + ~backend: + (C.native_backend + ~sw + ~clock:(scaled_mono (Eio.Stdenv.mono_clock environment) 100.) + ~network:(Eio.Stdenv.net environment) + ~authenticator:(fun ?ip:_ ~host:_ _ -> Ok None) + ~deadlines) + ~token:(fun ~epoch:_ ~refresh:_ -> Ok "synthetic") + in + let request _ = H.artifact ~base_url:origin ~uri:origin ~token:None in + let result = + if phase = "body_delay" + then + C.download + http + ~scope + ~epoch:1L + ~authenticated:false + ~destination:(Filename.concat support "artifact") + ~maximum_bytes:16 + request + else C.perform http ~scope ~epoch:1L ~authenticated:false request + in + if expires + then + expect + (function + | D.Network_failure _ -> true + | _ -> false) + result + else ignore (get result)) + in + check_retired report +;; + +let native_download_budget_starts_at_body () = + let case = + match Tls_peer.case (wire_response "Content-Length: 1\r\n" "x") with + | `Assoc fields -> `Assoc (("response_delay", `Float 0.2) :: fields) + | _ -> assert false + in + let report = + Tls_peer.with_peer [ case ] (fun ~environment ~sw ~support ~origin -> + let http = + native_client ~environment ~sw ~http_response_seconds:1. ~download_seconds:0.1 () + in + let destination = Filename.concat support "artifact" in + ignore + (get + (C.download + http + ~scope + ~epoch:1L + ~authenticated:false + ~destination + ~maximum_bytes:16 + (fun _ -> H.artifact ~base_url:origin ~uri:origin ~token:None))); + Alcotest.(check bool) + "body gets its own budget after headers" + true + (Sys.file_exists destination)) + in + check_retired report +;; + +let native_download_body_budget () = + let case = + match Tls_peer.case (wire_response "Content-Length: 1\r\n" "x") with + | `Assoc fields -> `Assoc (("body_delay", `Float 0.2) :: fields) + | _ -> assert false + in + let report = + Tls_peer.with_peer [ case ] (fun ~environment ~sw ~support ~origin -> + let destination = Filename.concat support "artifact" in + let result = + N.download_with_deadlines + ~setup_seconds:1. + ~tls_handshake_seconds:1. + ~http_response_seconds:1. + ~download_seconds:0.03 + ~sw + ~clock: + (Logseq_sync_native_transport.Transport_time.monotonic + (Eio.Stdenv.mono_clock environment)) + ~network:(Eio.Stdenv.net environment) + ~authenticator:(fun ?ip:_ ~host:_ _ -> Ok None) + ~request:(H.artifact ~base_url:origin ~uri:origin ~token:None) + ~destination + ~maximum_bytes:16 + ~on_progress:(fun _ -> ()) + in + Alcotest.(check bool) + "native body timer expires without outer timeout" + true + (match result with + | Error (N.Network _) -> true + | _ -> false); + Alcotest.(check bool) + "body timeout prevents publication" + false + (Sys.file_exists destination)) + in + check_retired report +;; + +let native_setup_configured_budget () = + let report = + Tls_peer.with_peer + [ Tls_peer.case (wire_response "Content-Length: 1\r\n" "x") ] + (fun ~environment ~sw ~support:_ ~origin -> + let underlying = Eio.Stdenv.net environment in + let attempts = ref 0 in + let module Network = struct + type t = unit + + type tag = + [ `Generic + | `Unix + ] + + let connect () ~sw address = + incr attempts; + Eio.Time.sleep (Eio.Stdenv.clock environment) 0.2; + if !attempts = 1 then raise (Unix.Unix_error (Unix.ECONNREFUSED, "", "")); + Eio.Net.connect ~sw underlying address + ;; + + let listen () ~reuse_addr:_ ~reuse_port:_ ~backlog:_ ~sw:_ _ = assert false + let datagram_socket () ~reuse_addr:_ ~reuse_port:_ ~sw:_ _ = assert false + let getaddrinfo () ~service:_ _ = assert false + let getnameinfo () _ = assert false + end + in + let network = Eio.Resource.T ((), Eio.Net.Pi.network (module Network)) in + let deadlines : Execute.Core.network_deadlines = + { setup_seconds = 60. + ; tls_handshake_seconds = 60. + ; http_response_seconds = 60. + ; download_seconds = 600. + } + in + let http = + C.create + ~backend: + (C.native_backend + ~sw + ~clock:(scaled_mono (Eio.Stdenv.mono_clock environment) 100.) + ~network + ~authenticator:(fun ?ip:_ ~host:_ _ -> Ok None) + ~deadlines) + ~token:(fun ~epoch:_ ~refresh:_ -> Ok "synthetic") + in + ignore + (get + (C.perform http ~scope ~epoch:1L ~authenticated:false (fun _ -> + H.artifact ~base_url:origin ~uri:origin ~token:None)))) + in + check_retired report +;; + +let native_setup_budget () = + Eio_main.run (fun environment -> + Eio.Switch.run (fun sw -> + let module Network = struct + type t = unit + type tag = [ `Generic ] + + let connect () ~sw:_ _ = + Eio.Time.sleep (Eio.Stdenv.clock environment) 0.2; + assert false + ;; + + let listen () ~reuse_addr:_ ~reuse_port:_ ~backlog:_ ~sw:_ _ = assert false + let datagram_socket () ~reuse_addr:_ ~reuse_port:_ ~sw:_ _ = assert false + let getaddrinfo () ~service:_ _ = assert false + let getnameinfo () _ = assert false + end + in + let network = Eio.Resource.T ((), Eio.Net.Pi.network (module Network)) in + let deadlines : Execute.Core.network_deadlines = + { setup_seconds = 0.03 + ; tls_handshake_seconds = 1. + ; http_response_seconds = 1. + ; download_seconds = 3. + } + in + let http = + C.create + ~backend: + (C.native_backend + ~sw + ~clock:(Eio.Stdenv.mono_clock environment) + ~network + ~authenticator:(fun ?ip:_ ~host:_ _ -> Ok None) + ~deadlines) + ~token:(fun ~epoch:_ ~refresh:_ -> Ok "synthetic") + in + Eio.Time.with_timeout_exn (Eio.Stdenv.clock environment) 0.1 (fun () -> + expect + (function + | D.Network_failure _ -> true + | _ -> false) + (C.perform http ~scope ~epoch:1L ~authenticated:false (fun _ -> + H.artifact + ~base_url:(Uri.of_string "https://localhost:1") + ~uri:(Uri.of_string "https://localhost:1") + ~token:None))))) +;; + +let native_reject_auth () = + let report = + Tls_peer.with_peer + [ Tls_peer.case "" ] + (fun ~environment ~sw ~support:_ ~origin -> + let deadlines : Execute.Core.network_deadlines = + { setup_seconds = 1. + ; tls_handshake_seconds = 1. + ; http_response_seconds = 1. + ; download_seconds = 3. + } + in + let tokens = ref [] in + let http = + C.create + ~backend: + (C.native_backend + ~sw + ~clock:(Eio.Stdenv.mono_clock environment) + ~network:(Eio.Stdenv.net environment) + ~authenticator:(fun ?ip:_ ~host:_ _ -> + Error (`Msg "synthetic credential must not appear")) + ~deadlines) + ~token:(fun ~epoch:_ ~refresh -> + tokens := refresh :: !tokens; + Ok "synthetic") + in + let result = + C.perform http ~scope ~epoch:1L ~authenticated:true (fun _ -> + H.artifact ~base_url:origin ~uri:origin ~token:None) + in + expect + (function + | D.Authentication_failure message -> message = "TLS authentication failed" + | _ -> false) + result; + Alcotest.(check (list bool)) + "TLS authentication does not refresh" + [ false ] + !tokens) + in + check_retired report +;; + +let native_error_cause kind () = + let wire = + match kind with + | "content-type" -> + wire_response "Content-Type: credential-secret\r\nContent-Length: 1\r\n" "x" + | "crypto" -> wire_response "Content-Length: 2\r\n" "x" + | _ -> wire_response "Content-Length: 1\r\n" "x" + in + let case = + match Tls_peer.case wire with + | `Assoc fields when kind = "crypto" -> `Assoc (("corrupt_tls", `Bool true) :: fields) + | case -> case + in + let report = + Tls_peer.with_peer [ case ] (fun ~environment ~sw ~support ~origin -> + let http = native_client ~environment ~sw () in + let result = + C.download + http + ~scope + ~epoch:1L + ~authenticated:false + ~destination: + (Filename.concat + support + (if kind = "storage" then "missing/artifact" else "artifact")) + ~maximum_bytes:16 + (fun _ -> H.artifact ~base_url:origin ~uri:origin ~token:None) + in + expect + (function + | D.Protocol_failure message when kind = "content-type" -> + message = "Invalid HTTP response Content-Type" + | D.Storage_failure _ when kind = "storage" -> true + | D.Crypto_failure _ when kind = "crypto" -> true + | _ -> false) + result; + Alcotest.(check bool) + "no failed artifact publication" + false + (Sys.file_exists (Filename.concat support "artifact"))) + in + check_retired report +;; + +let () = + let tests = + [ ( "encrypted native snapshot activation respects crypto bounds" + , fun () -> encrypted_snapshot_activation () ) + ; ( "AES protected append reordered echo and six exact-payload rejections" + , fun () -> encrypted_snapshot_activation ~echo:false () ) + ; ( "AES protected append duplicate operations retain exact datoms" + , fun () -> encrypted_snapshot_activation ~echo:true () ) + ; ("App owned native absent mirror bootstrap", fun () -> native_service_bootstrap ()) + ; ( "native bootstrap rejects wrong Hello checksum" + , fun () -> native_service_bootstrap ~checksum_mode:`Invalid () ) + ; ( "native bootstrap accepts missing Hello checksum" + , fun () -> native_service_bootstrap ~checksum_mode:`Missing () ) + ; "encryption metadata rejects and cleans", encryption_failure + ; "actual framed artifact and original ledger", success + ; "metadata rejects before download", metadata_failure + ; "framing rejects and cleans", framing_failure + ; "row count rejects and cleans", row_failure + ; "handoff failure cleans", handoff_failure + ; "HTTP refresh once", auth + ; "HTTP status classification", http_status_policy + ; "native response phase deadline", native_response_deadline + ; "native setup phase deadline", native_setup_budget + ; "native setup configured budget", native_setup_configured_budget + ; "native TLS authentication cause", native_reject_auth + ; "native content type cause and redaction", native_error_cause "content-type" + ; "native storage cause", native_error_cause "storage" + ; "native crypto cause", native_error_cause "crypto" + ; "native TLS configured budget", native_phase_budget "tls_delay" ~expires:false + ; "native TLS phase expires", native_phase_budget "tls_delay" ~expires:true + ; ( "native response configured budget" + , native_phase_budget "response_delay" ~expires:false ) + ; "native response phase expires", native_phase_budget "response_delay" ~expires:true + ; "native download configured budget", native_phase_budget "body_delay" ~expires:false + ; "native download phase expires", native_phase_budget "body_delay" ~expires:true + ; "native download budget starts at body", native_download_budget_starts_at_body + ; "native body phase without outer timer", native_download_body_budget + ; "native asset GET and PUT", native_assets + ; "native deadline physically joins", native_cancel + ] + in + let tests = + tests + @ List.init 4 (fun layers -> + "native snapshot gzip layers " ^ string_of_int layers, native_snapshot layers) + in + let cases = + [ "fixed", true, false, wire_response "Content-Length: 1\r\n" "x" + ; ( "chunked" + , true + , false + , wire_response "Transfer-Encoding: chunked\r\n" "1\r\nx\r\n0\r\n\r\n" ) + ; "close delimited", true, true, wire_response "" "x" + ; "malformed status", false, true, "not HTTP\r\n\r\n" + ; "HTTP 400", false, false, "HTTP/1.1 400 Bad Request\r\nContent-Length: 0\r\n\r\n" + ; ( "HTTP 422" + , false + , false + , "HTTP/1.1 422 Unprocessable Content\r\nContent-Length: 0\r\n\r\n" ) + ; "truncated", false, true, wire_response "Content-Length: 2\r\n" "x" + ; ( "body bound" + , false + , false + , wire_response "Content-Length: 17\r\n" (String.make 17 'x') ) + ; ( "bad chunk" + , false + , true + , wire_response "Transfer-Encoding: chunked\r\n" "Z\r\nx\r\n" ) + ] + in + let tests = + tests + @ List.concat_map + (fun (name, success, eof, wire) -> + List.map + (fun download -> + ( (if download then "download " else "GET ") ^ name + , native_http ~download ~success ~eof wire )) + [ false; true ]) + cases + in + Alcotest.run + "snapshot producer" + [ ( "acquisition" + , List.map (fun (name, test) -> Alcotest.test_case name `Quick test) tests ) + ] +;; diff --git a/logseq_sync/test/production_transport/snapshot_rows.json b/logseq_sync/test/production_transport/snapshot_rows.json new file mode 100644 index 00000000..3c329b15 --- /dev/null +++ b/logseq_sync/test/production_transport/snapshot_rows.json @@ -0,0 +1,2030 @@ +{ + "formatVersion": 1, + "source": { + "logseqCommit": "4f21d068aed43bb2ea5823247cae73ecdd8d60f8", + "schema": "65.33" + }, + "tableSql": "CREATE TABLE kvs (addr INTEGER primary key, content TEXT, addresses JSON);", + "rows": [ + { + "addr": 0, + "content": "[\"^ \",\"~:schema\",[\"^ \",\"~:logseq.property.view/gallery-display-properties\",[\"^ \",\"~:db/index\",true,\"~:db/valueType\",\"~:db.type/ref\",\"~:db/cardinality\",\"~:db.cardinality/many\",\"~:db/ident\",\"^1\"],\"~i32\",\"~:block/created-at\",\"~i64\",\"~:logseq.property.pdf/hl-value\",\"~i96\",\"~:logseq.property.repeat/recur-unit.week\",\"~i128\",\"~:logseq.property.table/sized-columns\",\"~i160\",\"~:logseq.property.reaction/emoji-id\",\"~:logseq.property.view/gallery-card-height\",[\"^ \",\"^2\",true,\"^5\",\"~:db.cardinality/one\",\"^7\",\"^B\"],\"~:logseq.property.asset/align\",[\"^ \",\"^2\",true,\"^5\",\"^C\",\"^7\",\"^D\"],\"~:logseq.property.view/type.table\",[\"^ \",\"^7\",\"^E\"],\"~:logseq.property/ui-position\",[\"^ \",\"^2\",true,\"^5\",\"^C\",\"^7\",\"^F\"],\"~:file/created-at\",[\"^ \"],\"~:logseq.property.repeat/temporal-property\",[\"^ \",\"^2\",true,\"^3\",\"^4\",\"^5\",\"^C\",\"^7\",\"^H\"],\"~:logseq.property.user/name\",[\"^ \",\"^2\",true,\"^5\",\"^C\",\"^7\",\"^I\"],\"~i1\",\"~:logseq.class/Root\",\"~i33\",\"~:block/updated-at\",\"~i65\",\"~:logseq.property/order-list-type\",\"~i97\",\"~:logseq.property.repeat/recur-unit.month\",\"~:logseq.property.pdf/hl-image\",[\"^ \",\"^2\",true,\"^3\",\"^4\",\"^5\",\"^C\",\"^7\",\"^Q\"],\"~i129\",\"~:logseq.property.table/pinned-columns\",\"~i161\",\"~:logseq.property.reaction/target\",\"~:logseq.property.fsrs/state\",[\"^ \",\"^2\",true,\"^5\",\"^C\",\"^7\",\"^V\"],\"~:logseq.property.pdf/hl-color\",[\"^ \",\"^2\",true,\"^3\",\"^4\",\"^5\",\"^C\",\"^7\",\"^W\"],\"~:logseq.kv/graph-initial-schema-version\",[\"^ \",\"^7\",\"^X\"],\"~:block/tx-id\",[\"^ \"],\"~:logseq.property/value\",[\"^ \",\"^2\",true,\"^5\",\"^C\",\"^7\",\"^Z\"],\"^=\",[\"^ \",\"^7\",\"^=\"],\"~i2\",\"~:logseq.class/Tag\",\"~i34\",\"~:logseq.property.node/display-type\",\"~i66\",\"~:logseq.property.linked-references/includes\",\"~i98\",\"~:logseq.property.repeat/recur-unit.year\",\"~i130\",\"~:logseq.property/view-for\",\"~i162\",\"~:logseq.property.agent/session-id\",\"~:logseq.property.table/sorting\",[\"^ \",\"^2\",true,\"^5\",\"^C\",\"^7\",\"^1:\"],\"~:logseq.property/choice-exclusions\",[\"^ \",\"^2\",true,\"^3\",\"^4\",\"^5\",\"^6\",\"^7\",\"^1;\"],\"^19\",[\"^ \",\"^2\",true,\"^5\",\"^C\",\"^7\",\"^19\"],\"^S\",[\"^ \",\"^2\",true,\"^3\",\"^4\",\"^5\",\"^6\",\"^7\",\"^S\"],\"~:logseq.property.repeat/recur-unit.day\",[\"^ \",\"^7\",\"^1<\"],\"~i3\",\"~:logseq.class/Property\",\"~i35\",\"~:logseq.property/description\",\"~i67\",\"~:logseq.property.linked-references/excludes\",\"~i99\",\"~:logseq.property.repeat/repeated?\",\"~i131\",\"~:logseq.property.asset/type\",\"~:logseq.property.sync/large-title-object\",[\"^ \",\"^2\",true,\"^5\",\"^C\",\"^7\",\"^1F\"],\"~i4\",\"~:logseq.class/Page\",\"~i36\",\"~:logseq.property.code/lang\",\"~i68\",\"~:logseq.property.comments/blocks\",\"~i100\",\"~:logseq.property.repeat/repeat-type\",\"~i132\",\"~:logseq.property.asset/external-url\",\"~i164\",\"~:logseq.property/used-template\",\"~:logseq.property/deadline\",[\"^ \",\"^2\",true,\"^5\",\"^C\",\"^7\",\"^1R\"],\"~:logseq.property/type\",[\"^ \",\"^2\",true,\"^5\",\"^C\",\"^7\",\"^1S\"],\"^15\",[\"^ \",\"^7\",\"^15\"],\"~:file/content\",[\"^ \"],\"~i5\",\"~:logseq.class/Template\",\"~i37\",\"~:logseq.property/default-value\",\"~i69\",\"~:logseq.property.journal/title-format\",\"~i101\",\"~:logseq.property.repeat/repeat-type.dotted-plus\",\"~:logseq.property.view/sort-groups-desc?\",[\"^ \",\"^2\",true,\"^5\",\"^C\",\"^7\",\"^20\"],\"~i133\",\"~:logseq.property.asset/external-file-name\",\"~i165\",\"~:logseq.property/template-applied-to\",\"^?\",[\"^ \",\"^2\",true,\"^5\",\"^C\",\"^7\",\"^?\"],\"~:block/alias\",[\"^ \",\"^2\",true,\"^3\",\"^4\",\"^5\",\"^6\",\"^7\",\"^25\"],\"~:logseq.property.publish/published-url\",[\"^ \",\"^2\",true,\"^3\",\"^4\",\"^5\",\"^C\",\"^7\",\"^26\"],\"^1O\",[\"^ \",\"^2\",true,\"^5\",\"^C\",\"^7\",\"^1O\"],\"~i6\",\"~:logseq.kv/db-type\",\"~:logseq.property.user/avatar\",[\"^ \",\"^2\",true,\"^5\",\"^C\",\"^7\",\"^28\"],\"~i38\",\"~:logseq.property/scalar-default-value\",\"~i70\",\"~:logseq.property/choice-checkbox-state\",\"~i102\",\"~:logseq.property.repeat/repeat-type.plus\",\"~i134\",\"~:logseq.property.asset/size\",\"~:logseq.property/assignee\",[\"^ \",\"^2\",true,\"^3\",\"^4\",\"^5\",\"^6\",\"^7\",\"^2A\"],\"~i166\",\"^1F\",\"~:kv/value\",[\"^ \"],\"^2:\",[\"^ \",\"^2\",true,\"^5\",\"^C\",\"^7\",\"^2:\"],\"^1E\",[\"^ \",\"^2\",true,\"^5\",\"^C\",\"^7\",\"^1E\"],\"~:logseq.class/Asset\",[\"^ \",\"^7\",\"^2D\"],\"~:logseq.property.recycle/original-order\",[\"^ \",\"^2\",true,\"^5\",\"^C\",\"^7\",\"^2E\"],\"~:logseq.property.class/enable-bidirectional?\",[\"^ \",\"^2\",true,\"^5\",\"^C\",\"^7\",\"^2F\"],\"~:logseq.property/color.green\",[\"^ \",\"^7\",\"^2G\"],\"~i7\",\"~:logseq.kv/schema-version\",\"~i39\",\"~:logseq.property.class/extends\",\"~i71\",\"~:logseq.property/choice-classes\",\"~i103\",\"~:logseq.property.repeat/repeat-type.double-plus\",\"~i135\",\"~:logseq.property.asset/width\",\"~i167\",\"~:logseq.class/Journal\",\"^13\",[\"^ \",\"^2\",true,\"^3\",\"^4\",\"^5\",\"^6\",\"^7\",\"^13\"],\"~:logseq.property.repeat/recur-unit\",[\"^ \",\"^2\",true,\"^3\",\"^4\",\"^5\",\"^C\",\"^7\",\"^2S\"],\"~:logseq.property/checkbox-display-properties\",[\"^ \",\"^2\",true,\"^3\",\"^4\",\"^5\",\"^6\",\"^7\",\"^2T\"],\"~:block/link\",[\"^ \",\"^2\",true,\"^3\",\"^4\",\"^5\",\"^C\",\"^7\",\"^2U\"],\"~:logseq.property.view/type\",[\"^ \",\"^2\",true,\"^3\",\"^4\",\"^5\",\"^C\",\"^7\",\"^2V\"],\"~:logseq.class/Quote-block\",[\"^ \",\"^7\",\"^2W\"],\"^2N\",[\"^ \",\"^7\",\"^2N\"],\"~:logseq.property/heading\",[\"^ \",\"^2\",true,\"^5\",\"^C\",\"^7\",\"^2X\"],\"~i8\",\"^X\",\"~i72\",\"^1;\",\"~i104\",\"^H\",\"~:block/uuid\",[\"^ \",\"~:db/unique\",\"~:db.unique/identity\"],\"~i136\",\"~:logseq.property.asset/height\",\"~i168\",\"~:logseq.class/Whiteboard\",\"~:logseq.property/color.purple\",[\"^ \",\"^7\",\"^36\"],\"^22\",[\"^ \",\"^2\",true,\"^5\",\"^C\",\"^7\",\"^22\"],\"~:logseq.property.table/hidden-columns\",[\"^ \",\"^2\",true,\"^5\",\"^6\",\"^7\",\"^37\"],\"^1=\",[\"^ \",\"^7\",\"^1=\"],\"~i9\",\"~:logseq.kv/graph-created-at\",\"~i41\",\"~:logseq.property.class/properties\",\"~i73\",\"^2T\",\"~i105\",\"~:logseq.property.repeat/checked-property\",\"^1G\",[\"^ \",\"^7\",\"^1G\"],\"~i137\",\"~:logseq.property.asset/checksum\",\"~i169\",\"~:logseq.class/Task\",\"~:logseq.kv/local-graph-uuid\",[\"^ \",\"^7\",\"^3B\"],\"^L\",[\"^ \",\"^2\",true,\"^5\",\"^C\",\"^7\",\"^L\"],\"~:logseq.property/color.blue\",[\"^ \",\"^7\",\"^3C\"],\"~i10\",\"~:logseq.property/empty-placeholder\",\"~:logseq.class/Comment\",[\"^ \",\"^7\",\"^3F\"],\"~:logseq.property.view/gallery-asset-property\",[\"^ \",\"^2\",true,\"^3\",\"^4\",\"^5\",\"^C\",\"^7\",\"^3G\"],\"~i42\",\"~:logseq.property.class/bidirectional-property-title\",\"~i74\",\"~:logseq.property/status\",\"~i106\",\"^2A\",\"~i138\",\"~:logseq.property.asset/last-visit-page\",\"~i170\",\"~:logseq.class/Comments\",\"~:logseq.property.view/type.list\",[\"^ \",\"^7\",\"^3Q\"],\"^U\",[\"^ \",\"^2\",true,\"^3\",\"^4\",\"^5\",\"^C\",\"^7\",\"^U\"],\"~i11\",\"^3B\",\"~:file/size\",[\"^ \"],\"~i43\",\"^2F\",\"^3K\",[\"^ \",\"^2\",true,\"^3\",\"^4\",\"^5\",\"^C\",\"^7\",\"^3K\"],\"~i75\",\"~:logseq.property/status.backlog\",\"^2J\",[\"^ \",\"^2\",true,\"^3\",\"^4\",\"^5\",\"^6\",\"^7\",\"^2J\"],\"~i107\",\"~:logseq.property/icon\",\"~i139\",\"~:logseq.property.asset/remote-metadata\",\"~:logseq.property/hide-empty-value\",[\"^ \",\"^2\",true,\"^5\",\"^C\",\"^7\",\"^3[\"],\"~i171\",\"^3F\",\"~:logseq.property.view/sort-groups-by-property\",[\"^ \",\"^2\",true,\"^3\",\"^4\",\"^5\",\"^C\",\"^7\",\"^41\"],\"~:block/refs\",[\"^ \",\"^2\",true,\"^3\",\"^4\",\"^5\",\"^6\",\"^7\",\"^42\"],\"~:logseq.class/Card\",[\"^ \",\"^7\",\"^43\"],\"^A\",[\"^ \",\"^2\",true,\"^5\",\"^C\",\"^7\",\"^A\"],\"~i12\",\"^1S\",\"~i76\",\"~:logseq.property/status.todo\",\"~i108\",\"~:logseq.property/publishing-public?\",\"~i172\",\"~:logseq.class/Query\",\"~:logseq.property.view/group-by-property\",[\"^ \",\"^2\",true,\"^3\",\"^4\",\"^5\",\"^C\",\"^7\",\"^4;\"],\"~:logseq.property/view-context\",[\"^ \",\"^2\",true,\"^5\",\"^C\",\"^7\",\"^4<\"],\"^2H\",[\"^ \",\"^7\",\"^2H\"],\"~:logseq.property.pdf/hl-page\",[\"^ \",\"^2\",true,\"^5\",\"^C\",\"^7\",\"^4=\"],\"~i13\",\"~:logseq.property/built-in?\",\"~i45\",\"^3[\",\"~:block/closed-value-property\",[\"^ \",\"^2\",true,\"^3\",\"^4\",\"^5\",\"^6\",\"^7\",\"^4A\"],\"~i77\",\"~:logseq.property/status.doing\",\"~i109\",\"^26\",\"^3?\",[\"^ \",\"^2\",true,\"^5\",\"^C\",\"^7\",\"^3?\"],\"^48\",[\"^ \",\"^2\",true,\"^5\",\"^C\",\"^7\",\"^48\"],\"~i141\",\"~:logseq.property.asset/resize-metadata\",\"^1K\",[\"^ \",\"^2\",true,\"^3\",\"^4\",\"^5\",\"^6\",\"^7\",\"^1K\"],\"~i173\",\"^43\",\"^11\",[\"^ \",\"^2\",true,\"^5\",\"^C\",\"^7\",\"^11\"],\"^3V\",[\"^ \",\"^7\",\"^3V\"],\"~i14\",\"~:logseq.property/hide?\",\"~:file/last-modified-at\",[\"^ \"],\"~i78\",\"~:logseq.property/status.in-review\",\"~i110\",\"~:logseq.property/exclude-from-graph-view\",\"~i142\",\"^D\",\"~i174\",\"~:logseq.class/Cards\",\"^3:\",[\"^ \",\"^2\",true,\"^3\",\"^4\",\"^5\",\"^6\",\"^7\",\"^3:\"],\"^1[\",[\"^ \",\"^7\",\"^1[\"],\"^1Q\",[\"^ \",\"^2\",true,\"^3\",\"^4\",\"^5\",\"^C\",\"^7\",\"^1Q\"],\"~:logseq.property/enable-history?\",[\"^ \",\"^2\",true,\"^5\",\"^C\",\"^7\",\"^4R\"],\"~i15\",\"~:logseq.property/created-from-property\",\"~i47\",\"~:logseq.property.class/hide-from-node\",\"~i79\",\"~:logseq.property/status.done\",\"~:logseq.class/Math-block\",[\"^ \",\"^7\",\"^4Y\"],\"~i111\",\"^2V\",\"~i143\",\"~:logseq.property.fsrs/due\",\"~i175\",\"^2D\",\"^9\",[\"^ \",\"^2\",true,\"^5\",\"^C\",\"^7\",\"^9\"],\"~:logseq.property.history/scalar-value\",[\"^ \",\"^2\",true,\"^5\",\"^C\",\"^7\",\"^52\"],\"~:logseq.property/page-tags\",[\"^ \",\"^2\",true,\"^3\",\"^4\",\"^5\",\"^6\",\"^7\",\"^53\"],\"~:logseq.property/scheduled\",[\"^ \",\"^2\",true,\"^5\",\"^C\",\"^7\",\"^54\"],\"^3A\",[\"^ \",\"^7\",\"^3A\"],\"~:block/collapsed?\",[\"^ \",\"^2\",true,\"^5\",\"^C\",\"^7\",\"^55\"],\"^3N\",[\"^ \",\"^2\",true,\"^5\",\"^C\",\"^7\",\"^3N\"],\"~i16\",\"~:logseq.property/public?\",\"~i48\",\"~:logseq.property/query\",\"~i80\",\"~:logseq.property/status.canceled\",\"~i112\",\"^E\",\"~i144\",\"^V\",\"~i176\",\"~:logseq.class/Code-block\",\"^4?\",[\"^ \",\"^2\",true,\"^5\",\"^C\",\"^7\",\"^4?\"],\"~i17\",\"^4<\",\"~i49\",\"^53\",\"~i81\",\"~:logseq.property/priority\",\"^4V\",[\"^ \",\"^2\",true,\"^5\",\"^C\",\"^7\",\"^4V\"],\"~i113\",\"^3Q\",\"~i145\",\"^I\",\"~i177\",\"^2W\",\"^1A\",[\"^ \",\"^2\",true,\"^3\",\"^4\",\"^5\",\"^6\",\"^7\",\"^1A\"],\"~:logseq.property.view/feature-type\",[\"^ \",\"^2\",true,\"^5\",\"^C\",\"^7\",\"^5G\"],\"^5?\",[\"^ \",\"^7\",\"^5?\"],\"^2<\",[\"^ \",\"^2\",true,\"^5\",\"^C\",\"^7\",\"^2<\"],\"^3=\",[\"^ \",\"^2\",true,\"^3\",\"^4\",\"^5\",\"^C\",\"^7\",\"^3=\"],\"^4F\",[\"^ \",\"^2\",true,\"^5\",\"^C\",\"^7\",\"^4F\"],\"~:logseq.property/ls-type\",[\"^ \",\"^2\",true,\"^5\",\"^C\",\"^7\",\"^5H\"],\"^4Q\",[\"^ \",\"^7\",\"^4Q\"],\"~:logseq.property.user/email\",[\"^ \",\"^2\",true,\"^5\",\"^C\",\"^7\",\"^5I\"],\"^17\",[\"^ \",\"^2\",true,\"^3\",\"^4\",\"^5\",\"^C\",\"^7\",\"^17\"],\"^2P\",[\"^ \",\"^2\",true,\"^5\",\"^C\",\"^7\",\"^2P\"],\"~i18\",\"^F\",\"^59\",[\"^ \",\"^2\",true,\"^3\",\"^4\",\"^5\",\"^C\",\"^7\",\"^59\"],\"~i82\",\"~:logseq.property/priority.low\",\"~i114\",\"~:logseq.property.view/type.gallery\",\"~i146\",\"^5I\",\"~i178\",\"^4Y\",\"^4I\",[\"^ \",\"^2\",true,\"^5\",\"^C\",\"^7\",\"^4I\"],\"~:logseq.property/background-color\",[\"^ \",\"^2\",true,\"^3\",\"^4\",\"^5\",\"^C\",\"^7\",\"^5Q\"],\"~:logseq.property.repeat/recur-frequency\",[\"^ \",\"^2\",true,\"^3\",\"^4\",\"^5\",\"^C\",\"^7\",\"^5R\"],\"^1W\",[\"^ \",\"^2\",true,\"^3\",\"^4\",\"^5\",\"^C\",\"^7\",\"^1W\"],\"~:block/journal-day\",[\"^ \",\"^2\",true,\"^5\",\"^C\",\"^7\",\"^5S\"],\"~:logseq.property/color.red\",[\"^ \",\"^7\",\"^5T\"],\"~i19\",\"~:logseq.property/classes\",\"^J\",[\"^ \",\"^7\",\"^J\"],\"~i51\",\"^5Q\",\"~i83\",\"~:logseq.property/priority.medium\",\"~i115\",\"^5G\",\"~i147\",\"^28\",\"~i179\",\"~:logseq.class/Pdf-annotation\",\"~:logseq.property.repeat/recur-unit.minute\",[\"^ \",\"^7\",\"^62\"],\"^3E\",[\"^ \",\"^7\",\"^3E\"],\"^1I\",[\"^ \",\"^2\",true,\"^5\",\"^C\",\"^7\",\"^1I\"],\"~i20\",\"^Z\",\"~i52\",\"^2X\",\"~i84\",\"~:logseq.property/priority.high\",\"~i116\",\"^4;\",\"~:logseq.property/created-by-ref\",[\"^ \",\"^2\",true,\"^3\",\"^4\",\"^5\",\"^C\",\"^7\",\"^68\"],\"~i148\",\"^4R\",\"^66\",[\"^ \",\"^7\",\"^66\"],\"^5L\",[\"^ \",\"^7\",\"^5L\"],\"~:block/tags\",[\"^ \",\"^2\",true,\"^3\",\"^4\",\"^5\",\"^6\",\"^7\",\"^6:\"],\"~:block/title\",[\"^ \",\"^2\",true,\"^5\",\"^C\",\"^7\",\"^6;\"],\"^35\",[\"^ \",\"^7\",\"^35\"],\"~i21\",\"^25\",\"~i53\",\"~:logseq.property/asset\",\"~i85\",\"~:logseq.property/priority.urgent\",\"~i117\",\"^3G\",\"^4T\",[\"^ \",\"^2\",true,\"^3\",\"^4\",\"^5\",\"^C\",\"^7\",\"^4T\"],\"~:logseq.property.view/gallery-card-width\",[\"^ \",\"^2\",true,\"^5\",\"^C\",\"^7\",\"^6B\"],\"^3X\",[\"^ \",\"^2\",true,\"^5\",\"^C\",\"^7\",\"^3X\"],\"^3I\",[\"^ \",\"^2\",true,\"^5\",\"^C\",\"^7\",\"^3I\"],\"^6@\",[\"^ \",\"^7\",\"^6@\"],\"^2L\",[\"^ \",\"^2\",true,\"^3\",\"^4\",\"^5\",\"^6\",\"^7\",\"^2L\"],\"^2@\",[\"^ \",\"^2\",true,\"^5\",\"^C\",\"^7\",\"^2@\"],\"~i22\",\"^6:\",\"~i54\",\"^5H\",\"~i86\",\"^1R\",\"~i118\",\"^1\",\"~i150\",\"~:logseq.property.history/block\",\"^5V\",[\"^ \",\"^2\",true,\"^3\",\"^4\",\"^5\",\"^6\",\"^7\",\"^5V\"],\"^6>\",[\"^ \",\"^2\",true,\"^3\",\"^4\",\"^5\",\"^C\",\"^7\",\"^6>\"],\"^38\",[\"^ \",\"^7\",\"^38\"],\"~i23\",\"~:block/parent\",\"~i55\",\"~:logseq.property.pdf/hl-type\",\"~:logseq.property/deleted-at\",[\"^ \",\"^2\",true,\"^5\",\"^C\",\"^7\",\"^6M\"],\"~i119\",\"~:logseq.property.view/gallery-card-size\",\"~i151\",\"~:logseq.property.history/property\",\"^N\",[\"^ \",\"^2\",true,\"^3\",\"^4\",\"^5\",\"^C\",\"^7\",\"^N\"],\"^2>\",[\"^ \",\"^7\",\"^2>\"],\"^4N\",[\"^ \",\"^2\",true,\"^5\",\"^C\",\"^7\",\"^4N\"],\"~i24\",\"~:block/order\",\"~:logseq.property/deleted-by-ref\",[\"^ \",\"^2\",true,\"^3\",\"^4\",\"^5\",\"^C\",\"^7\",\"^6T\"],\"~i56\",\"^W\",\"^6L\",[\"^ \",\"^2\",true,\"^5\",\"^C\",\"^7\",\"^6L\"],\"~i88\",\"^54\",\"~i120\",\"^6B\",\"~i152\",\"~:logseq.property.history/ref-value\",\"^7\",[\"^ \",\"^30\",\"^31\"],\"^5N\",[\"^ \",\"^7\",\"^5N\"],\"^1?\",[\"^ \",\"^2\",true,\"^3\",\"^4\",\"^5\",\"^C\",\"^7\",\"^1?\"],\"^6H\",[\"^ \",\"^2\",true,\"^3\",\"^4\",\"^5\",\"^C\",\"^7\",\"^6H\"],\"^33\",[\"^ \",\"^2\",true,\"^5\",\"^C\",\"^7\",\"^33\"],\"^6J\",[\"^ \",\"^2\",true,\"^3\",\"^4\",\"^5\",\"^C\",\"^7\",\"^6J\"],\"~i25\",\"^55\",\"~i57\",\"~:logseq.property/color.yellow\",\"~:logseq.property.recycle/original-page\",[\"^ \",\"^2\",true,\"^3\",\"^4\",\"^5\",\"^C\",\"^7\",\"^71\"],\"~i121\",\"^B\",\"~i153\",\"^52\",\"^2R\",[\"^ \",\"^7\",\"^2R\"],\"~i26\",\"~:block/page\",\"~i58\",\"^5T\",\"~i90\",\"^5R\",\"~:logseq.property.repeat/recur-unit.hour\",[\"^ \",\"^7\",\"^78\"],\"^70\",[\"^ \",\"^7\",\"^70\"],\"~i122\",\"^41\",\"~i154\",\"^68\",\"~:logseq.property.recycle/original-parent\",[\"^ \",\"^2\",true,\"^3\",\"^4\",\"^5\",\"^C\",\"^7\",\"^7;\"],\"^5Y\",[\"^ \",\"^7\",\"^5Y\"],\"^6Q\",[\"^ \",\"^2\",true,\"^3\",\"^4\",\"^5\",\"^C\",\"^7\",\"^6Q\"],\"^;\",[\"^ \",\"^2\",true,\"^5\",\"^C\",\"^7\",\"^;\"],\"^4:\",[\"^ \",\"^7\",\"^4:\"],\"~i27\",\"^42\",\"~i59\",\"^2G\",\"~i123\",\"^20\",\"~i155\",\"^6M\",\"^3Z\",[\"^ \",\"^2\",true,\"^5\",\"^C\",\"^7\",\"^3Z\"],\"^P\",[\"^ \",\"^7\",\"^P\"],\"^6S\",[\"^ \",\"^2\",true,\"^5\",\"^C\",\"^7\",\"^6S\"],\"^5C\",[\"^ \",\"^2\",true,\"^3\",\"^4\",\"^5\",\"^C\",\"^7\",\"^5C\"],\"^6Y\",[\"^ \",\"^2\",true,\"^3\",\"^4\",\"^5\",\"^C\",\"^7\",\"^6Y\"],\"^50\",[\"^ \",\"^2\",true,\"^5\",\"^C\",\"^7\",\"^50\"],\"~i28\",\"^2U\",\"^4C\",[\"^ \",\"^7\",\"^4C\"],\"~i60\",\"^3C\",\"~i92\",\"^2S\",\"~i124\",\"^1:\",\"^75\",[\"^ \",\"^2\",true,\"^3\",\"^4\",\"^5\",\"^C\",\"^7\",\"^75\"],\"~i156\",\"^6T\",\"~:block/name\",[\"^ \",\"^2\",true],\"^61\",[\"^ \",\"^7\",\"^61\"],\"~:logseq.property.table/ordered-columns\",[\"^ \",\"^2\",true,\"^5\",\"^C\",\"^7\",\"^7F\"],\"~:file/path\",[\"^ \",\"^30\",\"^31\"],\"~i29\",\"^6;\",\"~i61\",\"^36\",\"~i93\",\"^62\",\"~i125\",\"~:logseq.property.table/filters\",\"~i157\",\"^7;\",\"^1C\",[\"^ \",\"^2\",true,\"^5\",\"^C\",\"^7\",\"^1C\"],\"^[\",[\"^ \",\"^7\",\"^[\"],\"^4L\",[\"^ \",\"^7\",\"^4L\"],\"^57\",[\"^ \",\"^2\",true,\"^5\",\"^C\",\"^7\",\"^57\"],\"~i30\",\"^4A\",\"~i62\",\"^4=\",\"~i94\",\"^78\",\"~i126\",\"^37\",\"~i158\",\"^71\",\"^5;\",[\"^ \",\"^7\",\"^5;\"],\"^24\",[\"^ \",\"^2\",true,\"^3\",\"^4\",\"^5\",\"^6\",\"^7\",\"^24\"],\"^1U\",[\"^ \",\"^7\",\"^1U\"],\"^1Y\",[\"^ \",\"^2\",true,\"^5\",\"^C\",\"^7\",\"^1Y\"],\"^4X\",[\"^ \",\"^7\",\"^4X\"],\"~i31\",\"^5S\",\"~i63\",\"^Q\",\"^27\",[\"^ \",\"^7\",\"^27\"],\"~i95\",\"^1<\",\"~i127\",\"^7F\",\"~i159\",\"^2E\",\"^7L\",[\"^ \",\"^2\",true,\"^5\",\"^C\",\"^7\",\"^7L\"],\"^6O\",[\"^ \",\"^2\",true,\"^5\",\"^C\",\"^7\",\"^6O\"],\"^1M\",[\"^ \",\"^2\",true,\"^3\",\"^4\",\"^5\",\"^C\",\"^7\",\"^1M\"],\"^3P\",[\"^ \",\"^7\",\"^3P\"],\"^46\",[\"^ \",\"^7\",\"^46\"]],\"~:max-tx\",536870913,\"~:aevt\",1000290,\"~:max-addr\",1000402,\"~:avet\",1000402,\"~:branching-factor\",32,\"~:max-eid\",190,\"~:ref-type\",\"~:strong\",\"~:eavt\",1000149,\"~:aevt-metadata\",[\"^ \",\"~:count\",2384,\"~:shift\",2],\"~:avet-metadata\",[\"^ \",\"^86\",2064,\"^87\",2],\"~:eavt-metadata\",[\"^ \",\"^86\",2384,\"^87\",2]]", + "addresses": null + }, + { + "addr": 1, + "content": "[[[191,\"~:db/ident\",\"~:logseq.kv/graph-last-gc-at\",536870914],[191,\"~:kv/value\",1786627375904,536870914]],[[192,\"^0\",\"~:logseq.kv/recycle-last-gc-at\",536870915],[192,\"^2\",1786627375905,536870915]],[[193,\"~:block/name\",\"oracle page\",536870916],[193,\"~:block/title\",\"Oracle Page\",536870916],[193,\"~:block/uuid\",\"~u11111111-1111-4111-8111-111111111111\",536870916],[193,\"~:block/created-at\",1786627376173,536870916],[193,\"~:block/updated-at\",1786627376173,536870916],[193,\"~:block/tags\",4,536870916],[193,\"~:block/refs\",4,536870918],[193,\"^:\",22,536870918],[193,\"~:block/tx-id\",536870916,536870918]]]", + "addresses": null + }, + { + "addr": 1000001, + "content": "[\"^ \",\"~:keys\",[[1,\"~:block/created-at\",1786627375725,536870913],[1,\"~:block/name\",\"root tag\",536870913],[1,\"~:block/tags\",2,536870913],[1,\"~:block/title\",\"Root Tag\",536870913],[1,\"~:block/updated-at\",1786627375725,536870913],[1,\"~:block/uuid\",\"~u00000002-2737-8382-7000-000000000000\",536870913],[1,\"~:db/ident\",\"~:logseq.class/Root\",536870913],[1,\"~:logseq.property/built-in?\",true,536870913],[2,\"^1\",1786627375725,536870913],[2,\"^2\",\"tag\",536870913],[2,\"^3\",2,536870913],[2,\"^4\",\"Tag\",536870913],[2,\"^5\",1786627375725,536870913],[2,\"^6\",\"~u00000002-5389-0208-3000-000000000000\",536870913],[2,\"^7\",\"~:logseq.class/Tag\",536870913],[2,\"^9\",true,536870913]]]", + "addresses": null + }, + { + "addr": 1000002, + "content": "[\"^ \",\"~:keys\",[[57,\"~:block/closed-value-property\",56,536870913],[58,\"^1\",56,536870913],[59,\"^1\",56,536870913],[60,\"^1\",56,536870913],[61,\"^1\",56,536870913],[75,\"^1\",74,536870913],[76,\"^1\",74,536870913],[77,\"^1\",74,536870913],[78,\"^1\",74,536870913],[79,\"^1\",74,536870913],[80,\"^1\",74,536870913],[82,\"^1\",81,536870913],[83,\"^1\",81,536870913],[84,\"^1\",81,536870913],[85,\"^1\",81,536870913],[93,\"^1\",92,536870913],[94,\"^1\",92,536870913]]]", + "addresses": null + }, + { + "addr": 1000003, + "content": "[\"^ \",\"~:keys\",[[57,\"~:block/closed-value-property\",56,536870913],[58,\"^1\",56,536870913],[59,\"^1\",56,536870913],[60,\"^1\",56,536870913],[61,\"^1\",56,536870913],[75,\"^1\",74,536870913],[76,\"^1\",74,536870913],[77,\"^1\",74,536870913],[78,\"^1\",74,536870913],[79,\"^1\",74,536870913],[80,\"^1\",74,536870913],[82,\"^1\",81,536870913],[83,\"^1\",81,536870913],[84,\"^1\",81,536870913],[85,\"^1\",81,536870913],[93,\"^1\",92,536870913],[94,\"^1\",92,536870913]]]", + "addresses": null + }, + { + "addr": 1000004, + "content": "[\"^ \",\"~:keys\",[[2,\"~:logseq.property.class/extends\",1,536870913],[3,\"~:block/created-at\",1786627375725,536870913],[3,\"~:block/name\",\"property\",536870913],[3,\"~:block/tags\",2,536870913],[3,\"~:block/title\",\"Property\",536870913],[3,\"~:block/updated-at\",1786627375725,536870913],[3,\"~:block/uuid\",\"~u00000002-1038-7670-4800-000000000000\",536870913],[3,\"~:db/ident\",\"~:logseq.class/Property\",536870913],[3,\"~:logseq.property/built-in?\",true,536870913],[3,\"^1\",1,536870913],[4,\"^2\",1786627375725,536870913],[4,\"^3\",\"page\",536870913],[4,\"^4\",2,536870913],[4,\"^5\",\"Page\",536870913],[4,\"^6\",1786627375725,536870913],[4,\"^7\",\"~u00000002-1484-3403-2900-000000000000\",536870913]]]", + "addresses": null + }, + { + "addr": 1000005, + "content": "[\"^ \",\"~:keys\",[[4,\"~:db/ident\",\"~:logseq.class/Page\",536870913],[4,\"~:logseq.property/built-in?\",true,536870913],[4,\"~:logseq.property.class/extends\",1,536870913],[5,\"~:block/created-at\",1786627375725,536870913],[5,\"~:block/name\",\"template\",536870913],[5,\"~:block/tags\",2,536870913],[5,\"~:block/title\",\"Template\",536870913],[5,\"~:block/updated-at\",1786627375725,536870913],[5,\"~:block/uuid\",\"~u00000002-1720-8548-4600-000000000000\",536870913],[5,\"^1\",\"~:logseq.class/Template\",536870913],[5,\"^3\",true,536870913],[5,\"^4\",1,536870913],[5,\"~:logseq.property.class/properties\",165,536870913],[6,\"^1\",\"~:logseq.kv/db-type\",536870913],[6,\"~:kv/value\",\"db\",536870913],[7,\"^1\",\"~:logseq.kv/schema-version\",536870913],[7,\"^>\",[\"^ \",\"~:major\",65,\"~:minor\",33],536870913],[8,\"^1\",\"~:logseq.kv/graph-initial-schema-version\",536870913],[8,\"^>\",[\"^ \",\"^@\",65,\"^A\",33],536870913]]]", + "addresses": null + }, + { + "addr": 1000006, + "content": "[\"^ \",\"~:keys\",[[9,\"~:db/ident\",\"~:logseq.kv/graph-created-at\",536870913],[9,\"~:kv/value\",1786627375714,536870913],[10,\"~:block/uuid\",\"~u00000004-1595-0218-3700-000000000000\",536870913],[10,\"^1\",\"~:logseq.property/empty-placeholder\",536870913],[11,\"^1\",\"~:logseq.kv/local-graph-uuid\",536870913],[11,\"^3\",\"~u00000000-7b10-4374-96ba-e7f6ea9cbb32\",536870913],[12,\"~:block/created-at\",1786627375714,536870913],[12,\"~:block/name\",\"property type\",536870913],[12,\"~:block/order\",\"a0\",536870913],[12,\"~:block/tags\",3,536870913],[12,\"~:block/title\",\"Property type\",536870913],[12,\"~:block/updated-at\",1786627375714,536870913],[12,\"^4\",\"~u00000002-8384-2404-0000-000000000000\",536870913],[12,\"~:db/cardinality\",\"~:db.cardinality/one\",536870913],[12,\"^1\",\"~:logseq.property/type\",536870913],[12,\"~:db/index\",true,536870913]]]", + "addresses": null + }, + { + "addr": 1000007, + "content": "[\"^ \",\"~:keys\",[[12,\"~:logseq.property/built-in?\",true,536870913],[12,\"~:logseq.property/hide?\",true,536870913],[12,\"~:logseq.property/type\",\"~:keyword\",536870913],[13,\"~:block/created-at\",1786627375714,536870913],[13,\"~:block/name\",\"built in?\",536870913],[13,\"~:block/order\",\"a1\",536870913],[13,\"~:block/tags\",3,536870913],[13,\"~:block/title\",\"Built in?\",536870913],[13,\"~:block/updated-at\",1786627375714,536870913],[13,\"~:block/uuid\",\"~u00000002-1125-9581-6000-000000000000\",536870913],[13,\"~:db/cardinality\",\"~:db.cardinality/one\",536870913],[13,\"~:db/ident\",\"^1\",536870913],[13,\"~:db/index\",true,536870913],[13,\"^1\",true,536870913],[13,\"^2\",true,536870913],[13,\"^3\",\"~:checkbox\",536870913],[14,\"^5\",1786627375714,536870913],[14,\"^6\",\"hide this property or page\",536870913],[14,\"^7\",\"a2\",536870913],[14,\"^8\",3,536870913],[14,\"^9\",\"Hide this property or page\",536870913]]]", + "addresses": null + }, + { + "addr": 1000008, + "content": "[\"^ \",\"~:keys\",[[14,\"~:block/updated-at\",1786627375714,536870913],[14,\"~:block/uuid\",\"~u00000002-6836-5746-0000-000000000000\",536870913],[14,\"~:db/cardinality\",\"~:db.cardinality/one\",536870913],[14,\"~:db/ident\",\"~:logseq.property/hide?\",536870913],[14,\"~:db/index\",true,536870913],[14,\"~:logseq.property/built-in?\",true,536870913],[14,\"^6\",true,536870913],[14,\"~:logseq.property/type\",\"~:checkbox\",536870913],[15,\"~:block/created-at\",1786627375714,536870913],[15,\"~:block/name\",\"created from property\",536870913],[15,\"~:block/order\",\"a3\",536870913],[15,\"~:block/tags\",3,536870913],[15,\"~:block/title\",\"Created from property\",536870913],[15,\"^1\",1786627375714,536870913],[15,\"^2\",\"~u00000002-8618-9226-7000-000000000000\",536870913],[15,\"^3\",\"^4\",536870913],[15,\"^5\",\"~:logseq.property/created-from-property\",536870913]]]", + "addresses": null + }, + { + "addr": 1000009, + "content": "[\"^ \",\"~:keys\",[[15,\"~:db/index\",true,536870913],[15,\"~:db/valueType\",\"~:db.type/ref\",536870913],[15,\"~:logseq.property/built-in?\",true,536870913],[15,\"~:logseq.property/hide?\",true,536870913],[15,\"~:logseq.property/type\",\"~:entity\",536870913],[16,\"~:block/created-at\",1786627375715,536870913],[16,\"~:block/name\",\"property public?\",536870913],[16,\"~:block/order\",\"a4\",536870913],[16,\"~:block/tags\",3,536870913],[16,\"~:block/title\",\"Property public?\",536870913],[16,\"~:block/updated-at\",1786627375715,536870913],[16,\"~:block/uuid\",\"~u00000002-1843-0851-4900-000000000000\",536870913],[16,\"~:db/cardinality\",\"~:db.cardinality/one\",536870913],[16,\"~:db/ident\",\"~:logseq.property/public?\",536870913],[16,\"^1\",true,536870913],[16,\"^4\",true,536870913],[16,\"^5\",true,536870913],[16,\"^6\",\"~:checkbox\",536870913]]]", + "addresses": null + }, + { + "addr": 1000010, + "content": "[\"^ \",\"~:keys\",[[17,\"~:block/created-at\",1786627375715,536870913],[17,\"~:block/name\",\"property view context\",536870913],[17,\"~:block/order\",\"a5\",536870913],[17,\"~:block/tags\",3,536870913],[17,\"~:block/title\",\"Property view context\",536870913],[17,\"~:block/updated-at\",1786627375715,536870913],[17,\"~:block/uuid\",\"~u00000002-1547-3958-2800-000000000000\",536870913],[17,\"~:db/cardinality\",\"~:db.cardinality/one\",536870913],[17,\"~:db/ident\",\"~:logseq.property/view-context\",536870913],[17,\"~:db/index\",true,536870913],[17,\"~:logseq.property/built-in?\",true,536870913],[17,\"~:logseq.property/hide?\",true,536870913],[17,\"~:logseq.property/type\",\"~:keyword\",536870913],[18,\"^1\",1786627375715,536870913],[18,\"^2\",\"property position\",536870913],[18,\"^3\",\"a6\",536870913],[18,\"^4\",3,536870913]]]", + "addresses": null + }, + { + "addr": 1000011, + "content": "[\"^ \",\"~:keys\",[[18,\"~:block/title\",\"Property position\",536870913],[18,\"~:block/updated-at\",1786627375715,536870913],[18,\"~:block/uuid\",\"~u00000002-1869-2008-6400-000000000000\",536870913],[18,\"~:db/cardinality\",\"~:db.cardinality/one\",536870913],[18,\"~:db/ident\",\"~:logseq.property/ui-position\",536870913],[18,\"~:db/index\",true,536870913],[18,\"~:logseq.property/built-in?\",true,536870913],[18,\"~:logseq.property/hide?\",true,536870913],[18,\"~:logseq.property/type\",\"~:keyword\",536870913],[19,\"~:block/created-at\",1786627375715,536870913],[19,\"~:block/name\",\"property classes\",536870913],[19,\"~:block/order\",\"a7\",536870913],[19,\"~:block/tags\",3,536870913],[19,\"^1\",\"Property classes\",536870913],[19,\"^2\",1786627375715,536870913],[19,\"^3\",\"~u00000002-9137-5048-6000-000000000000\",536870913],[19,\"^4\",\"~:db.cardinality/many\",536870913]]]", + "addresses": null + }, + { + "addr": 1000012, + "content": "[\"^ \",\"~:keys\",[[19,\"~:db/ident\",\"~:logseq.property/classes\",536870913],[19,\"~:db/index\",true,536870913],[19,\"~:db/valueType\",\"~:db.type/ref\",536870913],[19,\"~:logseq.property/built-in?\",true,536870913],[19,\"~:logseq.property/hide?\",true,536870913],[19,\"~:logseq.property/public?\",false,536870913],[19,\"~:logseq.property/type\",\"~:entity\",536870913],[20,\"~:block/created-at\",1786627375716,536870913],[20,\"~:block/name\",\"property value\",536870913],[20,\"~:block/order\",\"a8\",536870913],[20,\"~:block/tags\",3,536870913],[20,\"~:block/title\",\"Property value\",536870913],[20,\"~:block/updated-at\",1786627375716,536870913],[20,\"~:block/uuid\",\"~u00000002-1396-5248-6500-000000000000\",536870913],[20,\"~:db/cardinality\",\"~:db.cardinality/one\",536870913],[20,\"^1\",\"~:logseq.property/value\",536870913],[20,\"^3\",true,536870913]]]", + "addresses": null + }, + { + "addr": 1000013, + "content": "[\"^ \",\"~:keys\",[[20,\"~:logseq.property/built-in?\",true,536870913],[20,\"~:logseq.property/hide?\",true,536870913],[20,\"~:logseq.property/public?\",false,536870913],[20,\"~:logseq.property/type\",\"~:any\",536870913],[21,\"~:block/created-at\",1786627375716,536870913],[21,\"~:block/name\",\"alias\",536870913],[21,\"~:block/order\",\"a9\",536870913],[21,\"~:block/tags\",3,536870913],[21,\"~:block/title\",\"Alias\",536870913],[21,\"~:block/updated-at\",1786627375716,536870913],[21,\"~:block/uuid\",\"~u00000002-2112-6446-9900-000000000000\",536870913],[21,\"~:db/cardinality\",\"~:db.cardinality/many\",536870913],[21,\"~:db/ident\",\"~:block/alias\",536870913],[21,\"~:db/index\",true,536870913],[21,\"~:db/valueType\",\"~:db.type/ref\",536870913],[21,\"^1\",true,536870913],[21,\"^3\",true,536870913],[21,\"^4\",\"~:page\",536870913]]]", + "addresses": null + }, + { + "addr": 1000014, + "content": "[\"^ \",\"~:keys\",[[21,\"~:logseq.property/view-context\",\"~:page\",536870913],[22,\"~:block/created-at\",1786627375716,536870913],[22,\"~:block/name\",\"tags\",536870913],[22,\"~:block/order\",\"aA\",536870913],[22,\"~:block/tags\",3,536870913],[22,\"~:block/title\",\"Tags\",536870913],[22,\"~:block/updated-at\",1786627375716,536870913],[22,\"~:block/uuid\",\"~u00000002-1814-9483-4000-000000000000\",536870913],[22,\"~:db/cardinality\",\"~:db.cardinality/many\",536870913],[22,\"~:db/ident\",\"^6\",536870913],[22,\"~:db/index\",true,536870913],[22,\"~:db/valueType\",\"~:db.type/ref\",536870913],[22,\"~:logseq.property/built-in?\",true,536870913],[22,\"~:logseq.property/public?\",true,536870913],[22,\"~:logseq.property/type\",\"~:class\",536870913],[23,\"^3\",1786627375716,536870913],[23,\"^4\",\"node parent\",536870913]]]", + "addresses": null + }, + { + "addr": 1000015, + "content": "[\"^ \",\"~:keys\",[[23,\"~:block/order\",\"aB\",536870913],[23,\"~:block/tags\",3,536870913],[23,\"~:block/title\",\"Node parent\",536870913],[23,\"~:block/updated-at\",1786627375716,536870913],[23,\"~:block/uuid\",\"~u00000002-9183-0906-4000-000000000000\",536870913],[23,\"~:db/cardinality\",\"~:db.cardinality/one\",536870913],[23,\"~:db/ident\",\"~:block/parent\",536870913],[23,\"~:db/index\",true,536870913],[23,\"~:db/valueType\",\"~:db.type/ref\",536870913],[23,\"~:logseq.property/built-in?\",true,536870913],[23,\"~:logseq.property/hide?\",true,536870913],[23,\"~:logseq.property/public?\",false,536870913],[23,\"~:logseq.property/type\",\"~:entity\",536870913],[24,\"~:block/created-at\",1786627375716,536870913],[24,\"~:block/name\",\"node order\",536870913],[24,\"^1\",\"aC\",536870913],[24,\"^2\",3,536870913]]]", + "addresses": null + }, + { + "addr": 1000016, + "content": "[\"^ \",\"~:keys\",[[24,\"~:block/title\",\"Node order\",536870913],[24,\"~:block/updated-at\",1786627375716,536870913],[24,\"~:block/uuid\",\"~u00000002-1429-2824-3700-000000000000\",536870913],[24,\"~:db/cardinality\",\"~:db.cardinality/one\",536870913],[24,\"~:db/ident\",\"~:block/order\",536870913],[24,\"~:db/index\",true,536870913],[24,\"~:logseq.property/built-in?\",true,536870913],[24,\"~:logseq.property/hide?\",true,536870913],[24,\"~:logseq.property/public?\",false,536870913],[24,\"~:logseq.property/type\",\"~:string\",536870913],[25,\"~:block/created-at\",1786627375716,536870913],[25,\"~:block/name\",\"node collapsed?\",536870913],[25,\"^7\",\"aD\",536870913],[25,\"~:block/tags\",3,536870913],[25,\"^1\",\"Node collapsed?\",536870913],[25,\"^2\",1786627375716,536870913],[25,\"^3\",\"~u00000002-2140-2109-9100-000000000000\",536870913]]]", + "addresses": null + }, + { + "addr": 1000017, + "content": "[\"^ \",\"~:keys\",[[25,\"~:db/cardinality\",\"~:db.cardinality/one\",536870913],[25,\"~:db/ident\",\"~:block/collapsed?\",536870913],[25,\"~:db/index\",true,536870913],[25,\"~:logseq.property/built-in?\",true,536870913],[25,\"~:logseq.property/hide?\",true,536870913],[25,\"~:logseq.property/public?\",false,536870913],[25,\"~:logseq.property/type\",\"~:checkbox\",536870913],[26,\"~:block/created-at\",1786627375716,536870913],[26,\"~:block/name\",\"node page\",536870913],[26,\"~:block/order\",\"aE\",536870913],[26,\"~:block/tags\",3,536870913],[26,\"~:block/title\",\"Node page\",536870913],[26,\"~:block/updated-at\",1786627375716,536870913],[26,\"~:block/uuid\",\"~u00000002-8223-1410-8000-000000000000\",536870913],[26,\"^1\",\"^2\",536870913],[26,\"^3\",\"~:block/page\",536870913],[26,\"^5\",true,536870913]]]", + "addresses": null + }, + { + "addr": 1000018, + "content": "[\"^ \",\"~:keys\",[[26,\"~:db/valueType\",\"~:db.type/ref\",536870913],[26,\"~:logseq.property/built-in?\",true,536870913],[26,\"~:logseq.property/hide?\",true,536870913],[26,\"~:logseq.property/public?\",false,536870913],[26,\"~:logseq.property/type\",\"~:entity\",536870913],[27,\"~:block/created-at\",1786627375716,536870913],[27,\"~:block/name\",\"node references\",536870913],[27,\"~:block/order\",\"aF\",536870913],[27,\"~:block/tags\",3,536870913],[27,\"~:block/title\",\"Node references\",536870913],[27,\"~:block/updated-at\",1786627375716,536870913],[27,\"~:block/uuid\",\"~u00000002-1214-4953-4900-000000000000\",536870913],[27,\"~:db/cardinality\",\"~:db.cardinality/many\",536870913],[27,\"~:db/ident\",\"~:block/refs\",536870913],[27,\"~:db/index\",true,536870913],[27,\"^1\",\"^2\",536870913],[27,\"^3\",true,536870913],[27,\"^4\",true,536870913]]]", + "addresses": null + }, + { + "addr": 1000019, + "content": "[\"^ \",\"~:keys\",[[27,\"~:logseq.property/public?\",false,536870913],[27,\"~:logseq.property/type\",\"~:entity\",536870913],[28,\"~:block/created-at\",1786627375716,536870913],[28,\"~:block/name\",\"node links to\",536870913],[28,\"~:block/order\",\"aG\",536870913],[28,\"~:block/tags\",3,536870913],[28,\"~:block/title\",\"Node links to\",536870913],[28,\"~:block/updated-at\",1786627375716,536870913],[28,\"~:block/uuid\",\"~u00000002-1872-3999-9300-000000000000\",536870913],[28,\"~:db/cardinality\",\"~:db.cardinality/one\",536870913],[28,\"~:db/ident\",\"~:block/link\",536870913],[28,\"~:db/index\",true,536870913],[28,\"~:db/valueType\",\"~:db.type/ref\",536870913],[28,\"~:logseq.property/built-in?\",true,536870913],[28,\"~:logseq.property/hide?\",true,536870913],[28,\"^1\",false,536870913],[28,\"^2\",\"^3\",536870913]]]", + "addresses": null + }, + { + "addr": 1000020, + "content": "[\"^ \",\"~:keys\",[[29,\"~:block/created-at\",1786627375717,536870913],[29,\"~:block/name\",\"node title\",536870913],[29,\"~:block/order\",\"aH\",536870913],[29,\"~:block/tags\",3,536870913],[29,\"~:block/title\",\"Node title\",536870913],[29,\"~:block/updated-at\",1786627375717,536870913],[29,\"~:block/uuid\",\"~u00000002-7104-4568-4000-000000000000\",536870913],[29,\"~:db/cardinality\",\"~:db.cardinality/one\",536870913],[29,\"~:db/ident\",\"^5\",536870913],[29,\"~:db/index\",true,536870913],[29,\"~:logseq.property/built-in?\",true,536870913],[29,\"~:logseq.property/hide?\",true,536870913],[29,\"~:logseq.property/public?\",false,536870913],[29,\"~:logseq.property/type\",\"~:string\",536870913],[30,\"^1\",1786627375717,536870913],[30,\"^2\",\"closed value property\",536870913],[30,\"^3\",\"aI\",536870913]]]", + "addresses": null + }, + { + "addr": 1000021, + "content": "[\"^ \",\"~:keys\",[[2,\"~:logseq.property/built-in?\",true,536870913],[4,\"~:block/uuid\",\"~u00000002-1484-3403-2900-000000000000\",536870913],[8,\"~:kv/value\",[\"^ \",\"~:major\",65,\"~:minor\",33],536870913],[12,\"~:db/index\",true,536870913],[14,\"~:block/title\",\"Hide this property or page\",536870913],[15,\"~:db/ident\",\"~:logseq.property/created-from-property\",536870913],[16,\"~:logseq.property/type\",\"~:checkbox\",536870913],[18,\"~:block/tags\",3,536870913],[19,\"~:db/cardinality\",\"~:db.cardinality/many\",536870913],[20,\"^6\",true,536870913],[21,\"^:\",\"~:page\",536870913],[23,\"~:block/name\",\"node parent\",536870913],[24,\"^<\",3,536870913],[25,\"^2\",\"~u00000002-2140-2109-9100-000000000000\",536870913],[26,\"^6\",true,536870913],[27,\"~:logseq.property/hide?\",true,536870913],[28,\"^:\",\"~:entity\",536870913],[30,\"~:block/order\",\"aI\",536870913]]]", + "addresses": "[1000001,1000004,1000005,1000006,1000007,1000008,1000009,1000010,1000011,1000012,1000013,1000014,1000015,1000016,1000017,1000018,1000019,1000020]" + }, + { + "addr": 1000022, + "content": "[\"^ \",\"~:keys\",[[30,\"~:block/tags\",3,536870913],[30,\"~:block/title\",\"Closed value property\",536870913],[30,\"~:block/updated-at\",1786627375717,536870913],[30,\"~:block/uuid\",\"~u00000002-1157-7928-1300-000000000000\",536870913],[30,\"~:db/cardinality\",\"~:db.cardinality/one\",536870913],[30,\"~:db/ident\",\"~:block/closed-value-property\",536870913],[30,\"~:db/index\",true,536870913],[30,\"~:db/valueType\",\"~:db.type/ref\",536870913],[30,\"~:logseq.property/built-in?\",true,536870913],[30,\"~:logseq.property/hide?\",true,536870913],[30,\"~:logseq.property/public?\",false,536870913],[30,\"~:logseq.property/type\",\"~:entity\",536870913],[31,\"~:block/created-at\",1786627375717,536870913],[31,\"~:block/name\",\"journal date\",536870913],[31,\"~:block/order\",\"aJ\",536870913],[31,\"^1\",3,536870913],[31,\"^2\",\"Journal date\",536870913]]]", + "addresses": null + }, + { + "addr": 1000023, + "content": "[\"^ \",\"~:keys\",[[31,\"~:block/updated-at\",1786627375717,536870913],[31,\"~:block/uuid\",\"~u00000002-1457-4836-6000-000000000000\",536870913],[31,\"~:db/cardinality\",\"~:db.cardinality/one\",536870913],[31,\"~:db/ident\",\"~:block/journal-day\",536870913],[31,\"~:db/index\",true,536870913],[31,\"~:logseq.property/built-in?\",true,536870913],[31,\"~:logseq.property/hide?\",true,536870913],[31,\"~:logseq.property/public?\",false,536870913],[31,\"~:logseq.property/type\",\"~:raw-number\",536870913],[32,\"~:block/created-at\",1786627375717,536870913],[32,\"~:block/name\",\"node created at\",536870913],[32,\"~:block/order\",\"aK\",536870913],[32,\"~:block/tags\",3,536870913],[32,\"~:block/title\",\"Node created at\",536870913],[32,\"^1\",1786627375717,536870913],[32,\"^2\",\"~u00000002-1440-0150-0000-000000000000\",536870913],[32,\"^3\",\"^4\",536870913]]]", + "addresses": null + }, + { + "addr": 1000024, + "content": "[\"^ \",\"~:keys\",[[32,\"~:db/ident\",\"~:block/created-at\",536870913],[32,\"~:db/index\",true,536870913],[32,\"~:logseq.property/built-in?\",true,536870913],[32,\"~:logseq.property/hide?\",true,536870913],[32,\"~:logseq.property/public?\",false,536870913],[32,\"~:logseq.property/type\",\"~:datetime\",536870913],[33,\"^2\",1786627375717,536870913],[33,\"~:block/name\",\"node updated at\",536870913],[33,\"~:block/order\",\"aL\",536870913],[33,\"~:block/tags\",3,536870913],[33,\"~:block/title\",\"Node updated at\",536870913],[33,\"~:block/updated-at\",1786627375717,536870913],[33,\"~:block/uuid\",\"~u00000002-1516-5505-5100-000000000000\",536870913],[33,\"~:db/cardinality\",\"~:db.cardinality/one\",536870913],[33,\"^1\",\"^=\",536870913],[33,\"^3\",true,536870913],[33,\"^4\",true,536870913],[33,\"^5\",true,536870913]]]", + "addresses": null + }, + { + "addr": 1000025, + "content": "[\"^ \",\"~:keys\",[[33,\"~:logseq.property/public?\",false,536870913],[33,\"~:logseq.property/type\",\"~:datetime\",536870913],[34,\"~:block/created-at\",1786627375717,536870913],[34,\"~:block/name\",\"node display type\",536870913],[34,\"~:block/order\",\"aM\",536870913],[34,\"~:block/tags\",3,536870913],[34,\"~:block/title\",\"Node Display Type\",536870913],[34,\"~:block/updated-at\",1786627375717,536870913],[34,\"~:block/uuid\",\"~u00000002-4424-4618-9000-000000000000\",536870913],[34,\"~:db/cardinality\",\"~:db.cardinality/one\",536870913],[34,\"~:db/ident\",\"~:logseq.property.node/display-type\",536870913],[34,\"~:db/index\",true,536870913],[34,\"~:logseq.property/built-in?\",true,536870913],[34,\"~:logseq.property/hide?\",true,536870913],[34,\"^1\",false,536870913],[34,\"^2\",\"~:keyword\",536870913],[34,\"~:logseq.property/view-context\",\"~:block\",536870913]]]", + "addresses": null + }, + { + "addr": 1000026, + "content": "[\"^ \",\"~:keys\",[[35,\"~:block/created-at\",1786627375717,536870913],[35,\"~:block/name\",\"description\",536870913],[35,\"~:block/order\",\"aN\",536870913],[35,\"~:block/tags\",3,536870913],[35,\"~:block/title\",\"Description\",536870913],[35,\"~:block/updated-at\",1786627375717,536870913],[35,\"~:block/uuid\",\"~u00000002-3362-3620-0000-000000000000\",536870913],[35,\"~:db/cardinality\",\"~:db.cardinality/one\",536870913],[35,\"~:db/ident\",\"~:logseq.property/description\",536870913],[35,\"~:db/index\",true,536870913],[35,\"~:db/valueType\",\"~:db.type/ref\",536870913],[35,\"~:logseq.property/built-in?\",true,536870913],[35,\"~:logseq.property/public?\",true,536870913],[35,\"~:logseq.property/type\",\"~:default\",536870913],[36,\"^1\",1786627375717,536870913],[36,\"^2\",\"code mode\",536870913],[36,\"^3\",\"aO\",536870913]]]", + "addresses": null + }, + { + "addr": 1000027, + "content": "[\"^ \",\"~:keys\",[[36,\"~:block/tags\",3,536870913],[36,\"~:block/title\",\"Code Mode\",536870913],[36,\"~:block/updated-at\",1786627375717,536870913],[36,\"~:block/uuid\",\"~u00000002-8578-9716-5000-000000000000\",536870913],[36,\"~:db/cardinality\",\"~:db.cardinality/one\",536870913],[36,\"~:db/ident\",\"~:logseq.property.code/lang\",536870913],[36,\"~:db/index\",true,536870913],[36,\"~:logseq.property/built-in?\",true,536870913],[36,\"~:logseq.property/hide?\",true,536870913],[36,\"~:logseq.property/public?\",false,536870913],[36,\"~:logseq.property/type\",\"~:string\",536870913],[36,\"~:logseq.property/view-context\",\"~:block\",536870913],[37,\"~:block/created-at\",1786627375717,536870913],[37,\"~:block/name\",\"default value\",536870913],[37,\"~:block/order\",\"aP\",536870913],[37,\"^1\",3,536870913],[37,\"^2\",\"Default value\",536870913]]]", + "addresses": null + }, + { + "addr": 1000028, + "content": "[\"^ \",\"~:keys\",[[37,\"~:block/updated-at\",1786627375717,536870913],[37,\"~:block/uuid\",\"~u00000002-8920-7966-2000-000000000000\",536870913],[37,\"~:db/cardinality\",\"~:db.cardinality/one\",536870913],[37,\"~:db/ident\",\"~:logseq.property/default-value\",536870913],[37,\"~:db/index\",true,536870913],[37,\"~:db/valueType\",\"~:db.type/ref\",536870913],[37,\"~:logseq.property/built-in?\",true,536870913],[37,\"~:logseq.property/hide?\",true,536870913],[37,\"~:logseq.property/public?\",false,536870913],[37,\"~:logseq.property/type\",\"~:entity\",536870913],[37,\"~:logseq.property/view-context\",\"~:property\",536870913],[38,\"~:block/created-at\",1786627375717,536870913],[38,\"~:block/name\",\"non ref type default value\",536870913],[38,\"~:block/order\",\"aQ\",536870913],[38,\"~:block/tags\",3,536870913],[38,\"~:block/title\",\"Non ref type default value\",536870913],[38,\"^1\",1786627375717,536870913]]]", + "addresses": null + }, + { + "addr": 1000029, + "content": "[\"^ \",\"~:keys\",[[38,\"~:block/uuid\",\"~u00000002-1595-7230-1400-000000000000\",536870913],[38,\"~:db/cardinality\",\"~:db.cardinality/one\",536870913],[38,\"~:db/ident\",\"~:logseq.property/scalar-default-value\",536870913],[38,\"~:db/index\",true,536870913],[38,\"~:logseq.property/built-in?\",true,536870913],[38,\"~:logseq.property/hide?\",true,536870913],[38,\"~:logseq.property/public?\",false,536870913],[38,\"~:logseq.property/type\",\"~:any\",536870913],[38,\"~:logseq.property/view-context\",\"~:property\",536870913],[39,\"~:block/created-at\",1786627375717,536870913],[39,\"~:block/name\",\"extends\",536870913],[39,\"~:block/order\",\"aR\",536870913],[39,\"~:block/tags\",3,536870913],[39,\"~:block/title\",\"Extends\",536870913],[39,\"~:block/updated-at\",1786627375717,536870913],[39,\"^1\",\"~u00000002-7475-9380-3000-000000000000\",536870913],[39,\"^2\",\"~:db.cardinality/many\",536870913]]]", + "addresses": null + }, + { + "addr": 1000030, + "content": "[\"^ \",\"~:keys\",[[39,\"~:db/ident\",\"~:logseq.property.class/extends\",536870913],[39,\"~:db/index\",true,536870913],[39,\"~:db/valueType\",\"~:db.type/ref\",536870913],[39,\"~:logseq.property/built-in?\",true,536870913],[39,\"~:logseq.property/description\",40,536870913],[39,\"~:logseq.property/public?\",true,536870913],[39,\"~:logseq.property/type\",\"~:class\",536870913],[39,\"~:logseq.property/view-context\",\"^:\",536870913],[40,\"~:block/created-at\",1786627375717,536870913],[40,\"~:block/order\",\"aS\",536870913],[40,\"~:block/page\",39,536870913],[40,\"~:block/parent\",39,536870913],[40,\"~:block/title\",\"This enables tags to inherit properties from other tags\",536870913],[40,\"~:block/updated-at\",1786627375717,536870913],[40,\"~:block/uuid\",\"~u00000004-4485-6574-6000-000000000000\",536870913],[40,\"^6\",true,536870913],[40,\"~:logseq.property/created-from-property\",35,536870913],[41,\"^<\",1786627375717,536870913]]]", + "addresses": null + }, + { + "addr": 1000031, + "content": "[\"^ \",\"~:keys\",[[41,\"~:block/name\",\"tag properties\",536870913],[41,\"~:block/order\",\"aT\",536870913],[41,\"~:block/tags\",3,536870913],[41,\"~:block/title\",\"Tag Properties\",536870913],[41,\"~:block/updated-at\",1786627375717,536870913],[41,\"~:block/uuid\",\"~u00000002-2123-7120-5000-000000000000\",536870913],[41,\"~:db/cardinality\",\"~:db.cardinality/many\",536870913],[41,\"~:db/ident\",\"~:logseq.property.class/properties\",536870913],[41,\"~:db/index\",true,536870913],[41,\"~:db/valueType\",\"~:db.type/ref\",536870913],[41,\"~:logseq.property/built-in?\",true,536870913],[41,\"~:logseq.property/public?\",true,536870913],[41,\"~:logseq.property/type\",\"~:property\",536870913],[41,\"~:logseq.property/view-context\",\"~:never\",536870913],[42,\"~:block/created-at\",1786627375717,536870913],[42,\"^1\",\"bidirectional property title\",536870913],[42,\"^2\",\"aU\",536870913]]]", + "addresses": null + }, + { + "addr": 1000032, + "content": "[\"^ \",\"~:keys\",[[42,\"~:block/tags\",3,536870913],[42,\"~:block/title\",\"Bidirectional property title\",536870913],[42,\"~:block/updated-at\",1786627375717,536870913],[42,\"~:block/uuid\",\"~u00000002-6050-5418-7000-000000000000\",536870913],[42,\"~:db/cardinality\",\"~:db.cardinality/one\",536870913],[42,\"~:db/ident\",\"~:logseq.property.class/bidirectional-property-title\",536870913],[42,\"~:db/index\",true,536870913],[42,\"~:logseq.property/built-in?\",true,536870913],[42,\"~:logseq.property/public?\",true,536870913],[42,\"~:logseq.property/type\",\"~:string\",536870913],[42,\"~:logseq.property/view-context\",\"~:class\",536870913],[43,\"~:block/created-at\",1786627375717,536870913],[43,\"~:block/name\",\"enable bidirectional properties\",536870913],[43,\"~:block/order\",\"aV\",536870913],[43,\"^1\",3,536870913],[43,\"^2\",\"Enable bidirectional properties\",536870913],[43,\"^3\",1786627375717,536870913]]]", + "addresses": null + }, + { + "addr": 1000033, + "content": "[\"^ \",\"~:keys\",[[43,\"~:block/uuid\",\"~u00000002-2917-8613-8000-000000000000\",536870913],[43,\"~:db/cardinality\",\"~:db.cardinality/one\",536870913],[43,\"~:db/ident\",\"~:logseq.property.class/enable-bidirectional?\",536870913],[43,\"~:db/index\",true,536870913],[43,\"~:logseq.property/built-in?\",true,536870913],[43,\"~:logseq.property/description\",44,536870913],[43,\"~:logseq.property/public?\",true,536870913],[43,\"~:logseq.property/type\",\"~:checkbox\",536870913],[43,\"~:logseq.property/view-context\",\"~:class\",536870913],[44,\"~:block/created-at\",1786627375717,536870913],[44,\"~:block/order\",\"aW\",536870913],[44,\"~:block/page\",43,536870913],[44,\"~:block/parent\",43,536870913],[44,\"~:block/title\",\"When enabled, this tag will show reverse nodes that link to the current node via properties.\",536870913],[44,\"~:block/updated-at\",1786627375717,536870913],[44,\"^1\",\"~u00000004-1830-0481-2500-000000000000\",536870913],[44,\"^7\",true,536870913],[44,\"~:logseq.property/created-from-property\",35,536870913]]]", + "addresses": null + }, + { + "addr": 1000034, + "content": "[\"^ \",\"~:keys\",[[45,\"~:block/created-at\",1786627375717,536870913],[45,\"~:block/name\",\"hide empty value\",536870913],[45,\"~:block/order\",\"aX\",536870913],[45,\"~:block/tags\",3,536870913],[45,\"~:block/title\",\"Hide empty value\",536870913],[45,\"~:block/updated-at\",1786627375717,536870913],[45,\"~:block/uuid\",\"~u00000002-2062-3258-9900-000000000000\",536870913],[45,\"~:db/cardinality\",\"~:db.cardinality/one\",536870913],[45,\"~:db/ident\",\"~:logseq.property/hide-empty-value\",536870913],[45,\"~:db/index\",true,536870913],[45,\"~:logseq.property/built-in?\",true,536870913],[45,\"~:logseq.property/description\",46,536870913],[45,\"~:logseq.property/public?\",true,536870913],[45,\"~:logseq.property/type\",\"~:checkbox\",536870913],[45,\"~:logseq.property/view-context\",\"~:property\",536870913],[46,\"^1\",1786627375718,536870913],[46,\"^3\",\"aY\",536870913]]]", + "addresses": null + }, + { + "addr": 1000035, + "content": "[\"^ \",\"~:keys\",[[46,\"~:block/page\",45,536870913],[46,\"~:block/parent\",45,536870913],[46,\"~:block/title\",\"Hides a property's value on any node when empty e.g. when a property appears on a node through a tag.\",536870913],[46,\"~:block/updated-at\",1786627375718,536870913],[46,\"~:block/uuid\",\"~u00000004-1118-8585-8000-000000000000\",536870913],[46,\"~:logseq.property/built-in?\",true,536870913],[46,\"~:logseq.property/created-from-property\",35,536870913],[47,\"~:block/created-at\",1786627375718,536870913],[47,\"~:block/name\",\"hide from node\",536870913],[47,\"~:block/order\",\"aZ\",536870913],[47,\"~:block/tags\",3,536870913],[47,\"^3\",\"Hide from Node\",536870913],[47,\"^4\",1786627375718,536870913],[47,\"^5\",\"~u00000002-2610-3727-0000-000000000000\",536870913],[47,\"~:db/cardinality\",\"~:db.cardinality/one\",536870913],[47,\"~:db/ident\",\"~:logseq.property.class/hide-from-node\",536870913],[47,\"~:db/index\",true,536870913]]]", + "addresses": null + }, + { + "addr": 1000036, + "content": "[\"^ \",\"~:keys\",[[47,\"~:logseq.property/built-in?\",true,536870913],[47,\"~:logseq.property/public?\",true,536870913],[47,\"~:logseq.property/type\",\"~:checkbox\",536870913],[47,\"~:logseq.property/view-context\",\"~:class\",536870913],[48,\"~:block/created-at\",1786627375718,536870913],[48,\"~:block/name\",\"query\",536870913],[48,\"~:block/order\",\"aa\",536870913],[48,\"~:block/tags\",3,536870913],[48,\"~:block/title\",\"Query\",536870913],[48,\"~:block/updated-at\",1786627375718,536870913],[48,\"~:block/uuid\",\"~u00000002-9741-4126-0000-000000000000\",536870913],[48,\"~:db/cardinality\",\"~:db.cardinality/one\",536870913],[48,\"~:db/ident\",\"~:logseq.property/query\",536870913],[48,\"~:db/index\",true,536870913],[48,\"~:db/valueType\",\"~:db.type/ref\",536870913],[48,\"^1\",true,536870913],[48,\"~:logseq.property/hide?\",true,536870913],[48,\"^2\",true,536870913]]]", + "addresses": null + }, + { + "addr": 1000037, + "content": "[\"^ \",\"~:keys\",[[48,\"~:logseq.property/type\",\"~:default\",536870913],[48,\"~:logseq.property/view-context\",\"~:block\",536870913],[49,\"~:block/created-at\",1786627375718,536870913],[49,\"~:block/name\",\"page tags\",536870913],[49,\"~:block/order\",\"ab\",536870913],[49,\"~:block/tags\",3,536870913],[49,\"~:block/title\",\"Page Tags\",536870913],[49,\"~:block/updated-at\",1786627375718,536870913],[49,\"~:block/uuid\",\"~u00000002-2133-5311-8500-000000000000\",536870913],[49,\"~:db/cardinality\",\"~:db.cardinality/many\",536870913],[49,\"~:db/ident\",\"~:logseq.property/page-tags\",536870913],[49,\"~:db/index\",true,536870913],[49,\"~:db/valueType\",\"~:db.type/ref\",536870913],[49,\"~:logseq.property/built-in?\",true,536870913],[49,\"~:logseq.property/description\",50,536870913],[49,\"~:logseq.property/public?\",true,536870913],[49,\"^1\",\"~:page\",536870913]]]", + "addresses": null + }, + { + "addr": 1000038, + "content": "[\"^ \",\"~:keys\",[[31,\"~:block/title\",\"Journal date\",536870913],[32,\"~:db/cardinality\",\"~:db.cardinality/one\",536870913],[33,\"~:logseq.property/hide?\",true,536870913],[34,\"~:logseq.property/view-context\",\"~:block\",536870913],[36,\"~:block/order\",\"aO\",536870913],[37,\"^1\",\"Default value\",536870913],[38,\"~:block/updated-at\",1786627375717,536870913],[39,\"^2\",\"~:db.cardinality/many\",536870913],[41,\"~:block/created-at\",1786627375717,536870913],[42,\"^7\",\"aU\",536870913],[43,\"^8\",1786627375717,536870913],[44,\"~:logseq.property/created-from-property\",35,536870913],[46,\"^7\",\"aY\",536870913],[47,\"~:db/index\",true,536870913],[48,\"~:logseq.property/public?\",true,536870913],[49,\"~:logseq.property/type\",\"~:page\",536870913]]]", + "addresses": "[1000022,1000023,1000024,1000025,1000026,1000027,1000028,1000029,1000030,1000031,1000032,1000033,1000034,1000035,1000036,1000037]" + }, + { + "addr": 1000039, + "content": "[\"^ \",\"~:keys\",[[49,\"~:logseq.property/view-context\",\"~:page\",536870913],[50,\"~:block/created-at\",1786627375718,536870913],[50,\"~:block/order\",\"ac\",536870913],[50,\"~:block/page\",49,536870913],[50,\"~:block/parent\",49,536870913],[50,\"~:block/title\",\"Provides a way for a page to associate to another page i.e. backward compatible tagging.\",536870913],[50,\"~:block/updated-at\",1786627375718,536870913],[50,\"~:block/uuid\",\"~u00000004-1223-8671-8000-000000000000\",536870913],[50,\"~:logseq.property/built-in?\",true,536870913],[50,\"~:logseq.property/created-from-property\",35,536870913],[51,\"^3\",1786627375718,536870913],[51,\"~:block/name\",\"background color\",536870913],[51,\"^4\",\"ad\",536870913],[51,\"~:block/tags\",3,536870913],[51,\"^7\",\"Background color\",536870913],[51,\"^8\",1786627375718,536870913],[51,\"^9\",\"~u00000002-5191-2660-6000-000000000000\",536870913]]]", + "addresses": null + }, + { + "addr": 1000040, + "content": "[\"^ \",\"~:keys\",[[51,\"~:db/cardinality\",\"~:db.cardinality/one\",536870913],[51,\"~:db/ident\",\"~:logseq.property/background-color\",536870913],[51,\"~:db/index\",true,536870913],[51,\"~:db/valueType\",\"~:db.type/ref\",536870913],[51,\"~:logseq.property/built-in?\",true,536870913],[51,\"~:logseq.property/hide?\",true,536870913],[51,\"~:logseq.property/type\",\"~:default\",536870913],[52,\"~:block/created-at\",1786627375718,536870913],[52,\"~:block/name\",\"heading\",536870913],[52,\"~:block/order\",\"ae\",536870913],[52,\"~:block/tags\",3,536870913],[52,\"~:block/title\",\"Heading\",536870913],[52,\"~:block/updated-at\",1786627375718,536870913],[52,\"~:block/uuid\",\"~u00000002-1858-7494-1500-000000000000\",536870913],[52,\"^1\",\"^2\",536870913],[52,\"^3\",\"~:logseq.property/heading\",536870913],[52,\"^5\",true,536870913]]]", + "addresses": null + }, + { + "addr": 1000041, + "content": "[\"^ \",\"~:keys\",[[52,\"~:logseq.property/built-in?\",true,536870913],[52,\"~:logseq.property/hide?\",true,536870913],[52,\"~:logseq.property/type\",\"~:any\",536870913],[53,\"~:block/created-at\",1786627375718,536870913],[53,\"~:block/name\",\"asset\",536870913],[53,\"~:block/order\",\"af\",536870913],[53,\"~:block/tags\",3,536870913],[53,\"~:block/title\",\"Asset\",536870913],[53,\"~:block/updated-at\",1786627375718,536870913],[53,\"~:block/uuid\",\"~u00000002-8768-5679-0000-000000000000\",536870913],[53,\"~:db/cardinality\",\"~:db.cardinality/one\",536870913],[53,\"~:db/ident\",\"~:logseq.property/asset\",536870913],[53,\"~:db/index\",true,536870913],[53,\"~:db/valueType\",\"~:db.type/ref\",536870913],[53,\"^1\",true,536870913],[53,\"^2\",true,536870913],[53,\"^3\",\"~:entity\",536870913],[54,\"^5\",1786627375718,536870913]]]", + "addresses": null + }, + { + "addr": 1000042, + "content": "[\"^ \",\"~:keys\",[[54,\"~:block/name\",\"ls-type\",536870913],[54,\"~:block/order\",\"ag\",536870913],[54,\"~:block/tags\",3,536870913],[54,\"~:block/title\",\"ls-type\",536870913],[54,\"~:block/updated-at\",1786627375718,536870913],[54,\"~:block/uuid\",\"~u00000002-3269-7934-5000-000000000000\",536870913],[54,\"~:db/cardinality\",\"~:db.cardinality/one\",536870913],[54,\"~:db/ident\",\"~:logseq.property/ls-type\",536870913],[54,\"~:db/index\",true,536870913],[54,\"~:logseq.property/built-in?\",true,536870913],[54,\"~:logseq.property/hide?\",true,536870913],[54,\"~:logseq.property/type\",\"~:keyword\",536870913],[55,\"~:block/created-at\",1786627375718,536870913],[55,\"^1\",\"annotation type\",536870913],[55,\"^2\",\"ah\",536870913],[55,\"^3\",3,536870913],[55,\"^4\",\"Annotation type\",536870913]]]", + "addresses": null + }, + { + "addr": 1000043, + "content": "[\"^ \",\"~:keys\",[[55,\"~:block/updated-at\",1786627375718,536870913],[55,\"~:block/uuid\",\"~u00000002-9984-3783-2000-000000000000\",536870913],[55,\"~:db/cardinality\",\"~:db.cardinality/one\",536870913],[55,\"~:db/ident\",\"~:logseq.property.pdf/hl-type\",536870913],[55,\"~:db/index\",true,536870913],[55,\"~:logseq.property/built-in?\",true,536870913],[55,\"~:logseq.property/hide?\",true,536870913],[55,\"~:logseq.property/type\",\"~:keyword\",536870913],[56,\"~:block/created-at\",1786627375718,536870913],[56,\"~:block/name\",\"annotation color\",536870913],[56,\"~:block/order\",\"ai\",536870913],[56,\"~:block/tags\",3,536870913],[56,\"~:block/title\",\"Annotation color\",536870913],[56,\"^1\",1786627375718,536870913],[56,\"^2\",\"~u00000002-6747-9388-7000-000000000000\",536870913],[56,\"^3\",\"^4\",536870913],[56,\"^5\",\"~:logseq.property.pdf/hl-color\",536870913]]]", + "addresses": null + }, + { + "addr": 1000044, + "content": "[\"^ \",\"~:keys\",[[56,\"~:db/index\",true,536870913],[56,\"~:db/valueType\",\"~:db.type/ref\",536870913],[56,\"~:logseq.property/built-in?\",true,536870913],[56,\"~:logseq.property/hide?\",true,536870913],[56,\"~:logseq.property/type\",\"~:default\",536870913],[57,\"~:block/closed-value-property\",56,536870913],[57,\"~:block/created-at\",1786627375718,536870913],[57,\"~:block/order\",\"aj\",536870913],[57,\"~:block/page\",56,536870913],[57,\"~:block/parent\",56,536870913],[57,\"~:block/title\",\"yellow\",536870913],[57,\"~:block/updated-at\",1786627375718,536870913],[57,\"~:block/uuid\",\"~u00000002-1752-1030-0200-000000000000\",536870913],[57,\"~:db/ident\",\"~:logseq.property/color.yellow\",536870913],[57,\"^4\",true,536870913],[57,\"~:logseq.property/created-from-property\",56,536870913],[58,\"^8\",56,536870913],[58,\"^9\",1786627375718,536870913]]]", + "addresses": null + }, + { + "addr": 1000045, + "content": "[\"^ \",\"~:keys\",[[58,\"~:block/order\",\"ak\",536870913],[58,\"~:block/page\",56,536870913],[58,\"~:block/parent\",56,536870913],[58,\"~:block/title\",\"red\",536870913],[58,\"~:block/updated-at\",1786627375718,536870913],[58,\"~:block/uuid\",\"~u00000002-4136-2216-2000-000000000000\",536870913],[58,\"~:db/ident\",\"~:logseq.property/color.red\",536870913],[58,\"~:logseq.property/built-in?\",true,536870913],[58,\"~:logseq.property/created-from-property\",56,536870913],[59,\"~:block/closed-value-property\",56,536870913],[59,\"~:block/created-at\",1786627375718,536870913],[59,\"^1\",\"al\",536870913],[59,\"^2\",56,536870913],[59,\"^3\",56,536870913],[59,\"^4\",\"green\",536870913],[59,\"^5\",1786627375718,536870913],[59,\"^6\",\"~u00000002-1992-2016-2600-000000000000\",536870913]]]", + "addresses": null + }, + { + "addr": 1000046, + "content": "[\"^ \",\"~:keys\",[[59,\"~:db/ident\",\"~:logseq.property/color.green\",536870913],[59,\"~:logseq.property/built-in?\",true,536870913],[59,\"~:logseq.property/created-from-property\",56,536870913],[60,\"~:block/closed-value-property\",56,536870913],[60,\"~:block/created-at\",1786627375718,536870913],[60,\"~:block/order\",\"am\",536870913],[60,\"~:block/page\",56,536870913],[60,\"~:block/parent\",56,536870913],[60,\"~:block/title\",\"blue\",536870913],[60,\"~:block/updated-at\",1786627375718,536870913],[60,\"~:block/uuid\",\"~u00000002-1836-0512-4100-000000000000\",536870913],[60,\"^1\",\"~:logseq.property/color.blue\",536870913],[60,\"^3\",true,536870913],[60,\"^4\",56,536870913],[61,\"^5\",56,536870913],[61,\"^6\",1786627375718,536870913],[61,\"^7\",\"an\",536870913],[61,\"^8\",56,536870913]]]", + "addresses": null + }, + { + "addr": 1000047, + "content": "[\"^ \",\"~:keys\",[[61,\"~:block/parent\",56,536870913],[61,\"~:block/title\",\"purple\",536870913],[61,\"~:block/updated-at\",1786627375718,536870913],[61,\"~:block/uuid\",\"~u00000002-1104-3848-5600-000000000000\",536870913],[61,\"~:db/ident\",\"~:logseq.property/color.purple\",536870913],[61,\"~:logseq.property/built-in?\",true,536870913],[61,\"~:logseq.property/created-from-property\",56,536870913],[62,\"~:block/created-at\",1786627375719,536870913],[62,\"~:block/name\",\"annotation page\",536870913],[62,\"~:block/order\",\"ao\",536870913],[62,\"~:block/tags\",3,536870913],[62,\"^2\",\"Annotation page\",536870913],[62,\"^3\",1786627375719,536870913],[62,\"^4\",\"~u00000002-7532-8459-6000-000000000000\",536870913],[62,\"~:db/cardinality\",\"~:db.cardinality/one\",536870913],[62,\"^5\",\"~:logseq.property.pdf/hl-page\",536870913],[62,\"~:db/index\",true,536870913]]]", + "addresses": null + }, + { + "addr": 1000048, + "content": "[\"^ \",\"~:keys\",[[62,\"~:logseq.property/built-in?\",true,536870913],[62,\"~:logseq.property/hide?\",true,536870913],[62,\"~:logseq.property/type\",\"~:raw-number\",536870913],[63,\"~:block/created-at\",1786627375719,536870913],[63,\"~:block/name\",\"annotation image\",536870913],[63,\"~:block/order\",\"ap\",536870913],[63,\"~:block/tags\",3,536870913],[63,\"~:block/title\",\"Annotation image\",536870913],[63,\"~:block/updated-at\",1786627375719,536870913],[63,\"~:block/uuid\",\"~u00000002-1377-6700-9000-000000000000\",536870913],[63,\"~:db/cardinality\",\"~:db.cardinality/one\",536870913],[63,\"~:db/ident\",\"~:logseq.property.pdf/hl-image\",536870913],[63,\"~:db/index\",true,536870913],[63,\"~:db/valueType\",\"~:db.type/ref\",536870913],[63,\"^1\",true,536870913],[63,\"^2\",true,536870913],[63,\"^3\",\"~:entity\",536870913],[64,\"^5\",1786627375719,536870913]]]", + "addresses": null + }, + { + "addr": 1000049, + "content": "[\"^ \",\"~:keys\",[[64,\"~:block/name\",\"annotation data\",536870913],[64,\"~:block/order\",\"aq\",536870913],[64,\"~:block/tags\",3,536870913],[64,\"~:block/title\",\"Annotation data\",536870913],[64,\"~:block/updated-at\",1786627375719,536870913],[64,\"~:block/uuid\",\"~u00000002-5458-2940-2000-000000000000\",536870913],[64,\"~:db/cardinality\",\"~:db.cardinality/one\",536870913],[64,\"~:db/ident\",\"~:logseq.property.pdf/hl-value\",536870913],[64,\"~:db/index\",true,536870913],[64,\"~:logseq.property/built-in?\",true,536870913],[64,\"~:logseq.property/hide?\",true,536870913],[64,\"~:logseq.property/type\",\"~:map\",536870913],[65,\"~:block/created-at\",1786627375719,536870913],[65,\"^1\",\"list type\",536870913],[65,\"^2\",\"ar\",536870913],[65,\"^3\",3,536870913],[65,\"^4\",\"List type\",536870913]]]", + "addresses": null + }, + { + "addr": 1000050, + "content": "[\"^ \",\"~:keys\",[[65,\"~:block/updated-at\",1786627375719,536870913],[65,\"~:block/uuid\",\"~u00000002-6078-1711-1000-000000000000\",536870913],[65,\"~:db/cardinality\",\"~:db.cardinality/one\",536870913],[65,\"~:db/ident\",\"~:logseq.property/order-list-type\",536870913],[65,\"~:db/index\",true,536870913],[65,\"~:db/valueType\",\"~:db.type/ref\",536870913],[65,\"~:logseq.property/built-in?\",true,536870913],[65,\"~:logseq.property/hide?\",true,536870913],[65,\"~:logseq.property/type\",\"~:default\",536870913],[66,\"~:block/created-at\",1786627375719,536870913],[66,\"~:block/name\",\"included references\",536870913],[66,\"~:block/order\",\"as\",536870913],[66,\"~:block/tags\",3,536870913],[66,\"~:block/title\",\"Included references\",536870913],[66,\"^1\",1786627375719,536870913],[66,\"^2\",\"~u00000002-1680-5777-0300-000000000000\",536870913],[66,\"^3\",\"~:db.cardinality/many\",536870913]]]", + "addresses": null + }, + { + "addr": 1000051, + "content": "[\"^ \",\"~:keys\",[[66,\"~:db/ident\",\"~:logseq.property.linked-references/includes\",536870913],[66,\"~:db/index\",true,536870913],[66,\"~:db/valueType\",\"~:db.type/ref\",536870913],[66,\"~:logseq.property/built-in?\",true,536870913],[66,\"~:logseq.property/hide?\",true,536870913],[66,\"~:logseq.property/type\",\"~:node\",536870913],[67,\"~:block/created-at\",1786627375719,536870913],[67,\"~:block/name\",\"excluded references\",536870913],[67,\"~:block/order\",\"at\",536870913],[67,\"~:block/tags\",3,536870913],[67,\"~:block/title\",\"Excluded references\",536870913],[67,\"~:block/updated-at\",1786627375719,536870913],[67,\"~:block/uuid\",\"~u00000002-2426-7588-9000-000000000000\",536870913],[67,\"~:db/cardinality\",\"~:db.cardinality/many\",536870913],[67,\"^1\",\"~:logseq.property.linked-references/excludes\",536870913],[67,\"^3\",true,536870913],[67,\"^4\",\"^5\",536870913]]]", + "addresses": null + }, + { + "addr": 1000052, + "content": "[\"^ \",\"~:keys\",[[67,\"~:logseq.property/built-in?\",true,536870913],[67,\"~:logseq.property/hide?\",true,536870913],[67,\"~:logseq.property/type\",\"~:node\",536870913],[68,\"~:block/created-at\",1786627375719,536870913],[68,\"~:block/name\",\"commented blocks\",536870913],[68,\"~:block/order\",\"au\",536870913],[68,\"~:block/tags\",3,536870913],[68,\"~:block/title\",\"Commented blocks\",536870913],[68,\"~:block/updated-at\",1786627375719,536870913],[68,\"~:block/uuid\",\"~u00000002-2143-9561-0900-000000000000\",536870913],[68,\"~:db/cardinality\",\"~:db.cardinality/many\",536870913],[68,\"~:db/ident\",\"~:logseq.property.comments/blocks\",536870913],[68,\"~:db/index\",true,536870913],[68,\"~:db/valueType\",\"~:db.type/ref\",536870913],[68,\"^1\",true,536870913],[68,\"^2\",true,536870913],[68,\"~:logseq.property/public?\",false,536870913],[68,\"^3\",\"^4\",536870913]]]", + "addresses": null + }, + { + "addr": 1000053, + "content": "[\"^ \",\"~:keys\",[[69,\"~:block/created-at\",1786627375719,536870913],[69,\"~:block/name\",\"title format\",536870913],[69,\"~:block/order\",\"av\",536870913],[69,\"~:block/tags\",3,536870913],[69,\"~:block/title\",\"Title Format\",536870913],[69,\"~:block/updated-at\",1786627375719,536870913],[69,\"~:block/uuid\",\"~u00000002-1536-4979-5400-000000000000\",536870913],[69,\"~:db/cardinality\",\"~:db.cardinality/one\",536870913],[69,\"~:db/ident\",\"~:logseq.property.journal/title-format\",536870913],[69,\"~:db/index\",true,536870913],[69,\"~:logseq.property/built-in?\",true,536870913],[69,\"~:logseq.property/public?\",false,536870913],[69,\"~:logseq.property/type\",\"~:string\",536870913],[70,\"^1\",1786627375719,536870913],[70,\"^2\",\"choice checkbox state\",536870913],[70,\"^3\",\"aw\",536870913],[70,\"^4\",3,536870913]]]", + "addresses": null + }, + { + "addr": 1000054, + "content": "[\"^ \",\"~:keys\",[[70,\"~:block/title\",\"Choice checkbox state\",536870913],[70,\"~:block/updated-at\",1786627375719,536870913],[70,\"~:block/uuid\",\"~u00000002-1272-4228-6300-000000000000\",536870913],[70,\"~:db/cardinality\",\"~:db.cardinality/one\",536870913],[70,\"~:db/ident\",\"~:logseq.property/choice-checkbox-state\",536870913],[70,\"~:db/index\",true,536870913],[70,\"~:logseq.property/built-in?\",true,536870913],[70,\"~:logseq.property/hide?\",true,536870913],[70,\"~:logseq.property/type\",\"~:checkbox\",536870913],[71,\"~:block/created-at\",1786627375719,536870913],[71,\"~:block/name\",\"choice classes\",536870913],[71,\"~:block/order\",\"ax\",536870913],[71,\"~:block/tags\",3,536870913],[71,\"^1\",\"Choice classes\",536870913],[71,\"^2\",1786627375719,536870913],[71,\"^3\",\"~u00000002-7629-6621-9000-000000000000\",536870913],[71,\"^4\",\"~:db.cardinality/many\",536870913]]]", + "addresses": null + }, + { + "addr": 1000055, + "content": "[\"^ \",\"~:keys\",[[51,\"~:block/uuid\",\"~u00000002-5191-2660-6000-000000000000\",536870913],[52,\"~:db/index\",true,536870913],[54,\"~:block/created-at\",1786627375718,536870913],[55,\"~:block/title\",\"Annotation type\",536870913],[56,\"~:db/ident\",\"~:logseq.property.pdf/hl-color\",536870913],[58,\"^3\",1786627375718,536870913],[59,\"^1\",\"~u00000002-1992-2016-2600-000000000000\",536870913],[61,\"~:block/page\",56,536870913],[62,\"^2\",true,536870913],[64,\"^3\",1786627375719,536870913],[65,\"^4\",\"List type\",536870913],[66,\"~:db/cardinality\",\"~:db.cardinality/many\",536870913],[67,\"~:db/valueType\",\"~:db.type/ref\",536870913],[68,\"~:logseq.property/type\",\"~:node\",536870913],[70,\"~:block/tags\",3,536870913],[71,\"^8\",\"^9\",536870913]]]", + "addresses": "[1000039,1000040,1000041,1000042,1000043,1000044,1000045,1000046,1000047,1000048,1000049,1000050,1000051,1000052,1000053,1000054]" + }, + { + "addr": 1000056, + "content": "[\"^ \",\"~:keys\",[[71,\"~:db/ident\",\"~:logseq.property/choice-classes\",536870913],[71,\"~:db/index\",true,536870913],[71,\"~:db/valueType\",\"~:db.type/ref\",536870913],[71,\"~:logseq.property/built-in?\",true,536870913],[71,\"~:logseq.property/hide?\",true,536870913],[71,\"~:logseq.property/public?\",false,536870913],[71,\"~:logseq.property/type\",\"~:class\",536870913],[71,\"~:logseq.property/view-context\",\"~:never\",536870913],[72,\"~:block/created-at\",1786627375719,536870913],[72,\"~:block/name\",\"choice exclusions\",536870913],[72,\"~:block/order\",\"ay\",536870913],[72,\"~:block/tags\",3,536870913],[72,\"~:block/title\",\"Choice exclusions\",536870913],[72,\"~:block/updated-at\",1786627375719,536870913],[72,\"~:block/uuid\",\"~u00000002-1233-5229-4600-000000000000\",536870913],[72,\"~:db/cardinality\",\"~:db.cardinality/many\",536870913],[72,\"^1\",\"~:logseq.property/choice-exclusions\",536870913]]]", + "addresses": null + }, + { + "addr": 1000057, + "content": "[\"^ \",\"~:keys\",[[72,\"~:db/index\",true,536870913],[72,\"~:db/valueType\",\"~:db.type/ref\",536870913],[72,\"~:logseq.property/built-in?\",true,536870913],[72,\"~:logseq.property/hide?\",true,536870913],[72,\"~:logseq.property/public?\",false,536870913],[72,\"~:logseq.property/type\",\"~:node\",536870913],[72,\"~:logseq.property/view-context\",\"~:never\",536870913],[73,\"~:block/created-at\",1786627375719,536870913],[73,\"~:block/name\",\"properties displayed as checkbox\",536870913],[73,\"~:block/order\",\"az\",536870913],[73,\"~:block/tags\",3,536870913],[73,\"~:block/title\",\"Properties displayed as checkbox\",536870913],[73,\"~:block/updated-at\",1786627375719,536870913],[73,\"~:block/uuid\",\"~u00000002-3215-3256-9000-000000000000\",536870913],[73,\"~:db/cardinality\",\"~:db.cardinality/many\",536870913],[73,\"~:db/ident\",\"~:logseq.property/checkbox-display-properties\",536870913],[73,\"^1\",true,536870913]]]", + "addresses": null + }, + { + "addr": 1000058, + "content": "[\"^ \",\"~:keys\",[[73,\"~:db/valueType\",\"~:db.type/ref\",536870913],[73,\"~:logseq.property/built-in?\",true,536870913],[73,\"~:logseq.property/hide?\",true,536870913],[73,\"~:logseq.property/type\",\"~:property\",536870913],[74,\"~:block/created-at\",1786627375719,536870913],[74,\"~:block/name\",\"status\",536870913],[74,\"~:block/order\",\"b00\",536870913],[74,\"~:block/tags\",3,536870913],[74,\"~:block/title\",\"Status\",536870913],[74,\"~:block/updated-at\",1786627375719,536870913],[74,\"~:block/uuid\",\"~u00000002-9072-1685-3000-000000000000\",536870913],[74,\"~:db/cardinality\",\"~:db.cardinality/one\",536870913],[74,\"~:db/ident\",\"~:logseq.property/status\",536870913],[74,\"~:db/index\",true,536870913],[74,\"^1\",\"^2\",536870913],[74,\"^3\",true,536870913],[74,\"~:logseq.property/default-value\",76,536870913],[74,\"~:logseq.property/enable-history?\",true,536870913],[74,\"~:logseq.property/hide-empty-value\",true,536870913],[74,\"~:logseq.property/public?\",true,536870913],[74,\"^5\",\"~:default\",536870913]]]", + "addresses": null + }, + { + "addr": 1000059, + "content": "[\"^ \",\"~:keys\",[[74,\"~:logseq.property/ui-position\",\"~:block-left\",536870913],[75,\"~:block/closed-value-property\",74,536870913],[75,\"~:block/created-at\",1786627375719,536870913],[75,\"~:block/order\",\"b01\",536870913],[75,\"~:block/page\",74,536870913],[75,\"~:block/parent\",74,536870913],[75,\"~:block/title\",\"Backlog\",536870913],[75,\"~:block/updated-at\",1786627375719,536870913],[75,\"~:block/uuid\",\"~u00000002-7233-3491-0000-000000000000\",536870913],[75,\"~:db/ident\",\"~:logseq.property/status.backlog\",536870913],[75,\"~:logseq.property/built-in?\",true,536870913],[75,\"~:logseq.property/created-from-property\",74,536870913],[75,\"~:logseq.property/icon\",[\"^ \",\"~:type\",\"~:tabler-icon\",\"~:id\",\"Backlog\"],536870913],[76,\"^3\",74,536870913],[76,\"^4\",1786627375719,536870913],[76,\"^5\",\"b02\",536870913],[76,\"^6\",74,536870913]]]", + "addresses": null + }, + { + "addr": 1000060, + "content": "[\"^ \",\"~:keys\",[[76,\"~:block/parent\",74,536870913],[76,\"~:block/title\",\"Todo\",536870913],[76,\"~:block/updated-at\",1786627375719,536870913],[76,\"~:block/uuid\",\"~u00000002-1615-5853-7700-000000000000\",536870913],[76,\"~:db/ident\",\"~:logseq.property/status.todo\",536870913],[76,\"~:logseq.property/built-in?\",true,536870913],[76,\"~:logseq.property/choice-checkbox-state\",false,536870913],[76,\"~:logseq.property/created-from-property\",74,536870913],[76,\"~:logseq.property/icon\",[\"^ \",\"~:type\",\"~:tabler-icon\",\"~:id\",\"Todo\"],536870913],[77,\"~:block/closed-value-property\",74,536870913],[77,\"~:block/created-at\",1786627375719,536870913],[77,\"~:block/order\",\"b03\",536870913],[77,\"~:block/page\",74,536870913],[77,\"^1\",74,536870913],[77,\"^2\",\"Doing\",536870913],[77,\"^3\",1786627375719,536870913],[77,\"^4\",\"~u00000002-1840-1229-0800-000000000000\",536870913]]]", + "addresses": null + }, + { + "addr": 1000061, + "content": "[\"^ \",\"~:keys\",[[77,\"~:db/ident\",\"~:logseq.property/status.doing\",536870913],[77,\"~:logseq.property/built-in?\",true,536870913],[77,\"~:logseq.property/created-from-property\",74,536870913],[77,\"~:logseq.property/icon\",[\"^ \",\"~:type\",\"~:tabler-icon\",\"~:id\",\"InProgress50\"],536870913],[78,\"~:block/closed-value-property\",74,536870913],[78,\"~:block/created-at\",1786627375719,536870913],[78,\"~:block/order\",\"b04\",536870913],[78,\"~:block/page\",74,536870913],[78,\"~:block/parent\",74,536870913],[78,\"~:block/title\",\"In Review\",536870913],[78,\"~:block/updated-at\",1786627375719,536870913],[78,\"~:block/uuid\",\"~u00000002-1870-0014-4300-000000000000\",536870913],[78,\"^1\",\"~:logseq.property/status.in-review\",536870913],[78,\"^3\",true,536870913],[78,\"^4\",74,536870913],[78,\"^5\",[\"^ \",\"^6\",\"^7\",\"^8\",\"InReview\"],536870913],[79,\"^9\",74,536870913],[79,\"^:\",1786627375719,536870913]]]", + "addresses": null + }, + { + "addr": 1000062, + "content": "[\"^ \",\"~:keys\",[[79,\"~:block/order\",\"b05\",536870913],[79,\"~:block/page\",74,536870913],[79,\"~:block/parent\",74,536870913],[79,\"~:block/title\",\"Done\",536870913],[79,\"~:block/updated-at\",1786627375719,536870913],[79,\"~:block/uuid\",\"~u00000002-1827-5820-8200-000000000000\",536870913],[79,\"~:db/ident\",\"~:logseq.property/status.done\",536870913],[79,\"~:logseq.property/built-in?\",true,536870913],[79,\"~:logseq.property/choice-checkbox-state\",true,536870913],[79,\"~:logseq.property/created-from-property\",74,536870913],[79,\"~:logseq.property/icon\",[\"^ \",\"~:type\",\"~:tabler-icon\",\"~:id\",\"Done\"],536870913],[80,\"~:block/closed-value-property\",74,536870913],[80,\"~:block/created-at\",1786627375720,536870913],[80,\"^1\",\"b06\",536870913],[80,\"^2\",74,536870913],[80,\"^3\",74,536870913],[80,\"^4\",\"Canceled\",536870913]]]", + "addresses": null + }, + { + "addr": 1000063, + "content": "[\"^ \",\"~:keys\",[[80,\"~:block/updated-at\",1786627375720,536870913],[80,\"~:block/uuid\",\"~u00000002-7526-4326-2000-000000000000\",536870913],[80,\"~:db/ident\",\"~:logseq.property/status.canceled\",536870913],[80,\"~:logseq.property/built-in?\",true,536870913],[80,\"~:logseq.property/created-from-property\",74,536870913],[80,\"~:logseq.property/icon\",[\"^ \",\"~:type\",\"~:tabler-icon\",\"~:id\",\"Cancelled\"],536870913],[81,\"~:block/created-at\",1786627375720,536870913],[81,\"~:block/name\",\"priority\",536870913],[81,\"~:block/order\",\"b07\",536870913],[81,\"~:block/tags\",3,536870913],[81,\"~:block/title\",\"Priority\",536870913],[81,\"^1\",1786627375720,536870913],[81,\"^2\",\"~u00000002-2392-2841-1000-000000000000\",536870913],[81,\"~:db/cardinality\",\"~:db.cardinality/one\",536870913],[81,\"^3\",\"~:logseq.property/priority\",536870913],[81,\"~:db/index\",true,536870913],[81,\"~:db/valueType\",\"~:db.type/ref\",536870913]]]", + "addresses": null + }, + { + "addr": 1000064, + "content": "[\"^ \",\"~:keys\",[[81,\"~:logseq.property/built-in?\",true,536870913],[81,\"~:logseq.property/enable-history?\",true,536870913],[81,\"~:logseq.property/hide-empty-value\",true,536870913],[81,\"~:logseq.property/public?\",true,536870913],[81,\"~:logseq.property/type\",\"~:default\",536870913],[81,\"~:logseq.property/ui-position\",\"~:block-left\",536870913],[82,\"~:block/closed-value-property\",81,536870913],[82,\"~:block/created-at\",1786627375720,536870913],[82,\"~:block/order\",\"b08\",536870913],[82,\"~:block/page\",81,536870913],[82,\"~:block/parent\",81,536870913],[82,\"~:block/title\",\"Low\",536870913],[82,\"~:block/updated-at\",1786627375720,536870913],[82,\"~:block/uuid\",\"~u00000002-2107-4537-4800-000000000000\",536870913],[82,\"~:db/ident\",\"~:logseq.property/priority.low\",536870913],[82,\"^1\",true,536870913],[82,\"~:logseq.property/created-from-property\",81,536870913],[82,\"~:logseq.property/icon\",[\"^ \",\"~:type\",\"~:tabler-icon\",\"~:id\",\"priorityLvlLow\"],536870913],[83,\"^9\",81,536870913],[83,\"^:\",1786627375720,536870913]]]", + "addresses": null + }, + { + "addr": 1000065, + "content": "[\"^ \",\"~:keys\",[[83,\"~:block/order\",\"b09\",536870913],[83,\"~:block/page\",81,536870913],[83,\"~:block/parent\",81,536870913],[83,\"~:block/title\",\"Medium\",536870913],[83,\"~:block/updated-at\",1786627375720,536870913],[83,\"~:block/uuid\",\"~u00000002-1829-3222-7800-000000000000\",536870913],[83,\"~:db/ident\",\"~:logseq.property/priority.medium\",536870913],[83,\"~:logseq.property/built-in?\",true,536870913],[83,\"~:logseq.property/created-from-property\",81,536870913],[83,\"~:logseq.property/icon\",[\"^ \",\"~:type\",\"~:tabler-icon\",\"~:id\",\"priorityLvlMedium\"],536870913],[84,\"~:block/closed-value-property\",81,536870913],[84,\"~:block/created-at\",1786627375720,536870913],[84,\"^1\",\"b0A\",536870913],[84,\"^2\",81,536870913],[84,\"^3\",81,536870913],[84,\"^4\",\"High\",536870913],[84,\"^5\",1786627375720,536870913]]]", + "addresses": null + }, + { + "addr": 1000066, + "content": "[\"^ \",\"~:keys\",[[84,\"~:block/uuid\",\"~u00000002-5672-2766-8000-000000000000\",536870913],[84,\"~:db/ident\",\"~:logseq.property/priority.high\",536870913],[84,\"~:logseq.property/built-in?\",true,536870913],[84,\"~:logseq.property/created-from-property\",81,536870913],[84,\"~:logseq.property/icon\",[\"^ \",\"~:type\",\"~:tabler-icon\",\"~:id\",\"priorityLvlHigh\"],536870913],[85,\"~:block/closed-value-property\",81,536870913],[85,\"~:block/created-at\",1786627375720,536870913],[85,\"~:block/order\",\"b0B\",536870913],[85,\"~:block/page\",81,536870913],[85,\"~:block/parent\",81,536870913],[85,\"~:block/title\",\"Urgent\",536870913],[85,\"~:block/updated-at\",1786627375720,536870913],[85,\"^1\",\"~u00000002-1996-4346-6700-000000000000\",536870913],[85,\"^2\",\"~:logseq.property/priority.urgent\",536870913],[85,\"^4\",true,536870913],[85,\"^5\",81,536870913],[85,\"^6\",[\"^ \",\"^7\",\"^8\",\"^9\",\"priorityLvlUrgent\"],536870913],[86,\"^;\",1786627375720,536870913]]]", + "addresses": null + }, + { + "addr": 1000067, + "content": "[\"^ \",\"~:keys\",[[86,\"~:block/name\",\"deadline\",536870913],[86,\"~:block/order\",\"b0C\",536870913],[86,\"~:block/tags\",3,536870913],[86,\"~:block/title\",\"Deadline\",536870913],[86,\"~:block/updated-at\",1786627375720,536870913],[86,\"~:block/uuid\",\"~u00000002-1685-9016-0400-000000000000\",536870913],[86,\"~:db/cardinality\",\"~:db.cardinality/one\",536870913],[86,\"~:db/ident\",\"~:logseq.property/deadline\",536870913],[86,\"~:db/index\",true,536870913],[86,\"~:logseq.property/built-in?\",true,536870913],[86,\"~:logseq.property/description\",87,536870913],[86,\"~:logseq.property/hide-empty-value\",true,536870913],[86,\"~:logseq.property/public?\",true,536870913],[86,\"~:logseq.property/type\",\"~:datetime\",536870913],[86,\"~:logseq.property/ui-position\",\"~:block-below\",536870913],[87,\"~:block/created-at\",1786627375720,536870913],[87,\"^2\",\"b0D\",536870913]]]", + "addresses": null + }, + { + "addr": 1000068, + "content": "[\"^ \",\"~:keys\",[[87,\"~:block/page\",86,536870913],[87,\"~:block/parent\",86,536870913],[87,\"~:block/title\",\"Use it to finish something at a specific date(time).\",536870913],[87,\"~:block/updated-at\",1786627375720,536870913],[87,\"~:block/uuid\",\"~u00000004-1356-3664-2200-000000000000\",536870913],[87,\"~:logseq.property/built-in?\",true,536870913],[87,\"~:logseq.property/created-from-property\",35,536870913],[88,\"~:block/created-at\",1786627375720,536870913],[88,\"~:block/name\",\"scheduled\",536870913],[88,\"~:block/order\",\"b0E\",536870913],[88,\"~:block/tags\",3,536870913],[88,\"^3\",\"Scheduled\",536870913],[88,\"^4\",1786627375720,536870913],[88,\"^5\",\"~u00000002-1644-5209-4300-000000000000\",536870913],[88,\"~:db/cardinality\",\"~:db.cardinality/one\",536870913],[88,\"~:db/ident\",\"~:logseq.property/scheduled\",536870913],[88,\"~:db/index\",true,536870913]]]", + "addresses": null + }, + { + "addr": 1000069, + "content": "[\"^ \",\"~:keys\",[[88,\"~:logseq.property/built-in?\",true,536870913],[88,\"~:logseq.property/description\",89,536870913],[88,\"~:logseq.property/hide-empty-value\",true,536870913],[88,\"~:logseq.property/public?\",true,536870913],[88,\"~:logseq.property/type\",\"~:datetime\",536870913],[88,\"~:logseq.property/ui-position\",\"~:block-below\",536870913],[89,\"~:block/created-at\",1786627375720,536870913],[89,\"~:block/order\",\"b0F\",536870913],[89,\"~:block/page\",88,536870913],[89,\"~:block/parent\",88,536870913],[89,\"~:block/title\",\"Use it to plan something to start at a specific date(time).\",536870913],[89,\"~:block/updated-at\",1786627375720,536870913],[89,\"~:block/uuid\",\"~u00000004-9817-6380-0000-000000000000\",536870913],[89,\"^1\",true,536870913],[89,\"~:logseq.property/created-from-property\",35,536870913],[90,\"^9\",1786627375720,536870913],[90,\"~:block/name\",\"repeating recur frequency\",536870913],[90,\"^:\",\"b0G\",536870913]]]", + "addresses": null + }, + { + "addr": 1000070, + "content": "[\"^ \",\"~:keys\",[[90,\"~:block/tags\",3,536870913],[90,\"~:block/title\",\"Repeating recur frequency\",536870913],[90,\"~:block/updated-at\",1786627375720,536870913],[90,\"~:block/uuid\",\"~u00000002-8716-1592-2000-000000000000\",536870913],[90,\"~:db/cardinality\",\"~:db.cardinality/one\",536870913],[90,\"~:db/ident\",\"~:logseq.property.repeat/recur-frequency\",536870913],[90,\"~:db/index\",true,536870913],[90,\"~:db/valueType\",\"~:db.type/ref\",536870913],[90,\"~:logseq.property/built-in?\",true,536870913],[90,\"~:logseq.property/default-value\",91,536870913],[90,\"~:logseq.property/hide-empty-value\",true,536870913],[90,\"~:logseq.property/public?\",false,536870913],[90,\"~:logseq.property/type\",\"~:number\",536870913],[91,\"~:block/created-at\",1786627375720,536870913],[91,\"~:block/order\",\"b0H\",536870913],[91,\"~:block/page\",90,536870913],[91,\"~:block/parent\",90,536870913]]]", + "addresses": null + }, + { + "addr": 1000071, + "content": "[\"^ \",\"~:keys\",[[91,\"~:block/updated-at\",1786627375720,536870913],[91,\"~:block/uuid\",\"~u00000004-1654-1034-2600-000000000000\",536870913],[91,\"~:logseq.property/built-in?\",true,536870913],[91,\"~:logseq.property/created-from-property\",90,536870913],[91,\"~:logseq.property/value\",1,536870913],[92,\"~:block/created-at\",1786627375720,536870913],[92,\"~:block/name\",\"repeating recur unit\",536870913],[92,\"~:block/order\",\"b0I\",536870913],[92,\"~:block/tags\",3,536870913],[92,\"~:block/title\",\"Repeating recur unit\",536870913],[92,\"^1\",1786627375720,536870913],[92,\"^2\",\"~u00000002-6903-0624-7000-000000000000\",536870913],[92,\"~:db/cardinality\",\"~:db.cardinality/one\",536870913],[92,\"~:db/ident\",\"~:logseq.property.repeat/recur-unit\",536870913],[92,\"~:db/index\",true,536870913],[92,\"~:db/valueType\",\"~:db.type/ref\",536870913],[92,\"^3\",true,536870913],[92,\"~:logseq.property/default-value\",95,536870913],[92,\"~:logseq.property/hide-empty-value\",true,536870913],[92,\"~:logseq.property/public?\",false,536870913]]]", + "addresses": null + }, + { + "addr": 1000072, + "content": "[\"^ \",\"~:keys\",[[72,\"~:db/ident\",\"~:logseq.property/choice-exclusions\",536870913],[73,\"~:db/index\",true,536870913],[74,\"~:logseq.property/type\",\"~:default\",536870913],[76,\"~:block/page\",74,536870913],[77,\"~:block/uuid\",\"~u00000002-1840-1229-0800-000000000000\",536870913],[79,\"~:block/created-at\",1786627375719,536870913],[80,\"~:block/title\",\"Canceled\",536870913],[81,\"~:db/valueType\",\"~:db.type/ref\",536870913],[83,\"^8\",1786627375720,536870913],[84,\"~:block/updated-at\",1786627375720,536870913],[86,\"^8\",1786627375720,536870913],[87,\"~:block/order\",\"b0D\",536870913],[88,\"^3\",true,536870913],[90,\"^=\",\"b0G\",536870913],[91,\"~:block/parent\",90,536870913],[92,\"~:logseq.property/public?\",false,536870913]]]", + "addresses": "[1000056,1000057,1000058,1000059,1000060,1000061,1000062,1000063,1000064,1000065,1000066,1000067,1000068,1000069,1000070,1000071]" + }, + { + "addr": 1000073, + "content": "[\"^ \",\"~:keys\",[[92,\"~:logseq.property/type\",\"~:default\",536870913],[93,\"~:block/closed-value-property\",92,536870913],[93,\"~:block/created-at\",1786627375720,536870913],[93,\"~:block/order\",\"b0J\",536870913],[93,\"~:block/page\",92,536870913],[93,\"~:block/parent\",92,536870913],[93,\"~:block/title\",\"Minute\",536870913],[93,\"~:block/updated-at\",1786627375720,536870913],[93,\"~:block/uuid\",\"~u00000002-1513-6550-8500-000000000000\",536870913],[93,\"~:db/ident\",\"~:logseq.property.repeat/recur-unit.minute\",536870913],[93,\"~:logseq.property/built-in?\",true,536870913],[93,\"~:logseq.property/created-from-property\",92,536870913],[94,\"^3\",92,536870913],[94,\"^4\",1786627375720,536870913],[94,\"^5\",\"b0K\",536870913],[94,\"^6\",92,536870913],[94,\"^7\",92,536870913]]]", + "addresses": null + }, + { + "addr": 1000074, + "content": "[\"^ \",\"~:keys\",[[94,\"~:block/title\",\"Hour\",536870913],[94,\"~:block/updated-at\",1786627375720,536870913],[94,\"~:block/uuid\",\"~u00000002-1438-8849-5400-000000000000\",536870913],[94,\"~:db/ident\",\"~:logseq.property.repeat/recur-unit.hour\",536870913],[94,\"~:logseq.property/built-in?\",true,536870913],[94,\"~:logseq.property/created-from-property\",92,536870913],[95,\"~:block/closed-value-property\",92,536870913],[95,\"~:block/created-at\",1786627375721,536870913],[95,\"~:block/order\",\"b0L\",536870913],[95,\"~:block/page\",92,536870913],[95,\"~:block/parent\",92,536870913],[95,\"^1\",\"Day\",536870913],[95,\"^2\",1786627375721,536870913],[95,\"^3\",\"~u00000002-3924-1785-8000-000000000000\",536870913],[95,\"^4\",\"~:logseq.property.repeat/recur-unit.day\",536870913],[95,\"^6\",true,536870913],[95,\"^7\",92,536870913],[96,\"^8\",92,536870913]]]", + "addresses": null + }, + { + "addr": 1000075, + "content": "[\"^ \",\"~:keys\",[[96,\"~:block/created-at\",1786627375721,536870913],[96,\"~:block/order\",\"b0M\",536870913],[96,\"~:block/page\",92,536870913],[96,\"~:block/parent\",92,536870913],[96,\"~:block/title\",\"Week\",536870913],[96,\"~:block/updated-at\",1786627375721,536870913],[96,\"~:block/uuid\",\"~u00000002-2130-9244-4900-000000000000\",536870913],[96,\"~:db/ident\",\"~:logseq.property.repeat/recur-unit.week\",536870913],[96,\"~:logseq.property/built-in?\",true,536870913],[96,\"~:logseq.property/created-from-property\",92,536870913],[97,\"~:block/closed-value-property\",92,536870913],[97,\"^1\",1786627375721,536870913],[97,\"^2\",\"b0N\",536870913],[97,\"^3\",92,536870913],[97,\"^4\",92,536870913],[97,\"^5\",\"Month\",536870913],[97,\"^6\",1786627375721,536870913]]]", + "addresses": null + }, + { + "addr": 1000076, + "content": "[\"^ \",\"~:keys\",[[97,\"~:block/uuid\",\"~u00000002-2073-3937-9700-000000000000\",536870913],[97,\"~:db/ident\",\"~:logseq.property.repeat/recur-unit.month\",536870913],[97,\"~:logseq.property/built-in?\",true,536870913],[97,\"~:logseq.property/created-from-property\",92,536870913],[98,\"~:block/closed-value-property\",92,536870913],[98,\"~:block/created-at\",1786627375721,536870913],[98,\"~:block/order\",\"b0O\",536870913],[98,\"~:block/page\",92,536870913],[98,\"~:block/parent\",92,536870913],[98,\"~:block/title\",\"Year\",536870913],[98,\"~:block/updated-at\",1786627375721,536870913],[98,\"^1\",\"~u00000002-1520-4385-2400-000000000000\",536870913],[98,\"^2\",\"~:logseq.property.repeat/recur-unit.year\",536870913],[98,\"^4\",true,536870913],[98,\"^5\",92,536870913],[99,\"^7\",1786627375721,536870913],[99,\"~:block/name\",\"node repeats?\",536870913],[99,\"^8\",\"b0P\",536870913]]]", + "addresses": null + }, + { + "addr": 1000077, + "content": "[\"^ \",\"~:keys\",[[99,\"~:block/tags\",3,536870913],[99,\"~:block/title\",\"Node Repeats?\",536870913],[99,\"~:block/updated-at\",1786627375721,536870913],[99,\"~:block/uuid\",\"~u00000002-1908-1217-8900-000000000000\",536870913],[99,\"~:db/cardinality\",\"~:db.cardinality/one\",536870913],[99,\"~:db/ident\",\"~:logseq.property.repeat/repeated?\",536870913],[99,\"~:db/index\",true,536870913],[99,\"~:logseq.property/built-in?\",true,536870913],[99,\"~:logseq.property/hide?\",true,536870913],[99,\"~:logseq.property/type\",\"~:checkbox\",536870913],[100,\"~:block/created-at\",1786627375721,536870913],[100,\"~:block/name\",\"repeating type\",536870913],[100,\"~:block/order\",\"b0Q\",536870913],[100,\"^1\",3,536870913],[100,\"^2\",\"Repeating type\",536870913],[100,\"^3\",1786627375721,536870913],[100,\"^4\",\"~u00000002-1984-2972-8100-000000000000\",536870913]]]", + "addresses": null + }, + { + "addr": 1000078, + "content": "[\"^ \",\"~:keys\",[[100,\"~:db/cardinality\",\"~:db.cardinality/one\",536870913],[100,\"~:db/ident\",\"~:logseq.property.repeat/repeat-type\",536870913],[100,\"~:db/index\",true,536870913],[100,\"~:db/valueType\",\"~:db.type/ref\",536870913],[100,\"~:logseq.property/built-in?\",true,536870913],[100,\"~:logseq.property/default-value\",103,536870913],[100,\"~:logseq.property/hide-empty-value\",true,536870913],[100,\"~:logseq.property/public?\",false,536870913],[100,\"~:logseq.property/type\",\"~:default\",536870913],[101,\"~:block/closed-value-property\",100,536870913],[101,\"~:block/created-at\",1786627375721,536870913],[101,\"~:block/order\",\"b0R\",536870913],[101,\"~:block/page\",100,536870913],[101,\"~:block/parent\",100,536870913],[101,\"~:block/title\",\"Advance from completion\",536870913],[101,\"~:block/updated-at\",1786627375721,536870913],[101,\"~:block/uuid\",\"~u00000002-1406-5331-0600-000000000000\",536870913],[101,\"^3\",\"~:logseq.property.repeat/repeat-type.dotted-plus\",536870913],[101,\"^8\",true,536870913],[101,\"~:logseq.property/created-from-property\",100,536870913]]]", + "addresses": null + }, + { + "addr": 1000079, + "content": "[\"^ \",\"~:keys\",[[102,\"~:block/closed-value-property\",100,536870913],[102,\"~:block/created-at\",1786627375721,536870913],[102,\"~:block/order\",\"b0S\",536870913],[102,\"~:block/page\",100,536870913],[102,\"~:block/parent\",100,536870913],[102,\"~:block/title\",\"Advance from scheduled\",536870913],[102,\"~:block/updated-at\",1786627375721,536870913],[102,\"~:block/uuid\",\"~u00000002-1646-4063-5900-000000000000\",536870913],[102,\"~:db/ident\",\"~:logseq.property.repeat/repeat-type.plus\",536870913],[102,\"~:logseq.property/built-in?\",true,536870913],[102,\"~:logseq.property/created-from-property\",100,536870913],[103,\"^1\",100,536870913],[103,\"^2\",1786627375721,536870913],[103,\"^3\",\"b0T\",536870913],[103,\"^4\",100,536870913],[103,\"^5\",100,536870913],[103,\"^6\",\"Advance from scheduled, skip to future\",536870913]]]", + "addresses": null + }, + { + "addr": 1000080, + "content": "[\"^ \",\"~:keys\",[[103,\"~:block/updated-at\",1786627375721,536870913],[103,\"~:block/uuid\",\"~u00000002-7441-9615-3000-000000000000\",536870913],[103,\"~:db/ident\",\"~:logseq.property.repeat/repeat-type.double-plus\",536870913],[103,\"~:logseq.property/built-in?\",true,536870913],[103,\"~:logseq.property/created-from-property\",100,536870913],[104,\"~:block/created-at\",1786627375721,536870913],[104,\"~:block/name\",\"repeating temporal property\",536870913],[104,\"~:block/order\",\"b0U\",536870913],[104,\"~:block/tags\",3,536870913],[104,\"~:block/title\",\"Repeating Temporal Property\",536870913],[104,\"^1\",1786627375721,536870913],[104,\"^2\",\"~u00000002-8346-1078-4000-000000000000\",536870913],[104,\"~:db/cardinality\",\"~:db.cardinality/one\",536870913],[104,\"^3\",\"~:logseq.property.repeat/temporal-property\",536870913],[104,\"~:db/index\",true,536870913],[104,\"~:db/valueType\",\"~:db.type/ref\",536870913],[104,\"^5\",true,536870913],[104,\"~:logseq.property/hide?\",true,536870913]]]", + "addresses": null + }, + { + "addr": 1000081, + "content": "[\"^ \",\"~:keys\",[[104,\"~:logseq.property/type\",\"~:property\",536870913],[105,\"~:block/created-at\",1786627375721,536870913],[105,\"~:block/name\",\"repeating checked property\",536870913],[105,\"~:block/order\",\"b0V\",536870913],[105,\"~:block/tags\",3,536870913],[105,\"~:block/title\",\"Repeating Checked Property\",536870913],[105,\"~:block/updated-at\",1786627375721,536870913],[105,\"~:block/uuid\",\"~u00000002-1866-3655-5300-000000000000\",536870913],[105,\"~:db/cardinality\",\"~:db.cardinality/one\",536870913],[105,\"~:db/ident\",\"~:logseq.property.repeat/checked-property\",536870913],[105,\"~:db/index\",true,536870913],[105,\"~:db/valueType\",\"~:db.type/ref\",536870913],[105,\"~:logseq.property/built-in?\",true,536870913],[105,\"~:logseq.property/hide?\",true,536870913],[105,\"^1\",\"^2\",536870913],[106,\"^3\",1786627375721,536870913],[106,\"^4\",\"assignee\",536870913]]]", + "addresses": null + }, + { + "addr": 1000082, + "content": "[\"^ \",\"~:keys\",[[106,\"~:block/order\",\"b0W\",536870913],[106,\"~:block/tags\",3,536870913],[106,\"~:block/title\",\"Assignee\",536870913],[106,\"~:block/updated-at\",1786627375721,536870913],[106,\"~:block/uuid\",\"~u00000002-2190-1503-4000-000000000000\",536870913],[106,\"~:db/cardinality\",\"~:db.cardinality/many\",536870913],[106,\"~:db/ident\",\"~:logseq.property/assignee\",536870913],[106,\"~:db/index\",true,536870913],[106,\"~:db/valueType\",\"~:db.type/ref\",536870913],[106,\"~:logseq.property/built-in?\",true,536870913],[106,\"~:logseq.property/classes\",4,536870913],[106,\"~:logseq.property/hide-empty-value\",true,536870913],[106,\"~:logseq.property/public?\",true,536870913],[106,\"~:logseq.property/type\",\"~:node\",536870913],[106,\"~:logseq.property/ui-position\",\"~:block-below\",536870913],[107,\"~:block/created-at\",1786627375721,536870913],[107,\"~:block/name\",\"icon\",536870913]]]", + "addresses": null + }, + { + "addr": 1000083, + "content": "[\"^ \",\"~:keys\",[[107,\"~:block/order\",\"b0X\",536870913],[107,\"~:block/tags\",3,536870913],[107,\"~:block/title\",\"Icon\",536870913],[107,\"~:block/updated-at\",1786627375721,536870913],[107,\"~:block/uuid\",\"~u00000002-5891-2328-5000-000000000000\",536870913],[107,\"~:db/cardinality\",\"~:db.cardinality/one\",536870913],[107,\"~:db/ident\",\"~:logseq.property/icon\",536870913],[107,\"~:db/index\",true,536870913],[107,\"~:logseq.property/built-in?\",true,536870913],[107,\"~:logseq.property/type\",\"~:map\",536870913],[108,\"~:block/created-at\",1786627375721,536870913],[108,\"~:block/name\",\"publishing public?\",536870913],[108,\"^1\",\"b0Y\",536870913],[108,\"^2\",3,536870913],[108,\"^3\",\"Publishing Public?\",536870913],[108,\"^4\",1786627375721,536870913],[108,\"^5\",\"~u00000002-1094-6579-3900-000000000000\",536870913]]]", + "addresses": null + }, + { + "addr": 1000084, + "content": "[\"^ \",\"~:keys\",[[108,\"~:db/cardinality\",\"~:db.cardinality/one\",536870913],[108,\"~:db/ident\",\"~:logseq.property/publishing-public?\",536870913],[108,\"~:db/index\",true,536870913],[108,\"~:logseq.property/built-in?\",true,536870913],[108,\"~:logseq.property/hide?\",true,536870913],[108,\"~:logseq.property/public?\",true,536870913],[108,\"~:logseq.property/type\",\"~:checkbox\",536870913],[108,\"~:logseq.property/view-context\",\"~:page\",536870913],[109,\"~:block/created-at\",1786627375721,536870913],[109,\"~:block/name\",\"published url\",536870913],[109,\"~:block/order\",\"b0Z\",536870913],[109,\"~:block/tags\",3,536870913],[109,\"~:block/title\",\"Published URL\",536870913],[109,\"~:block/updated-at\",1786627375721,536870913],[109,\"~:block/uuid\",\"~u00000002-2045-1825-0100-000000000000\",536870913],[109,\"^1\",\"^2\",536870913],[109,\"^3\",\"~:logseq.property.publish/published-url\",536870913]]]", + "addresses": null + }, + { + "addr": 1000085, + "content": "[\"^ \",\"~:keys\",[[109,\"~:db/index\",true,536870913],[109,\"~:db/valueType\",\"~:db.type/ref\",536870913],[109,\"~:logseq.property/built-in?\",true,536870913],[109,\"~:logseq.property/public?\",true,536870913],[109,\"~:logseq.property/type\",\"~:url\",536870913],[109,\"~:logseq.property/view-context\",\"~:page\",536870913],[110,\"~:block/created-at\",1786627375721,536870913],[110,\"~:block/name\",\"excluded from graph view?\",536870913],[110,\"~:block/order\",\"b0a\",536870913],[110,\"~:block/tags\",3,536870913],[110,\"~:block/title\",\"Excluded from Graph view?\",536870913],[110,\"~:block/updated-at\",1786627375721,536870913],[110,\"~:block/uuid\",\"~u00000002-4524-3306-5000-000000000000\",536870913],[110,\"~:db/cardinality\",\"~:db.cardinality/one\",536870913],[110,\"~:db/ident\",\"~:logseq.property/exclude-from-graph-view\",536870913],[110,\"^1\",true,536870913],[110,\"^4\",true,536870913],[110,\"~:logseq.property/hide?\",true,536870913]]]", + "addresses": null + }, + { + "addr": 1000086, + "content": "[\"^ \",\"~:keys\",[[110,\"~:logseq.property/public?\",true,536870913],[110,\"~:logseq.property/type\",\"~:checkbox\",536870913],[110,\"~:logseq.property/view-context\",\"~:page\",536870913],[111,\"~:block/created-at\",1786627375721,536870913],[111,\"~:block/name\",\"view type\",536870913],[111,\"~:block/order\",\"b0b\",536870913],[111,\"~:block/tags\",3,536870913],[111,\"~:block/title\",\"View Type\",536870913],[111,\"~:block/updated-at\",1786627375721,536870913],[111,\"~:block/uuid\",\"~u00000002-2182-3760-7000-000000000000\",536870913],[111,\"~:db/cardinality\",\"~:db.cardinality/one\",536870913],[111,\"~:db/ident\",\"~:logseq.property.view/type\",536870913],[111,\"~:db/index\",true,536870913],[111,\"~:db/valueType\",\"~:db.type/ref\",536870913],[111,\"~:logseq.property/built-in?\",true,536870913],[111,\"~:logseq.property/default-value\",112,536870913],[111,\"~:logseq.property/hide?\",true,536870913],[111,\"^1\",false,536870913]]]", + "addresses": null + }, + { + "addr": 1000087, + "content": "[\"^ \",\"~:keys\",[[111,\"~:logseq.property/type\",\"~:default\",536870913],[112,\"~:block/closed-value-property\",111,536870913],[112,\"~:block/created-at\",1786627375721,536870913],[112,\"~:block/order\",\"b0c\",536870913],[112,\"~:block/page\",111,536870913],[112,\"~:block/parent\",111,536870913],[112,\"~:block/title\",\"Table View\",536870913],[112,\"~:block/updated-at\",1786627375721,536870913],[112,\"~:block/uuid\",\"~u00000002-1942-5424-0000-000000000000\",536870913],[112,\"~:db/ident\",\"~:logseq.property.view/type.table\",536870913],[112,\"~:logseq.property/built-in?\",true,536870913],[112,\"~:logseq.property/created-from-property\",111,536870913],[112,\"~:logseq.property/icon\",[\"^ \",\"~:type\",\"~:tabler-icon\",\"~:id\",\"table\"],536870913],[113,\"^3\",111,536870913],[113,\"^4\",1786627375721,536870913],[113,\"^5\",\"b0d\",536870913],[113,\"^6\",111,536870913]]]", + "addresses": null + }, + { + "addr": 1000088, + "content": "[\"^ \",\"~:keys\",[[113,\"~:block/parent\",111,536870913],[113,\"~:block/title\",\"List View\",536870913],[113,\"~:block/updated-at\",1786627375721,536870913],[113,\"~:block/uuid\",\"~u00000002-1164-8285-0200-000000000000\",536870913],[113,\"~:db/ident\",\"~:logseq.property.view/type.list\",536870913],[113,\"~:logseq.property/built-in?\",true,536870913],[113,\"~:logseq.property/created-from-property\",111,536870913],[113,\"~:logseq.property/icon\",[\"^ \",\"~:type\",\"~:tabler-icon\",\"~:id\",\"list\"],536870913],[114,\"~:block/closed-value-property\",111,536870913],[114,\"~:block/created-at\",1786627375721,536870913],[114,\"~:block/order\",\"b0e\",536870913],[114,\"~:block/page\",111,536870913],[114,\"^1\",111,536870913],[114,\"^2\",\"Gallery View\",536870913],[114,\"^3\",1786627375721,536870913],[114,\"^4\",\"~u00000002-1506-0511-2000-000000000000\",536870913],[114,\"^5\",\"~:logseq.property.view/type.gallery\",536870913]]]", + "addresses": null + }, + { + "addr": 1000089, + "content": "[\"^ \",\"~:keys\",[[94,\"~:block/parent\",92,536870913],[96,\"~:block/closed-value-property\",92,536870913],[97,\"~:block/updated-at\",1786627375721,536870913],[99,\"~:block/order\",\"b0P\",536870913],[100,\"~:block/uuid\",\"~u00000002-1984-2972-8100-000000000000\",536870913],[101,\"~:logseq.property/created-from-property\",100,536870913],[103,\"~:block/title\",\"Advance from scheduled, skip to future\",536870913],[104,\"~:logseq.property/hide?\",true,536870913],[106,\"~:block/name\",\"assignee\",536870913],[107,\"^9\",\"icon\",536870913],[108,\"^5\",\"~u00000002-1094-6579-3900-000000000000\",536870913],[109,\"~:db/ident\",\"~:logseq.property.publish/published-url\",536870913],[110,\"^8\",true,536870913],[111,\"~:logseq.property/public?\",false,536870913],[113,\"~:block/page\",111,536870913],[114,\"^:\",\"~:logseq.property.view/type.gallery\",536870913]]]", + "addresses": "[1000073,1000074,1000075,1000076,1000077,1000078,1000079,1000080,1000081,1000082,1000083,1000084,1000085,1000086,1000087,1000088]" + }, + { + "addr": 1000090, + "content": "[\"^ \",\"~:keys\",[[114,\"~:logseq.property/built-in?\",true,536870913],[114,\"~:logseq.property/created-from-property\",111,536870913],[114,\"~:logseq.property/icon\",[\"^ \",\"~:type\",\"~:tabler-icon\",\"~:id\",\"layout-grid\"],536870913],[115,\"~:block/created-at\",1786627375721,536870913],[115,\"~:block/name\",\"view feature type\",536870913],[115,\"~:block/order\",\"b0f\",536870913],[115,\"~:block/tags\",3,536870913],[115,\"~:block/title\",\"View Feature Type\",536870913],[115,\"~:block/updated-at\",1786627375721,536870913],[115,\"~:block/uuid\",\"~u00000002-9391-4187-1000-000000000000\",536870913],[115,\"~:db/cardinality\",\"~:db.cardinality/one\",536870913],[115,\"~:db/ident\",\"~:logseq.property.view/feature-type\",536870913],[115,\"~:db/index\",true,536870913],[115,\"^1\",true,536870913],[115,\"~:logseq.property/hide?\",true,536870913],[115,\"~:logseq.property/public?\",false,536870913],[115,\"~:logseq.property/type\",\"~:keyword\",536870913],[116,\"^7\",1786627375721,536870913]]]", + "addresses": null + }, + { + "addr": 1000091, + "content": "[\"^ \",\"~:keys\",[[116,\"~:block/name\",\"view group by property\",536870913],[116,\"~:block/order\",\"b0g\",536870913],[116,\"~:block/tags\",3,536870913],[116,\"~:block/title\",\"View group by property\",536870913],[116,\"~:block/updated-at\",1786627375721,536870913],[116,\"~:block/uuid\",\"~u00000002-8092-1623-6000-000000000000\",536870913],[116,\"~:db/cardinality\",\"~:db.cardinality/one\",536870913],[116,\"~:db/ident\",\"~:logseq.property.view/group-by-property\",536870913],[116,\"~:db/index\",true,536870913],[116,\"~:db/valueType\",\"~:db.type/ref\",536870913],[116,\"~:logseq.property/built-in?\",true,536870913],[116,\"~:logseq.property/hide?\",true,536870913],[116,\"~:logseq.property/public?\",false,536870913],[116,\"~:logseq.property/type\",\"~:property\",536870913],[117,\"~:block/created-at\",1786627375721,536870913],[117,\"^1\",\"gallery asset property\",536870913],[117,\"^2\",\"b0h\",536870913]]]", + "addresses": null + }, + { + "addr": 1000092, + "content": "[\"^ \",\"~:keys\",[[117,\"~:block/tags\",3,536870913],[117,\"~:block/title\",\"Gallery asset property\",536870913],[117,\"~:block/updated-at\",1786627375721,536870913],[117,\"~:block/uuid\",\"~u00000002-1668-3468-9000-000000000000\",536870913],[117,\"~:db/cardinality\",\"~:db.cardinality/one\",536870913],[117,\"~:db/ident\",\"~:logseq.property.view/gallery-asset-property\",536870913],[117,\"~:db/index\",true,536870913],[117,\"~:db/valueType\",\"~:db.type/ref\",536870913],[117,\"~:logseq.property/built-in?\",true,536870913],[117,\"~:logseq.property/hide?\",true,536870913],[117,\"~:logseq.property/public?\",false,536870913],[117,\"~:logseq.property/type\",\"~:property\",536870913],[118,\"~:block/created-at\",1786627375722,536870913],[118,\"~:block/name\",\"gallery display properties\",536870913],[118,\"~:block/order\",\"b0i\",536870913],[118,\"^1\",3,536870913],[118,\"^2\",\"Gallery display properties\",536870913]]]", + "addresses": null + }, + { + "addr": 1000093, + "content": "[\"^ \",\"~:keys\",[[118,\"~:block/updated-at\",1786627375722,536870913],[118,\"~:block/uuid\",\"~u00000002-1806-7066-8800-000000000000\",536870913],[118,\"~:db/cardinality\",\"~:db.cardinality/many\",536870913],[118,\"~:db/ident\",\"~:logseq.property.view/gallery-display-properties\",536870913],[118,\"~:db/index\",true,536870913],[118,\"~:db/valueType\",\"~:db.type/ref\",536870913],[118,\"~:logseq.property/built-in?\",true,536870913],[118,\"~:logseq.property/hide?\",true,536870913],[118,\"~:logseq.property/public?\",false,536870913],[118,\"~:logseq.property/type\",\"~:property\",536870913],[119,\"~:block/created-at\",1786627375722,536870913],[119,\"~:block/name\",\"gallery card size\",536870913],[119,\"~:block/order\",\"b0j\",536870913],[119,\"~:block/tags\",3,536870913],[119,\"~:block/title\",\"Gallery card size\",536870913],[119,\"^1\",1786627375722,536870913],[119,\"^2\",\"~u00000002-2012-8168-3300-000000000000\",536870913]]]", + "addresses": null + }, + { + "addr": 1000094, + "content": "[\"^ \",\"~:keys\",[[119,\"~:db/cardinality\",\"~:db.cardinality/one\",536870913],[119,\"~:db/ident\",\"~:logseq.property.view/gallery-card-size\",536870913],[119,\"~:db/index\",true,536870913],[119,\"~:logseq.property/built-in?\",true,536870913],[119,\"~:logseq.property/hide?\",true,536870913],[119,\"~:logseq.property/public?\",false,536870913],[119,\"~:logseq.property/scalar-default-value\",\"~:default\",536870913],[119,\"~:logseq.property/type\",\"~:keyword\",536870913],[120,\"~:block/created-at\",1786627375722,536870913],[120,\"~:block/name\",\"gallery card width\",536870913],[120,\"~:block/order\",\"b0k\",536870913],[120,\"~:block/tags\",3,536870913],[120,\"~:block/title\",\"Gallery card width\",536870913],[120,\"~:block/updated-at\",1786627375722,536870913],[120,\"~:block/uuid\",\"~u00000002-1403-2963-9500-000000000000\",536870913],[120,\"^1\",\"^2\",536870913],[120,\"^3\",\"~:logseq.property.view/gallery-card-width\",536870913]]]", + "addresses": null + }, + { + "addr": 1000095, + "content": "[\"^ \",\"~:keys\",[[120,\"~:db/index\",true,536870913],[120,\"~:logseq.property/built-in?\",true,536870913],[120,\"~:logseq.property/hide?\",true,536870913],[120,\"~:logseq.property/public?\",false,536870913],[120,\"~:logseq.property/type\",\"~:raw-number\",536870913],[121,\"~:block/created-at\",1786627375722,536870913],[121,\"~:block/name\",\"gallery card height\",536870913],[121,\"~:block/order\",\"b0l\",536870913],[121,\"~:block/tags\",3,536870913],[121,\"~:block/title\",\"Gallery card height\",536870913],[121,\"~:block/updated-at\",1786627375722,536870913],[121,\"~:block/uuid\",\"~u00000002-4342-2083-2000-000000000000\",536870913],[121,\"~:db/cardinality\",\"~:db.cardinality/one\",536870913],[121,\"~:db/ident\",\"~:logseq.property.view/gallery-card-height\",536870913],[121,\"^1\",true,536870913],[121,\"^2\",true,536870913],[121,\"^3\",true,536870913],[121,\"^4\",false,536870913]]]", + "addresses": null + }, + { + "addr": 1000096, + "content": "[\"^ \",\"~:keys\",[[121,\"~:logseq.property/type\",\"~:raw-number\",536870913],[122,\"~:block/created-at\",1786627375722,536870913],[122,\"~:block/name\",\"view sort groups by\",536870913],[122,\"~:block/order\",\"b0m\",536870913],[122,\"~:block/tags\",3,536870913],[122,\"~:block/title\",\"View sort groups by\",536870913],[122,\"~:block/updated-at\",1786627375722,536870913],[122,\"~:block/uuid\",\"~u00000002-6253-4002-1000-000000000000\",536870913],[122,\"~:db/cardinality\",\"~:db.cardinality/one\",536870913],[122,\"~:db/ident\",\"~:logseq.property.view/sort-groups-by-property\",536870913],[122,\"~:db/index\",true,536870913],[122,\"~:db/valueType\",\"~:db.type/ref\",536870913],[122,\"~:logseq.property/built-in?\",true,536870913],[122,\"~:logseq.property/hide?\",true,536870913],[122,\"~:logseq.property/public?\",false,536870913],[122,\"^1\",\"~:property\",536870913],[123,\"^3\",1786627375722,536870913]]]", + "addresses": null + }, + { + "addr": 1000097, + "content": "[\"^ \",\"~:keys\",[[123,\"~:block/name\",\"view sort groups desc\",536870913],[123,\"~:block/order\",\"b0n\",536870913],[123,\"~:block/tags\",3,536870913],[123,\"~:block/title\",\"View sort groups DESC\",536870913],[123,\"~:block/updated-at\",1786627375722,536870913],[123,\"~:block/uuid\",\"~u00000002-1081-4073-8700-000000000000\",536870913],[123,\"~:db/cardinality\",\"~:db.cardinality/one\",536870913],[123,\"~:db/ident\",\"~:logseq.property.view/sort-groups-desc?\",536870913],[123,\"~:db/index\",true,536870913],[123,\"~:logseq.property/built-in?\",true,536870913],[123,\"~:logseq.property/hide?\",true,536870913],[123,\"~:logseq.property/public?\",false,536870913],[123,\"~:logseq.property/scalar-default-value\",true,536870913],[123,\"~:logseq.property/type\",\"~:checkbox\",536870913],[124,\"~:block/created-at\",1786627375722,536870913],[124,\"^1\",\"view sorting\",536870913],[124,\"^2\",\"b0o\",536870913]]]", + "addresses": null + }, + { + "addr": 1000098, + "content": "[\"^ \",\"~:keys\",[[124,\"~:block/tags\",3,536870913],[124,\"~:block/title\",\"View sorting\",536870913],[124,\"~:block/updated-at\",1786627375722,536870913],[124,\"~:block/uuid\",\"~u00000002-2081-0259-4000-000000000000\",536870913],[124,\"~:db/cardinality\",\"~:db.cardinality/one\",536870913],[124,\"~:db/ident\",\"~:logseq.property.table/sorting\",536870913],[124,\"~:db/index\",true,536870913],[124,\"~:logseq.property/built-in?\",true,536870913],[124,\"~:logseq.property/hide?\",true,536870913],[124,\"~:logseq.property/public?\",false,536870913],[124,\"~:logseq.property/type\",\"~:coll\",536870913],[125,\"~:block/created-at\",1786627375722,536870913],[125,\"~:block/name\",\"view filters\",536870913],[125,\"~:block/order\",\"b0p\",536870913],[125,\"^1\",3,536870913],[125,\"^2\",\"View filters\",536870913],[125,\"^3\",1786627375722,536870913]]]", + "addresses": null + }, + { + "addr": 1000099, + "content": "[\"^ \",\"~:keys\",[[125,\"~:block/uuid\",\"~u00000002-1702-3936-3300-000000000000\",536870913],[125,\"~:db/cardinality\",\"~:db.cardinality/one\",536870913],[125,\"~:db/ident\",\"~:logseq.property.table/filters\",536870913],[125,\"~:db/index\",true,536870913],[125,\"~:logseq.property/built-in?\",true,536870913],[125,\"~:logseq.property/hide?\",true,536870913],[125,\"~:logseq.property/public?\",false,536870913],[125,\"~:logseq.property/type\",\"~:map\",536870913],[126,\"~:block/created-at\",1786627375722,536870913],[126,\"~:block/name\",\"view hidden columns\",536870913],[126,\"~:block/order\",\"b0q\",536870913],[126,\"~:block/tags\",3,536870913],[126,\"~:block/title\",\"View hidden columns\",536870913],[126,\"~:block/updated-at\",1786627375722,536870913],[126,\"^1\",\"~u00000002-9750-5719-2000-000000000000\",536870913],[126,\"^2\",\"~:db.cardinality/many\",536870913],[126,\"^4\",\"~:logseq.property.table/hidden-columns\",536870913]]]", + "addresses": null + }, + { + "addr": 1000100, + "content": "[\"^ \",\"~:keys\",[[126,\"~:db/index\",true,536870913],[126,\"~:logseq.property/built-in?\",true,536870913],[126,\"~:logseq.property/hide?\",true,536870913],[126,\"~:logseq.property/public?\",false,536870913],[126,\"~:logseq.property/type\",\"~:keyword\",536870913],[127,\"~:block/created-at\",1786627375722,536870913],[127,\"~:block/name\",\"view ordered columns\",536870913],[127,\"~:block/order\",\"b0r\",536870913],[127,\"~:block/tags\",3,536870913],[127,\"~:block/title\",\"View ordered columns\",536870913],[127,\"~:block/updated-at\",1786627375722,536870913],[127,\"~:block/uuid\",\"~u00000002-1485-5871-0000-000000000000\",536870913],[127,\"~:db/cardinality\",\"~:db.cardinality/one\",536870913],[127,\"~:db/ident\",\"~:logseq.property.table/ordered-columns\",536870913],[127,\"^1\",true,536870913],[127,\"^2\",true,536870913],[127,\"^3\",true,536870913],[127,\"^4\",false,536870913]]]", + "addresses": null + }, + { + "addr": 1000101, + "content": "[\"^ \",\"~:keys\",[[127,\"~:logseq.property/type\",\"~:coll\",536870913],[128,\"~:block/created-at\",1786627375722,536870913],[128,\"~:block/name\",\"view columns settings\",536870913],[128,\"~:block/order\",\"b0s\",536870913],[128,\"~:block/tags\",3,536870913],[128,\"~:block/title\",\"View columns settings\",536870913],[128,\"~:block/updated-at\",1786627375722,536870913],[128,\"~:block/uuid\",\"~u00000002-1675-5105-5500-000000000000\",536870913],[128,\"~:db/cardinality\",\"~:db.cardinality/one\",536870913],[128,\"~:db/ident\",\"~:logseq.property.table/sized-columns\",536870913],[128,\"~:db/index\",true,536870913],[128,\"~:logseq.property/built-in?\",true,536870913],[128,\"~:logseq.property/hide?\",true,536870913],[128,\"~:logseq.property/public?\",false,536870913],[128,\"^1\",\"~:map\",536870913],[129,\"^3\",1786627375722,536870913],[129,\"^4\",\"table view pinned columns\",536870913]]]", + "addresses": null + }, + { + "addr": 1000102, + "content": "[\"^ \",\"~:keys\",[[129,\"~:block/order\",\"b0t\",536870913],[129,\"~:block/tags\",3,536870913],[129,\"~:block/title\",\"Table view pinned columns\",536870913],[129,\"~:block/updated-at\",1786627375722,536870913],[129,\"~:block/uuid\",\"~u00000002-2673-7513-8000-000000000000\",536870913],[129,\"~:db/cardinality\",\"~:db.cardinality/many\",536870913],[129,\"~:db/ident\",\"~:logseq.property.table/pinned-columns\",536870913],[129,\"~:db/index\",true,536870913],[129,\"~:db/valueType\",\"~:db.type/ref\",536870913],[129,\"~:logseq.property/built-in?\",true,536870913],[129,\"~:logseq.property/hide?\",true,536870913],[129,\"~:logseq.property/public?\",false,536870913],[129,\"~:logseq.property/type\",\"~:property\",536870913],[130,\"~:block/created-at\",1786627375722,536870913],[130,\"~:block/name\",\"this view belongs to\",536870913],[130,\"^1\",\"b0u\",536870913],[130,\"^2\",3,536870913]]]", + "addresses": null + }, + { + "addr": 1000103, + "content": "[\"^ \",\"~:keys\",[[130,\"~:block/title\",\"This view belongs to\",536870913],[130,\"~:block/updated-at\",1786627375722,536870913],[130,\"~:block/uuid\",\"~u00000002-3627-4319-0000-000000000000\",536870913],[130,\"~:db/cardinality\",\"~:db.cardinality/one\",536870913],[130,\"~:db/ident\",\"~:logseq.property/view-for\",536870913],[130,\"~:db/index\",true,536870913],[130,\"~:db/valueType\",\"~:db.type/ref\",536870913],[130,\"~:logseq.property/built-in?\",true,536870913],[130,\"~:logseq.property/hide?\",true,536870913],[130,\"~:logseq.property/public?\",false,536870913],[130,\"~:logseq.property/type\",\"~:node\",536870913],[131,\"~:block/created-at\",1786627375722,536870913],[131,\"~:block/name\",\"file type\",536870913],[131,\"~:block/order\",\"b0v\",536870913],[131,\"~:block/tags\",3,536870913],[131,\"^1\",\"File Type\",536870913],[131,\"^2\",1786627375722,536870913]]]", + "addresses": null + }, + { + "addr": 1000104, + "content": "[\"^ \",\"~:keys\",[[131,\"~:block/uuid\",\"~u00000002-1142-0830-9800-000000000000\",536870913],[131,\"~:db/cardinality\",\"~:db.cardinality/one\",536870913],[131,\"~:db/ident\",\"~:logseq.property.asset/type\",536870913],[131,\"~:db/index\",true,536870913],[131,\"~:logseq.property/built-in?\",true,536870913],[131,\"~:logseq.property/hide?\",true,536870913],[131,\"~:logseq.property/public?\",false,536870913],[131,\"~:logseq.property/type\",\"~:string\",536870913],[132,\"~:block/created-at\",1786627375722,536870913],[132,\"~:block/name\",\"external url\",536870913],[132,\"~:block/order\",\"b0w\",536870913],[132,\"~:block/tags\",3,536870913],[132,\"~:block/title\",\"External URL\",536870913],[132,\"~:block/updated-at\",1786627375722,536870913],[132,\"^1\",\"~u00000002-1364-7751-6300-000000000000\",536870913],[132,\"^2\",\"^3\",536870913],[132,\"^4\",\"~:logseq.property.asset/external-url\",536870913]]]", + "addresses": null + }, + { + "addr": 1000105, + "content": "[\"^ \",\"~:keys\",[[132,\"~:db/index\",true,536870913],[132,\"~:logseq.property/built-in?\",true,536870913],[132,\"~:logseq.property/hide?\",false,536870913],[132,\"~:logseq.property/public?\",true,536870913],[132,\"~:logseq.property/type\",\"~:string\",536870913],[133,\"~:block/created-at\",1786627375722,536870913],[133,\"~:block/name\",\"external file name\",536870913],[133,\"~:block/order\",\"b0x\",536870913],[133,\"~:block/tags\",3,536870913],[133,\"~:block/title\",\"External file name\",536870913],[133,\"~:block/updated-at\",1786627375722,536870913],[133,\"~:block/uuid\",\"~u00000002-3995-2972-0000-000000000000\",536870913],[133,\"~:db/cardinality\",\"~:db.cardinality/one\",536870913],[133,\"~:db/ident\",\"~:logseq.property.asset/external-file-name\",536870913],[133,\"^1\",true,536870913],[133,\"^2\",true,536870913],[133,\"^3\",true,536870913],[133,\"^4\",false,536870913]]]", + "addresses": null + }, + { + "addr": 1000106, + "content": "[\"^ \",\"~:keys\",[[116,\"~:block/created-at\",1786627375721,536870913],[117,\"~:block/order\",\"b0h\",536870913],[118,\"~:block/title\",\"Gallery display properties\",536870913],[119,\"~:block/uuid\",\"~u00000002-2012-8168-3300-000000000000\",536870913],[120,\"~:db/ident\",\"~:logseq.property.view/gallery-card-width\",536870913],[121,\"~:logseq.property/public?\",false,536870913],[123,\"^1\",1786627375722,536870913],[124,\"^2\",\"b0o\",536870913],[125,\"~:block/updated-at\",1786627375722,536870913],[126,\"^5\",\"~:logseq.property.table/hidden-columns\",536870913],[127,\"^7\",false,536870913],[129,\"~:block/name\",\"table view pinned columns\",536870913],[130,\"~:block/tags\",3,536870913],[131,\"^8\",1786627375722,536870913],[132,\"^5\",\"~:logseq.property.asset/external-url\",536870913],[133,\"^7\",false,536870913]]]", + "addresses": "[1000090,1000091,1000092,1000093,1000094,1000095,1000096,1000097,1000098,1000099,1000100,1000101,1000102,1000103,1000104,1000105]" + }, + { + "addr": 1000107, + "content": "[\"^ \",\"~:keys\",[[133,\"~:logseq.property/type\",\"~:string\",536870913],[134,\"~:block/created-at\",1786627375722,536870913],[134,\"~:block/name\",\"file size\",536870913],[134,\"~:block/order\",\"b0y\",536870913],[134,\"~:block/tags\",3,536870913],[134,\"~:block/title\",\"File Size\",536870913],[134,\"~:block/updated-at\",1786627375722,536870913],[134,\"~:block/uuid\",\"~u00000002-1167-8621-9000-000000000000\",536870913],[134,\"~:db/cardinality\",\"~:db.cardinality/one\",536870913],[134,\"~:db/ident\",\"~:logseq.property.asset/size\",536870913],[134,\"~:db/index\",true,536870913],[134,\"~:logseq.property/built-in?\",true,536870913],[134,\"~:logseq.property/hide?\",true,536870913],[134,\"~:logseq.property/public?\",false,536870913],[134,\"^1\",\"~:raw-number\",536870913],[135,\"^3\",1786627375722,536870913],[135,\"^4\",\"image width\",536870913]]]", + "addresses": null + }, + { + "addr": 1000108, + "content": "[\"^ \",\"~:keys\",[[135,\"~:block/order\",\"b0z\",536870913],[135,\"~:block/tags\",3,536870913],[135,\"~:block/title\",\"Image width\",536870913],[135,\"~:block/updated-at\",1786627375722,536870913],[135,\"~:block/uuid\",\"~u00000002-1857-8658-3900-000000000000\",536870913],[135,\"~:db/cardinality\",\"~:db.cardinality/one\",536870913],[135,\"~:db/ident\",\"~:logseq.property.asset/width\",536870913],[135,\"~:db/index\",true,536870913],[135,\"~:logseq.property/built-in?\",true,536870913],[135,\"~:logseq.property/hide?\",true,536870913],[135,\"~:logseq.property/public?\",false,536870913],[135,\"~:logseq.property/type\",\"~:raw-number\",536870913],[136,\"~:block/created-at\",1786627375722,536870913],[136,\"~:block/name\",\"image height\",536870913],[136,\"^1\",\"b10\",536870913],[136,\"^2\",3,536870913],[136,\"^3\",\"Image height\",536870913]]]", + "addresses": null + }, + { + "addr": 1000109, + "content": "[\"^ \",\"~:keys\",[[136,\"~:block/updated-at\",1786627375722,536870913],[136,\"~:block/uuid\",\"~u00000002-3306-1650-4000-000000000000\",536870913],[136,\"~:db/cardinality\",\"~:db.cardinality/one\",536870913],[136,\"~:db/ident\",\"~:logseq.property.asset/height\",536870913],[136,\"~:db/index\",true,536870913],[136,\"~:logseq.property/built-in?\",true,536870913],[136,\"~:logseq.property/hide?\",true,536870913],[136,\"~:logseq.property/public?\",false,536870913],[136,\"~:logseq.property/type\",\"~:raw-number\",536870913],[137,\"~:block/created-at\",1786627375722,536870913],[137,\"~:block/name\",\"file checksum\",536870913],[137,\"~:block/order\",\"b11\",536870913],[137,\"~:block/tags\",3,536870913],[137,\"~:block/title\",\"File checksum\",536870913],[137,\"^1\",1786627375722,536870913],[137,\"^2\",\"~u00000002-1011-4169-7900-000000000000\",536870913],[137,\"^3\",\"^4\",536870913]]]", + "addresses": null + }, + { + "addr": 1000110, + "content": "[\"^ \",\"~:keys\",[[137,\"~:db/ident\",\"~:logseq.property.asset/checksum\",536870913],[137,\"~:db/index\",true,536870913],[137,\"~:logseq.property/built-in?\",true,536870913],[137,\"~:logseq.property/hide?\",true,536870913],[137,\"~:logseq.property/public?\",false,536870913],[137,\"~:logseq.property/type\",\"~:string\",536870913],[138,\"~:block/created-at\",1786627375722,536870913],[138,\"~:block/name\",\"last visit page\",536870913],[138,\"~:block/order\",\"b12\",536870913],[138,\"~:block/tags\",3,536870913],[138,\"~:block/title\",\"Last visit page\",536870913],[138,\"~:block/updated-at\",1786627375722,536870913],[138,\"~:block/uuid\",\"~u00000002-2107-8035-3500-000000000000\",536870913],[138,\"~:db/cardinality\",\"~:db.cardinality/one\",536870913],[138,\"^1\",\"~:logseq.property.asset/last-visit-page\",536870913],[138,\"^3\",true,536870913],[138,\"^4\",true,536870913],[138,\"^5\",true,536870913]]]", + "addresses": null + }, + { + "addr": 1000111, + "content": "[\"^ \",\"~:keys\",[[138,\"~:logseq.property/public?\",false,536870913],[138,\"~:logseq.property/type\",\"~:raw-number\",536870913],[139,\"~:block/created-at\",1786627375722,536870913],[139,\"~:block/name\",\"file remote metadata\",536870913],[139,\"~:block/order\",\"b13\",536870913],[139,\"~:block/tags\",3,536870913],[139,\"~:block/title\",\"File remote metadata\",536870913],[139,\"~:block/updated-at\",1786627375722,536870913],[139,\"~:block/uuid\",\"~u00000002-9907-5046-9000-000000000000\",536870913],[139,\"~:db/cardinality\",\"~:db.cardinality/one\",536870913],[139,\"~:db/ident\",\"~:logseq.property.asset/remote-metadata\",536870913],[139,\"~:db/index\",true,536870913],[139,\"~:logseq.property/built-in?\",true,536870913],[139,\"~:logseq.property/description\",140,536870913],[139,\"~:logseq.property/hide?\",true,536870913],[139,\"^1\",false,536870913],[139,\"^2\",\"~:map\",536870913]]]", + "addresses": null + }, + { + "addr": 1000112, + "content": "[\"^ \",\"~:keys\",[[140,\"~:block/created-at\",1786627375722,536870913],[140,\"~:block/order\",\"b14\",536870913],[140,\"~:block/page\",139,536870913],[140,\"~:block/parent\",139,536870913],[140,\"~:block/title\",\"Metadata of asset in remote storage\",536870913],[140,\"~:block/updated-at\",1786627375722,536870913],[140,\"~:block/uuid\",\"~u00000004-5450-2102-9000-000000000000\",536870913],[140,\"~:logseq.property/built-in?\",true,536870913],[140,\"~:logseq.property/created-from-property\",35,536870913],[141,\"^1\",1786627375722,536870913],[141,\"~:block/name\",\"asset resize metadata\",536870913],[141,\"^2\",\"b15\",536870913],[141,\"~:block/tags\",3,536870913],[141,\"^5\",\"Asset resize metadata\",536870913],[141,\"^6\",1786627375722,536870913],[141,\"^7\",\"~u00000002-1297-5230-5500-000000000000\",536870913],[141,\"~:db/cardinality\",\"~:db.cardinality/one\",536870913]]]", + "addresses": null + }, + { + "addr": 1000113, + "content": "[\"^ \",\"~:keys\",[[141,\"~:db/ident\",\"~:logseq.property.asset/resize-metadata\",536870913],[141,\"~:db/index\",true,536870913],[141,\"~:logseq.property/built-in?\",true,536870913],[141,\"~:logseq.property/hide?\",true,536870913],[141,\"~:logseq.property/public?\",false,536870913],[141,\"~:logseq.property/type\",\"~:map\",536870913],[142,\"~:block/created-at\",1786627375722,536870913],[142,\"~:block/name\",\"asset alignment\",536870913],[142,\"~:block/order\",\"b16\",536870913],[142,\"~:block/tags\",3,536870913],[142,\"~:block/title\",\"Asset alignment\",536870913],[142,\"~:block/updated-at\",1786627375722,536870913],[142,\"~:block/uuid\",\"~u00000002-7135-0412-8000-000000000000\",536870913],[142,\"~:db/cardinality\",\"~:db.cardinality/one\",536870913],[142,\"^1\",\"~:logseq.property.asset/align\",536870913],[142,\"^3\",true,536870913],[142,\"^4\",true,536870913],[142,\"^5\",true,536870913]]]", + "addresses": null + }, + { + "addr": 1000114, + "content": "[\"^ \",\"~:keys\",[[142,\"~:logseq.property/public?\",false,536870913],[142,\"~:logseq.property/type\",\"~:keyword\",536870913],[143,\"~:block/created-at\",1786627375723,536870913],[143,\"~:block/name\",\"due\",536870913],[143,\"~:block/order\",\"b17\",536870913],[143,\"~:block/tags\",3,536870913],[143,\"~:block/title\",\"Due\",536870913],[143,\"~:block/updated-at\",1786627375723,536870913],[143,\"~:block/uuid\",\"~u00000002-1089-0805-4900-000000000000\",536870913],[143,\"~:db/cardinality\",\"~:db.cardinality/one\",536870913],[143,\"~:db/ident\",\"~:logseq.property.fsrs/due\",536870913],[143,\"~:db/index\",true,536870913],[143,\"~:logseq.property/built-in?\",true,536870913],[143,\"~:logseq.property/hide?\",false,536870913],[143,\"^1\",false,536870913],[143,\"^2\",\"~:datetime\",536870913],[144,\"^4\",1786627375723,536870913]]]", + "addresses": null + }, + { + "addr": 1000115, + "content": "[\"^ \",\"~:keys\",[[144,\"~:block/name\",\"state\",536870913],[144,\"~:block/order\",\"b18\",536870913],[144,\"~:block/tags\",3,536870913],[144,\"~:block/title\",\"State\",536870913],[144,\"~:block/updated-at\",1786627375723,536870913],[144,\"~:block/uuid\",\"~u00000002-1165-1650-8700-000000000000\",536870913],[144,\"~:db/cardinality\",\"~:db.cardinality/one\",536870913],[144,\"~:db/ident\",\"~:logseq.property.fsrs/state\",536870913],[144,\"~:db/index\",true,536870913],[144,\"~:logseq.property/built-in?\",true,536870913],[144,\"~:logseq.property/hide?\",false,536870913],[144,\"~:logseq.property/public?\",false,536870913],[144,\"~:logseq.property/type\",\"~:map\",536870913],[145,\"~:block/created-at\",1786627375723,536870913],[145,\"^1\",\"user name\",536870913],[145,\"^2\",\"b19\",536870913],[145,\"^3\",3,536870913]]]", + "addresses": null + }, + { + "addr": 1000116, + "content": "[\"^ \",\"~:keys\",[[145,\"~:block/title\",\"User Name\",536870913],[145,\"~:block/updated-at\",1786627375723,536870913],[145,\"~:block/uuid\",\"~u00000002-1360-0260-1600-000000000000\",536870913],[145,\"~:db/cardinality\",\"~:db.cardinality/one\",536870913],[145,\"~:db/ident\",\"~:logseq.property.user/name\",536870913],[145,\"~:db/index\",true,536870913],[145,\"~:logseq.property/built-in?\",true,536870913],[145,\"~:logseq.property/hide?\",false,536870913],[145,\"~:logseq.property/public?\",true,536870913],[145,\"~:logseq.property/type\",\"~:string\",536870913],[146,\"~:block/created-at\",1786627375723,536870913],[146,\"~:block/name\",\"user email\",536870913],[146,\"~:block/order\",\"b1A\",536870913],[146,\"~:block/tags\",3,536870913],[146,\"^1\",\"User Email\",536870913],[146,\"^2\",1786627375723,536870913],[146,\"^3\",\"~u00000002-1655-2060-6300-000000000000\",536870913]]]", + "addresses": null + }, + { + "addr": 1000117, + "content": "[\"^ \",\"~:keys\",[[146,\"~:db/cardinality\",\"~:db.cardinality/one\",536870913],[146,\"~:db/ident\",\"~:logseq.property.user/email\",536870913],[146,\"~:db/index\",true,536870913],[146,\"~:logseq.property/built-in?\",true,536870913],[146,\"~:logseq.property/hide?\",false,536870913],[146,\"~:logseq.property/public?\",true,536870913],[146,\"~:logseq.property/type\",\"~:string\",536870913],[147,\"~:block/created-at\",1786627375723,536870913],[147,\"~:block/name\",\"user avatar\",536870913],[147,\"~:block/order\",\"b1B\",536870913],[147,\"~:block/tags\",3,536870913],[147,\"~:block/title\",\"User Avatar\",536870913],[147,\"~:block/updated-at\",1786627375723,536870913],[147,\"~:block/uuid\",\"~u00000002-4165-4885-8000-000000000000\",536870913],[147,\"^1\",\"^2\",536870913],[147,\"^3\",\"~:logseq.property.user/avatar\",536870913],[147,\"^5\",true,536870913]]]", + "addresses": null + }, + { + "addr": 1000118, + "content": "[\"^ \",\"~:keys\",[[147,\"~:logseq.property/built-in?\",true,536870913],[147,\"~:logseq.property/hide?\",false,536870913],[147,\"~:logseq.property/public?\",true,536870913],[147,\"~:logseq.property/type\",\"~:string\",536870913],[148,\"~:block/created-at\",1786627375723,536870913],[148,\"~:block/name\",\"enable property history\",536870913],[148,\"~:block/order\",\"b1C\",536870913],[148,\"~:block/tags\",3,536870913],[148,\"~:block/title\",\"Enable property history\",536870913],[148,\"~:block/updated-at\",1786627375723,536870913],[148,\"~:block/uuid\",\"~u00000002-8058-5960-2000-000000000000\",536870913],[148,\"~:db/cardinality\",\"~:db.cardinality/one\",536870913],[148,\"~:db/ident\",\"~:logseq.property/enable-history?\",536870913],[148,\"~:db/index\",true,536870913],[148,\"^1\",true,536870913],[148,\"~:logseq.property/description\",149,536870913],[148,\"^3\",true,536870913],[148,\"^4\",\"~:checkbox\",536870913]]]", + "addresses": null + }, + { + "addr": 1000119, + "content": "[\"^ \",\"~:keys\",[[148,\"~:logseq.property/view-context\",\"~:property\",536870913],[149,\"~:block/created-at\",1786627375723,536870913],[149,\"~:block/order\",\"b1D\",536870913],[149,\"~:block/page\",148,536870913],[149,\"~:block/parent\",148,536870913],[149,\"~:block/title\",\"Records history anytime a property's value changes on a node.\",536870913],[149,\"~:block/updated-at\",1786627375723,536870913],[149,\"~:block/uuid\",\"~u00000004-1438-1481-0000-000000000000\",536870913],[149,\"~:logseq.property/built-in?\",true,536870913],[149,\"~:logseq.property/created-from-property\",35,536870913],[150,\"^3\",1786627375723,536870913],[150,\"~:block/name\",\"history block\",536870913],[150,\"^4\",\"b1E\",536870913],[150,\"~:block/tags\",3,536870913],[150,\"^7\",\"History block\",536870913],[150,\"^8\",1786627375723,536870913],[150,\"^9\",\"~u00000002-1142-5541-6000-000000000000\",536870913]]]", + "addresses": null + }, + { + "addr": 1000120, + "content": "[\"^ \",\"~:keys\",[[150,\"~:db/cardinality\",\"~:db.cardinality/one\",536870913],[150,\"~:db/ident\",\"~:logseq.property.history/block\",536870913],[150,\"~:db/index\",true,536870913],[150,\"~:db/valueType\",\"~:db.type/ref\",536870913],[150,\"~:logseq.property/built-in?\",true,536870913],[150,\"~:logseq.property/hide?\",true,536870913],[150,\"~:logseq.property/type\",\"~:entity\",536870913],[151,\"~:block/created-at\",1786627375723,536870913],[151,\"~:block/name\",\"history property\",536870913],[151,\"~:block/order\",\"b1F\",536870913],[151,\"~:block/tags\",3,536870913],[151,\"~:block/title\",\"History property\",536870913],[151,\"~:block/updated-at\",1786627375723,536870913],[151,\"~:block/uuid\",\"~u00000002-1600-4090-8200-000000000000\",536870913],[151,\"^1\",\"^2\",536870913],[151,\"^3\",\"~:logseq.property.history/property\",536870913],[151,\"^5\",true,536870913]]]", + "addresses": null + }, + { + "addr": 1000121, + "content": "[\"^ \",\"~:keys\",[[151,\"~:db/valueType\",\"~:db.type/ref\",536870913],[151,\"~:logseq.property/built-in?\",true,536870913],[151,\"~:logseq.property/hide?\",true,536870913],[151,\"~:logseq.property/type\",\"~:property\",536870913],[152,\"~:block/created-at\",1786627375723,536870913],[152,\"~:block/name\",\"history value\",536870913],[152,\"~:block/order\",\"b1G\",536870913],[152,\"~:block/tags\",3,536870913],[152,\"~:block/title\",\"History value\",536870913],[152,\"~:block/updated-at\",1786627375723,536870913],[152,\"~:block/uuid\",\"~u00000002-5131-3603-7000-000000000000\",536870913],[152,\"~:db/cardinality\",\"~:db.cardinality/one\",536870913],[152,\"~:db/ident\",\"~:logseq.property.history/ref-value\",536870913],[152,\"~:db/index\",true,536870913],[152,\"^1\",\"^2\",536870913],[152,\"^3\",true,536870913],[152,\"^4\",true,536870913],[152,\"^5\",\"~:entity\",536870913]]]", + "addresses": null + }, + { + "addr": 1000122, + "content": "[\"^ \",\"~:keys\",[[153,\"~:block/created-at\",1786627375723,536870913],[153,\"~:block/name\",\"history scalar value\",536870913],[153,\"~:block/order\",\"b1H\",536870913],[153,\"~:block/tags\",3,536870913],[153,\"~:block/title\",\"History scalar value\",536870913],[153,\"~:block/updated-at\",1786627375723,536870913],[153,\"~:block/uuid\",\"~u00000002-2393-3777-5000-000000000000\",536870913],[153,\"~:db/cardinality\",\"~:db.cardinality/one\",536870913],[153,\"~:db/ident\",\"~:logseq.property.history/scalar-value\",536870913],[153,\"~:db/index\",true,536870913],[153,\"~:logseq.property/built-in?\",true,536870913],[153,\"~:logseq.property/hide?\",true,536870913],[153,\"~:logseq.property/type\",\"~:any\",536870913],[154,\"^1\",1786627375723,536870913],[154,\"^2\",\"node created by\",536870913],[154,\"^3\",\"b1I\",536870913],[154,\"^4\",3,536870913]]]", + "addresses": null + }, + { + "addr": 1000123, + "content": "[\"^ \",\"~:keys\",[[135,\"~:block/name\",\"image width\",536870913],[136,\"~:block/title\",\"Image height\",536870913],[137,\"~:db/cardinality\",\"~:db.cardinality/one\",536870913],[138,\"~:logseq.property/hide?\",true,536870913],[139,\"~:logseq.property/type\",\"~:map\",536870913],[141,\"^3\",\"^4\",536870913],[142,\"^5\",true,536870913],[144,\"~:block/created-at\",1786627375723,536870913],[145,\"~:block/tags\",3,536870913],[146,\"~:block/uuid\",\"~u00000002-1655-2060-6300-000000000000\",536870913],[147,\"~:db/index\",true,536870913],[148,\"^6\",\"~:checkbox\",536870913],[150,\"^:\",\"~u00000002-1142-5541-6000-000000000000\",536870913],[151,\"^;\",true,536870913],[152,\"^6\",\"~:entity\",536870913],[154,\"^9\",3,536870913]]]", + "addresses": "[1000107,1000108,1000109,1000110,1000111,1000112,1000113,1000114,1000115,1000116,1000117,1000118,1000119,1000120,1000121,1000122]" + }, + { + "addr": 1000124, + "content": "[\"^ \",\"~:keys\",[[154,\"~:block/title\",\"Node created by\",536870913],[154,\"~:block/updated-at\",1786627375723,536870913],[154,\"~:block/uuid\",\"~u00000002-8544-3390-8000-000000000000\",536870913],[154,\"~:db/cardinality\",\"~:db.cardinality/one\",536870913],[154,\"~:db/ident\",\"~:logseq.property/created-by-ref\",536870913],[154,\"~:db/index\",true,536870913],[154,\"~:db/valueType\",\"~:db.type/ref\",536870913],[154,\"~:logseq.property/built-in?\",true,536870913],[154,\"~:logseq.property/hide?\",true,536870913],[154,\"~:logseq.property/type\",\"~:entity\",536870913],[155,\"~:block/created-at\",1786627375723,536870913],[155,\"~:block/name\",\"deleted at\",536870913],[155,\"~:block/order\",\"b1J\",536870913],[155,\"~:block/tags\",3,536870913],[155,\"^1\",\"Deleted at\",536870913],[155,\"^2\",1786627375723,536870913],[155,\"^3\",\"~u00000002-6852-1375-1000-000000000000\",536870913]]]", + "addresses": null + }, + { + "addr": 1000125, + "content": "[\"^ \",\"~:keys\",[[155,\"~:db/cardinality\",\"~:db.cardinality/one\",536870913],[155,\"~:db/ident\",\"~:logseq.property/deleted-at\",536870913],[155,\"~:db/index\",true,536870913],[155,\"~:logseq.property/built-in?\",true,536870913],[155,\"~:logseq.property/hide?\",true,536870913],[155,\"~:logseq.property/public?\",false,536870913],[155,\"~:logseq.property/type\",\"~:datetime\",536870913],[156,\"~:block/created-at\",1786627375723,536870913],[156,\"~:block/name\",\"deleted by\",536870913],[156,\"~:block/order\",\"b1K\",536870913],[156,\"~:block/tags\",3,536870913],[156,\"~:block/title\",\"Deleted by\",536870913],[156,\"~:block/updated-at\",1786627375723,536870913],[156,\"~:block/uuid\",\"~u00000002-1998-0431-1200-000000000000\",536870913],[156,\"^1\",\"^2\",536870913],[156,\"^3\",\"~:logseq.property/deleted-by-ref\",536870913],[156,\"^5\",true,536870913]]]", + "addresses": null + }, + { + "addr": 1000126, + "content": "[\"^ \",\"~:keys\",[[156,\"~:db/valueType\",\"~:db.type/ref\",536870913],[156,\"~:logseq.property/built-in?\",true,536870913],[156,\"~:logseq.property/hide?\",true,536870913],[156,\"~:logseq.property/public?\",false,536870913],[156,\"~:logseq.property/type\",\"~:entity\",536870913],[157,\"~:block/created-at\",1786627375723,536870913],[157,\"~:block/name\",\"recycle original parent\",536870913],[157,\"~:block/order\",\"b1L\",536870913],[157,\"~:block/tags\",3,536870913],[157,\"~:block/title\",\"Recycle original parent\",536870913],[157,\"~:block/updated-at\",1786627375723,536870913],[157,\"~:block/uuid\",\"~u00000002-4970-0399-4000-000000000000\",536870913],[157,\"~:db/cardinality\",\"~:db.cardinality/one\",536870913],[157,\"~:db/ident\",\"~:logseq.property.recycle/original-parent\",536870913],[157,\"~:db/index\",true,536870913],[157,\"^1\",\"^2\",536870913],[157,\"^3\",true,536870913],[157,\"^4\",true,536870913]]]", + "addresses": null + }, + { + "addr": 1000127, + "content": "[\"^ \",\"~:keys\",[[157,\"~:logseq.property/public?\",false,536870913],[157,\"~:logseq.property/type\",\"~:node\",536870913],[158,\"~:block/created-at\",1786627375723,536870913],[158,\"~:block/name\",\"recycle original page\",536870913],[158,\"~:block/order\",\"b1M\",536870913],[158,\"~:block/tags\",3,536870913],[158,\"~:block/title\",\"Recycle original page\",536870913],[158,\"~:block/updated-at\",1786627375723,536870913],[158,\"~:block/uuid\",\"~u00000002-1658-7897-9900-000000000000\",536870913],[158,\"~:db/cardinality\",\"~:db.cardinality/one\",536870913],[158,\"~:db/ident\",\"~:logseq.property.recycle/original-page\",536870913],[158,\"~:db/index\",true,536870913],[158,\"~:db/valueType\",\"~:db.type/ref\",536870913],[158,\"~:logseq.property/built-in?\",true,536870913],[158,\"~:logseq.property/hide?\",true,536870913],[158,\"^1\",false,536870913],[158,\"^2\",\"^3\",536870913]]]", + "addresses": null + }, + { + "addr": 1000128, + "content": "[\"^ \",\"~:keys\",[[159,\"~:block/created-at\",1786627375723,536870913],[159,\"~:block/name\",\"recycle original order\",536870913],[159,\"~:block/order\",\"b1N\",536870913],[159,\"~:block/tags\",3,536870913],[159,\"~:block/title\",\"Recycle original order\",536870913],[159,\"~:block/updated-at\",1786627375723,536870913],[159,\"~:block/uuid\",\"~u00000002-8699-2537-0000-000000000000\",536870913],[159,\"~:db/cardinality\",\"~:db.cardinality/one\",536870913],[159,\"~:db/ident\",\"~:logseq.property.recycle/original-order\",536870913],[159,\"~:db/index\",true,536870913],[159,\"~:logseq.property/built-in?\",true,536870913],[159,\"~:logseq.property/hide?\",true,536870913],[159,\"~:logseq.property/public?\",false,536870913],[159,\"~:logseq.property/type\",\"~:string\",536870913],[160,\"^1\",1786627375723,536870913],[160,\"^2\",\"reaction emoji\",536870913],[160,\"^3\",\"b1O\",536870913]]]", + "addresses": null + }, + { + "addr": 1000129, + "content": "[\"^ \",\"~:keys\",[[160,\"~:block/tags\",3,536870913],[160,\"~:block/title\",\"Reaction emoji\",536870913],[160,\"~:block/updated-at\",1786627375723,536870913],[160,\"~:block/uuid\",\"~u00000002-9877-5864-5000-000000000000\",536870913],[160,\"~:db/cardinality\",\"~:db.cardinality/one\",536870913],[160,\"~:db/ident\",\"~:logseq.property.reaction/emoji-id\",536870913],[160,\"~:db/index\",true,536870913],[160,\"~:logseq.property/built-in?\",true,536870913],[160,\"~:logseq.property/hide?\",true,536870913],[160,\"~:logseq.property/public?\",false,536870913],[160,\"~:logseq.property/type\",\"~:string\",536870913],[161,\"~:block/created-at\",1786627375724,536870913],[161,\"~:block/name\",\"reaction target\",536870913],[161,\"~:block/order\",\"b1P\",536870913],[161,\"^1\",3,536870913],[161,\"^2\",\"Reaction target\",536870913],[161,\"^3\",1786627375724,536870913]]]", + "addresses": null + }, + { + "addr": 1000130, + "content": "[\"^ \",\"~:keys\",[[161,\"~:block/uuid\",\"~u00000002-1127-5718-5000-000000000000\",536870913],[161,\"~:db/cardinality\",\"~:db.cardinality/one\",536870913],[161,\"~:db/ident\",\"~:logseq.property.reaction/target\",536870913],[161,\"~:db/index\",true,536870913],[161,\"~:db/valueType\",\"~:db.type/ref\",536870913],[161,\"~:logseq.property/built-in?\",true,536870913],[161,\"~:logseq.property/hide?\",true,536870913],[161,\"~:logseq.property/public?\",false,536870913],[161,\"~:logseq.property/type\",\"~:node\",536870913],[162,\"~:block/created-at\",1786627375724,536870913],[162,\"~:block/name\",\"agent session id\",536870913],[162,\"~:block/order\",\"b1Q\",536870913],[162,\"~:block/tags\",3,536870913],[162,\"~:block/title\",\"Agent Session ID\",536870913],[162,\"~:block/updated-at\",1786627375724,536870913],[162,\"^1\",\"~u00000002-1783-0312-6200-000000000000\",536870913],[162,\"^2\",\"^3\",536870913]]]", + "addresses": null + }, + { + "addr": 1000131, + "content": "[\"^ \",\"~:keys\",[[162,\"~:db/ident\",\"~:logseq.property.agent/session-id\",536870913],[162,\"~:db/index\",true,536870913],[162,\"~:logseq.property/built-in?\",true,536870913],[162,\"~:logseq.property/description\",163,536870913],[162,\"~:logseq.property/hide?\",true,536870913],[162,\"~:logseq.property/public?\",true,536870913],[162,\"~:logseq.property/type\",\"~:string\",536870913],[163,\"~:block/created-at\",1786627375724,536870913],[163,\"~:block/order\",\"b1R\",536870913],[163,\"~:block/page\",162,536870913],[163,\"~:block/parent\",162,536870913],[163,\"~:block/title\",\"Stores the AgentBridge session ID for a routed task.\",536870913],[163,\"~:block/updated-at\",1786627375724,536870913],[163,\"~:block/uuid\",\"~u00000004-1209-4784-1500-000000000000\",536870913],[163,\"^4\",true,536870913],[163,\"~:logseq.property/created-from-property\",35,536870913],[164,\"^:\",1786627375724,536870913],[164,\"~:block/name\",\"used template\",536870913]]]", + "addresses": null + }, + { + "addr": 1000132, + "content": "[\"^ \",\"~:keys\",[[164,\"~:block/order\",\"b1S\",536870913],[164,\"~:block/tags\",3,536870913],[164,\"~:block/title\",\"Used template\",536870913],[164,\"~:block/updated-at\",1786627375724,536870913],[164,\"~:block/uuid\",\"~u00000002-9803-6990-6000-000000000000\",536870913],[164,\"~:db/cardinality\",\"~:db.cardinality/one\",536870913],[164,\"~:db/ident\",\"~:logseq.property/used-template\",536870913],[164,\"~:db/index\",true,536870913],[164,\"~:db/valueType\",\"~:db.type/ref\",536870913],[164,\"~:logseq.property/built-in?\",true,536870913],[164,\"~:logseq.property/classes\",5,536870913],[164,\"~:logseq.property/hide?\",true,536870913],[164,\"~:logseq.property/public?\",false,536870913],[164,\"~:logseq.property/type\",\"~:node\",536870913],[165,\"~:block/created-at\",1786627375724,536870913],[165,\"~:block/name\",\"apply template to tags\",536870913],[165,\"^1\",\"b1T\",536870913]]]", + "addresses": null + }, + { + "addr": 1000133, + "content": "[\"^ \",\"~:keys\",[[165,\"~:block/tags\",3,536870913],[165,\"~:block/title\",\"Apply template to tags\",536870913],[165,\"~:block/updated-at\",1786627375724,536870913],[165,\"~:block/uuid\",\"~u00000002-4291-2432-2000-000000000000\",536870913],[165,\"~:db/cardinality\",\"~:db.cardinality/many\",536870913],[165,\"~:db/ident\",\"~:logseq.property/template-applied-to\",536870913],[165,\"~:db/index\",true,536870913],[165,\"~:db/valueType\",\"~:db.type/ref\",536870913],[165,\"~:logseq.property/built-in?\",true,536870913],[165,\"~:logseq.property/public?\",true,536870913],[165,\"~:logseq.property/type\",\"~:class\",536870913],[166,\"~:block/created-at\",1786627375724,536870913],[166,\"~:block/name\",\"reference to large block title stored in remote object storage\",536870913],[166,\"~:block/order\",\"b1U\",536870913],[166,\"^1\",3,536870913],[166,\"^2\",\"Reference to large block title stored in remote object storage\",536870913]]]", + "addresses": null + }, + { + "addr": 1000134, + "content": "[\"^ \",\"~:keys\",[[166,\"~:block/updated-at\",1786627375724,536870913],[166,\"~:block/uuid\",\"~u00000002-1044-8271-6500-000000000000\",536870913],[166,\"~:db/cardinality\",\"~:db.cardinality/one\",536870913],[166,\"~:db/ident\",\"~:logseq.property.sync/large-title-object\",536870913],[166,\"~:db/index\",true,536870913],[166,\"~:logseq.property/built-in?\",true,536870913],[166,\"~:logseq.property/hide?\",true,536870913],[166,\"~:logseq.property/public?\",false,536870913],[166,\"~:logseq.property/type\",\"~:map\",536870913],[167,\"~:block/created-at\",1786627375725,536870913],[167,\"~:block/name\",\"journal\",536870913],[167,\"~:block/tags\",2,536870913],[167,\"~:block/title\",\"Journal\",536870913],[167,\"^1\",1786627375725,536870913],[167,\"^2\",\"~u00000002-1979-7410-8100-000000000000\",536870913],[167,\"^5\",\"~:logseq.class/Journal\",536870913]]]", + "addresses": null + }, + { + "addr": 1000135, + "content": "[\"^ \",\"~:keys\",[[167,\"~:logseq.property/built-in?\",true,536870913],[167,\"~:logseq.property.class/extends\",4,536870913],[167,\"~:logseq.property.journal/title-format\",\"MMM do, yyyy\",536870913],[168,\"~:block/created-at\",1786627375725,536870913],[168,\"~:block/name\",\"whiteboard\",536870913],[168,\"~:block/tags\",2,536870913],[168,\"~:block/title\",\"Whiteboard\",536870913],[168,\"~:block/updated-at\",1786627375725,536870913],[168,\"~:block/uuid\",\"~u00000002-1013-6984-5200-000000000000\",536870913],[168,\"~:db/ident\",\"~:logseq.class/Whiteboard\",536870913],[168,\"^1\",true,536870913],[168,\"^2\",4,536870913],[169,\"^4\",1786627375725,536870913],[169,\"^5\",\"task\",536870913],[169,\"^6\",2,536870913],[169,\"^7\",\"Task\",536870913]]]", + "addresses": null + }, + { + "addr": 1000136, + "content": "[\"^ \",\"~:keys\",[[169,\"~:block/updated-at\",1786627375725,536870913],[169,\"~:block/uuid\",\"~u00000002-1282-1814-5700-000000000000\",536870913],[169,\"~:db/ident\",\"~:logseq.class/Task\",536870913],[169,\"~:logseq.property/built-in?\",true,536870913],[169,\"~:logseq.property.class/extends\",1,536870913],[169,\"~:logseq.property.class/properties\",74,536870913],[169,\"^7\",81,536870913],[169,\"^7\",86,536870913],[169,\"^7\",88,536870913],[170,\"~:block/created-at\",1786627375725,536870913],[170,\"~:block/name\",\"comments\",536870913],[170,\"~:block/tags\",2,536870913],[170,\"~:block/title\",\"Comments\",536870913],[170,\"^1\",1786627375725,536870913],[170,\"^2\",\"~u00000002-2556-9161-5000-000000000000\",536870913],[170,\"^3\",\"~:logseq.class/Comments\",536870913]]]", + "addresses": null + }, + { + "addr": 1000137, + "content": "[\"^ \",\"~:keys\",[[170,\"~:logseq.property/built-in?\",true,536870913],[170,\"~:logseq.property/icon\",[\"^ \",\"~:type\",\"~:tabler-icon\",\"~:id\",\"message-circle\"],536870913],[170,\"~:logseq.property.class/extends\",1,536870913],[170,\"~:logseq.property.class/hide-from-node\",true,536870913],[170,\"~:logseq.property.class/properties\",68,536870913],[171,\"~:block/created-at\",1786627375725,536870913],[171,\"~:block/name\",\"comment\",536870913],[171,\"~:block/tags\",2,536870913],[171,\"~:block/title\",\"Comment\",536870913],[171,\"~:block/updated-at\",1786627375725,536870913],[171,\"~:block/uuid\",\"~u00000002-1323-0695-4000-000000000000\",536870913],[171,\"~:db/ident\",\"~:logseq.class/Comment\",536870913],[171,\"^1\",true,536870913],[171,\"^6\",1,536870913],[171,\"^7\",true,536870913],[172,\"^9\",1786627375725,536870913]]]", + "addresses": null + }, + { + "addr": 1000138, + "content": "[\"^ \",\"~:keys\",[[172,\"~:block/name\",\"query\",536870913],[172,\"~:block/tags\",2,536870913],[172,\"~:block/title\",\"Query\",536870913],[172,\"~:block/updated-at\",1786627375725,536870913],[172,\"~:block/uuid\",\"~u00000002-2324-8016-6000-000000000000\",536870913],[172,\"~:db/ident\",\"~:logseq.class/Query\",536870913],[172,\"~:logseq.property/built-in?\",true,536870913],[172,\"~:logseq.property/icon\",[\"^ \",\"~:type\",\"~:tabler-icon\",\"~:id\",\"search\"],536870913],[172,\"~:logseq.property.class/extends\",1,536870913],[172,\"~:logseq.property.class/properties\",48,536870913],[173,\"~:block/created-at\",1786627375725,536870913],[173,\"^1\",\"card\",536870913],[173,\"^2\",2,536870913],[173,\"^3\",\"Card\",536870913],[173,\"^4\",1786627375725,536870913],[173,\"^5\",\"~u00000002-1358-2811-0900-000000000000\",536870913]]]", + "addresses": null + }, + { + "addr": 1000139, + "content": "[\"^ \",\"~:keys\",[[173,\"~:db/ident\",\"~:logseq.class/Card\",536870913],[173,\"~:logseq.property/built-in?\",true,536870913],[173,\"~:logseq.property.class/extends\",1,536870913],[173,\"~:logseq.property.class/properties\",143,536870913],[173,\"^5\",144,536870913],[174,\"~:block/created-at\",1786627375725,536870913],[174,\"~:block/name\",\"cards\",536870913],[174,\"~:block/tags\",2,536870913],[174,\"~:block/title\",\"Cards\",536870913],[174,\"~:block/updated-at\",1786627375725,536870913],[174,\"~:block/uuid\",\"~u00000002-1284-2651-6700-000000000000\",536870913],[174,\"^1\",\"~:logseq.class/Cards\",536870913],[174,\"^3\",true,536870913],[174,\"~:logseq.property/icon\",[\"^ \",\"~:type\",\"~:tabler-icon\",\"~:id\",\"search\"],536870913],[174,\"^4\",172,536870913],[175,\"^6\",1786627375725,536870913]]]", + "addresses": null + }, + { + "addr": 1000140, + "content": "[\"^ \",\"~:keys\",[[175,\"~:block/name\",\"asset\",536870913],[175,\"~:block/tags\",2,536870913],[175,\"~:block/title\",\"Asset\",536870913],[175,\"~:block/updated-at\",1786627375725,536870913],[175,\"~:block/uuid\",\"~u00000002-7975-0297-0000-000000000000\",536870913],[175,\"~:db/ident\",\"~:logseq.class/Asset\",536870913],[175,\"~:logseq.property/built-in?\",true,536870913],[175,\"~:logseq.property.class/extends\",1,536870913],[175,\"~:logseq.property.class/hide-from-node\",true,536870913],[175,\"~:logseq.property.class/properties\",131,536870913],[175,\"^;\",134,536870913],[175,\"^;\",137,536870913],[175,\"~:logseq.property.view/type\",114,536870913],[176,\"~:block/created-at\",1786627375725,536870913],[176,\"^1\",\"code\",536870913],[176,\"^2\",2,536870913]]]", + "addresses": null + }, + { + "addr": 1000141, + "content": "[\"^ \",\"~:keys\",[[176,\"~:block/title\",\"Code\",536870913],[176,\"~:block/updated-at\",1786627375725,536870913],[176,\"~:block/uuid\",\"~u00000002-1454-9866-4100-000000000000\",536870913],[176,\"~:db/ident\",\"~:logseq.class/Code-block\",536870913],[176,\"~:logseq.property/built-in?\",true,536870913],[176,\"~:logseq.property.class/extends\",1,536870913],[176,\"~:logseq.property.class/hide-from-node\",true,536870913],[176,\"~:logseq.property.class/properties\",34,536870913],[176,\"^9\",36,536870913],[177,\"~:block/created-at\",1786627375725,536870913],[177,\"~:block/name\",\"quote\",536870913],[177,\"~:block/tags\",2,536870913],[177,\"^1\",\"Quote\",536870913],[177,\"^2\",1786627375725,536870913],[177,\"^3\",\"~u00000002-1176-1666-1700-000000000000\",536870913],[177,\"^4\",\"~:logseq.class/Quote-block\",536870913]]]", + "addresses": null + }, + { + "addr": 1000142, + "content": "[\"^ \",\"~:keys\",[[177,\"~:logseq.property/built-in?\",true,536870913],[177,\"~:logseq.property.class/extends\",1,536870913],[177,\"~:logseq.property.class/hide-from-node\",true,536870913],[177,\"~:logseq.property.class/properties\",34,536870913],[178,\"~:block/created-at\",1786627375725,536870913],[178,\"~:block/name\",\"math\",536870913],[178,\"~:block/tags\",2,536870913],[178,\"~:block/title\",\"Math\",536870913],[178,\"~:block/updated-at\",1786627375725,536870913],[178,\"~:block/uuid\",\"~u00000002-2038-9631-2100-000000000000\",536870913],[178,\"~:db/ident\",\"~:logseq.class/Math-block\",536870913],[178,\"^1\",true,536870913],[178,\"^2\",1,536870913],[178,\"^3\",true,536870913],[178,\"^4\",34,536870913],[179,\"^5\",1786627375725,536870913]]]", + "addresses": null + }, + { + "addr": 1000143, + "content": "[\"^ \",\"~:keys\",[[179,\"~:block/name\",\"pdf annotation\",536870913],[179,\"~:block/tags\",2,536870913],[179,\"~:block/title\",\"PDF Annotation\",536870913],[179,\"~:block/updated-at\",1786627375725,536870913],[179,\"~:block/uuid\",\"~u00000002-5049-5962-0000-000000000000\",536870913],[179,\"~:db/ident\",\"~:logseq.class/Pdf-annotation\",536870913],[179,\"~:logseq.property/built-in?\",true,536870913],[179,\"~:logseq.property.class/extends\",1,536870913],[179,\"~:logseq.property.class/hide-from-node\",true,536870913],[179,\"~:logseq.property.class/properties\",53,536870913],[179,\"^;\",54,536870913],[179,\"^;\",55,536870913],[179,\"^;\",56,536870913],[179,\"^;\",62,536870913],[179,\"^;\",63,536870913],[179,\"^;\",64,536870913]]]", + "addresses": null + }, + { + "addr": 1000144, + "content": "[\"^ \",\"~:keys\",[[180,\"~:block/uuid\",\"~u00000004-3919-3813-3000-000000000000\",536870913],[180,\"~:file/content\",\"{:meta/version 1\\n\\n ;; Hide empty block properties\\n ;; Default value: false\\n ;; :ui/hide-empty-properties? false\\n\\n ;; Enable tooltip preview on hover.\\n ;; Default value: true\\n :ui/enable-tooltip? true\\n\\n ;; Display brackets [[]] around page references.\\n ;; Default value: true\\n ;; :ui/show-brackets? true\\n\\n ;; Display all lines of a block when referencing ((block)).\\n ;; Default value: false\\n :ui/show-full-blocks? false\\n\\n ;; Automatically expand block references when zooming in.\\n ;; Default value: true\\n :ui/auto-expand-block-refs? true\\n\\n ;; Disable accent marks when searching.\\n ;; After changing this setting, rebuild the search index by pressing (^C ^S).\\n ;; Default value: true\\n :feature/enable-search-remove-accents? true\\n\\n ;; Enable flashcards.\\n ;; Default value: true\\n ;; :feature/enable-flashcards? true\\n\\n ;; Disable the journal's built-in 'Scheduled tasks and deadlines' query.\\n ;; Default value: false\\n ;; :feature/disable-scheduled-and-deadline-query? false\\n\\n ;; Specify the number of days displayed in the future for\\n ;; the 'scheduled tasks and deadlines' query.\\n ;; Example usage:\\n ;; Display all scheduled and deadline blocks for the next 14 days:\\n ;; :scheduled/future-days 14\\n ;; Default value: 7\\n ;; :scheduled/future-days 7\\n\\n ;; Specify the first day of the week.\\n ;; Available options:\\n ;; - integer from 0 to 6 (Monday to Sunday)\\n ;; Default value: 6 (Sunday)\\n :start-of-week 6\\n\\n ;; Specify a custom CSS import.\\n ;; This option takes precedence over the local `logseq/custom.css` file.\\n ;; Example usage:\\n ;; :custom-css-url \\\"@import url('https://cdn.jsdelivr.net/gh/dracula/logseq@master/custom.css');\\\"\\n\\n ;; Specify a custom JS import.\\n ;; This option takes precedence over the local `logseq/custom.js` file.\\n ;; Example usage:\\n ;; :custom-js-url \\\"https://cdn.logseq.com/custom.js\\\"\\n\\n ;; Set bullet indentation when exporting\\n ;; Available options:\\n ;; - `:eight-spaces` as eight spaces\\n ;; - `:four-spaces` as four spaces\\n ;; - `:two-spaces` as two spaces\\n ;; - `:tab` as a tab character (default)\\n ;; :export/bullet-indentation :tab\\n\\n ;; Publish all pages within the Graph\\n ;; Regardless of whether individual pages have been marked as public.\\n ;; Default value: false\\n ;; :publishing/all-pages-public? false\\n\\n ;; Define the default home page and sidebar status.\\n ;; If unspecified, the journal page will be loaded on startup and the right sidebar will stay hidden.\\n ;; The `:page` value represents the name of the page displayed at startup.\\n ;; Available options for `:sidebar` are:\\n ;; - \\\"Contents\\\" to display the Contents page in the right sidebar.\\n ;; - A specific page name to display in the right sidebar.\\n ;; - An array of multiple pages, e.g., [\\\"Contents\\\" \\\"Page A\\\" \\\"Page B\\\"].\\n ;; If `:sidebar` remains unset, the right sidebar will stay hidden.\\n ;; Examples:\\n ;; 1. Set \\\"Changelog\\\" as the home page and display \\\"Contents\\\" in the right sidebar:\\n ;; :default-home {:page \\\"Changelog\\\", :sidebar \\\"Contents\\\"}\\n ;; 2. Set \\\"Jun 3rd, 2021\\\" as the home page without the right sidebar:\\n ;; :default-home {:page \\\"Jun 3rd, 2021\\\"}\\n ;; 3. Set \\\"home\\\" as the home page and display multiple pages in the right sidebar:\\n ;; :default-home {:page \\\"home\\\", :sidebar [\\\"Page A\\\" \\\"Page B\\\"]}\\n\\n ;; Configure custom shortcuts.\\n ;; Syntax:\\n ;; 1. + indicates simultaneous key presses, e.g., `Ctrl+Shift+a`.\\n ;; 2. A space between keys represents key chords, e.g., `t s` means\\n ;; pressing `t` followed by `s`.\\n ;; 3. mod refers to `Ctrl` for Windows/Linux and `Command` for Mac.\\n ;; 4. Use false to disable a specific shortcut.\\n ;; 5. You can define multiple bindings for a single action, e.g., [\\\"ctrl+j\\\" \\\"down\\\"].\\n ;; The full list of configurable shortcuts is available at:\\n ;; https://github.com/logseq/logseq/blob/master/src/main/frontend/modules/shortcut/config.cljs\\n ;; Example:\\n ;; :shortcuts\\n ;; {:editor/new-block \\\"enter\\\"\\n ;; :editor/new-line \\\"shift+enter\\\"\\n ;; :editor/insert-link \\\"mod+shift+k\\\"\\n ;; :editor/highlight false\\n ;; :ui/toggle-settings \\\"t s\\\"\\n ;; :editor/up [\\\"ctrl+k\\\" \\\"up\\\"]\\n ;; :editor/down [\\\"ctrl+j\\\" \\\"down\\\"]\\n ;; :editor/left [\\\"ctrl+h\\\" \\\"left\\\"]\\n ;; :editor/right [\\\"ctrl+l\\\" \\\"right\\\"]}\\n :shortcuts {}\\n\\n ;; Configure the behavior of pressing Enter in document mode.\\n ;; if set to true, pressing Enter will create a new block.\\n ;; Default value: false\\n :shortcut/doc-mode-enter-for-new-block? false\\n\\n ;; Block content larger than `block/title-max-length` will not be searchable\\n ;; or editable for performance.\\n ;; Default value: 10000\\n :block/title-max-length 10000\\n\\n ;; Display command documentation on hover.\\n ;; Default value: true\\n :ui/show-command-doc? true\\n\\n ;; Display empty bullet points.\\n ;; Default value: false\\n :ui/show-empty-bullets? false\\n\\n ;; Pre-defined :view function to use with advanced queries.\\n :query/views\\n {:pprint\\n (fn [r] [:pre.code (pprint r)])}\\n\\n ;; Advanced queries `:result-transform` function.\\n ;; Transform the query result before displaying it.\\n ;; Example usage for DB graphs:\\n;; :query/result-transforms\\n;; {:sort-by-priority\\n;; (fn [result] (sort-by (fn [h] (get h :logseq.property/priority \\\"Z\\\")) result))}\\n\\n;; Queries will be displayed at the bottom of today's journal page.\\n;; Example usage:\\n;; :default-queries\\n;; {:journals []}\\n\\n ;; Add custom commands to the command palette\\n ;; Example usage:\\n ;; :commands\\n ;; [\\n ;; [\\\"js\\\" \\\"Javascript\\\"]\\n ;; [\\\"md\\\" \\\"Markdown\\\"]\\n ;; ]\\n :commands []\\n\\n ;; Enable collapsing blocks with titles but no children.\\n ;; By default, only blocks with children can be collapsed.\\n ;; Setting `:outliner/block-title-collapse-enabled?` to true allows collapsing\\n ;; blocks with titles (multiple lines) and content. For example:\\n ;; - block title\\n ;; block content\\n ;; Default value: false\\n :outliner/block-title-collapse-enabled? false\\n\\n ;; Macros replace texts and will make you more productive.\\n ;; Example usage:\\n ;; Change the :macros value below to:\\n ;; {\\\"poem\\\" \\\"Rose is $1, violet's $2. Life's ordered: Org assists you.\\\"}\\n ;; input \\\"{{poem red,blue}}\\\"\\n ;; becomes\\n ;; Rose is red, violet's blue. Life's ordered: Org assists you.\\n :macros {}\\n\\n ;; Configure the default expansion level for linked references.\\n ;; For example, consider the following block hierarchy:\\n ;; - a [[page]] (level 1)\\n ;; - b (level 2)\\n ;; - c (level 3)\\n ;; - d (level 4)\\n ;;\\n ;; With the default value of level 2, block b will be collapsed.\\n ;; If the level's value is set to 3, block c will be collapsed.\\n ;; Default value: 2\\n :ref/default-open-blocks-level 2\\n\\n ;; Mobile photo upload configuration.\\n ;; :mobile/photo\\n ;; {:allow-editing? true\\n ;; :quality 80}\\n\\n ;; Mobile features options\\n ;; Gestures\\n ;; Example usage:\\n ;; :mobile\\n ;; {:gestures/disabled-in-block-with-tags [\\\"kanban\\\"]}\\n\\n ;; Extra CodeMirror options\\n ;; See https://codemirror.net/5/doc/manual.html#config for possible options\\n ;; Example usage:\\n ;; :editor/extra-codemirror-options\\n ;; {:lineWrapping false ; Default value: false\\n ;; :lineNumbers true ; Default value: true\\n ;; :readOnly false} ; Default value: false\\n\\n ;; Enable logical outdenting\\n ;; Default value: false\\n ;; :editor/logical-outdenting? false\\n\\n ;; Prefer pasting the file when text and a file are in the clipboard.\\n ;; Default value: false\\n ;; :editor/preferred-pasting-file? false\\n\\n ;; Quick capture templates for receiving content from other apps.\\n ;; Each template contains three elements {time}, {text} and {url}, which can be auto-expanded\\n ;; by receiving content from other apps. Note: the {} cannot be omitted.\\n ;; - {time}: capture time\\n ;; - {date}: capture date using current date format, use `[[{date}]]` to get a page reference\\n ;; - {text}: text that users selected before sharing.\\n ;; - {url}: URL or assets path for media files stored in Logseq.\\n ;; You can also reorder them or use only one or two of them in the template.\\n ;; You can also insert or format any text in the template, as shown in the following examples.\\n ;; :quick-capture-templates\\n ;; {:text \\\"[[quick capture]] **{time}**: {text} from {url}\\\"\\n ;; :media \\\"[[quick capture]] **{time}**: {url}\\\"}\\n\\n ;; Quick capture options.\\n ;; - insert-today? Insert the capture at the end of today's journal page (boolean).\\n ;; - redirect-page? Redirect to the quick capture page after capturing (boolean).\\n ;; - default-page The default page to capture to if insert-today? is false (string).\\n ;; - prettify-url? Whether urls to YouTube or X should be wrapped with {{video ...}} or {{twitter ...}}\\n ;; :quick-capture-options\\n ;; {:insert-today? false ;; Default value: true\\n ;; :redirect-page? false ;; Default value: false\\n ;; :default-page \\\"quick capture\\\" ;; Default page: \\\"quick capture\\\"\\n ;; :prettify-url? true} ;; Default value: true\\n\\n }\\n\",536870913],[180,\"~:file/created-at\",\"~m1786627375714\",536870913],[180,\"~:file/last-modified-at\",\"~m1786627375714\",536870913],[180,\"~:file/path\",\"logseq/config.edn\",536870913],[181,\"^1\",\"~u00000004-1713-4660-3800-000000000000\",536870913],[181,\"^2\",\"\",536870913],[181,\"^3\",\"~m1786627375714\",536870913],[181,\"^4\",\"~m1786627375714\",536870913],[181,\"^5\",\"logseq/custom.css\",536870913],[182,\"^1\",\"~u00000004-1335-6485-2300-000000000000\",536870913],[182,\"^2\",\"\",536870913],[182,\"^3\",\"~m1786627375714\",536870913],[182,\"^4\",\"~m1786627375714\",536870913],[182,\"^5\",\"logseq/custom.js\",536870913],[183,\"^1\",\"~u00000004-4049-4381-0000-000000000000\",536870913]]]", + "addresses": null + }, + { + "addr": 1000145, + "content": "[\"^ \",\"~:keys\",[[183,\"~:file/content\",\"\",536870913],[183,\"~:file/created-at\",\"~m1786627375714\",536870913],[183,\"~:file/last-modified-at\",\"~m1786627375714\",536870913],[183,\"~:file/path\",\"logseq/publish.css\",536870913],[184,\"~:block/uuid\",\"~u00000004-2116-2824-5200-000000000000\",536870913],[184,\"^1\",\"\",536870913],[184,\"^2\",\"~m1786627375714\",536870913],[184,\"^3\",\"~m1786627375714\",536870913],[184,\"^4\",\"logseq/publish.js\",536870913],[185,\"~:block/created-at\",1786627375725,536870913],[185,\"~:block/name\",\"library\",536870913],[185,\"~:block/tags\",4,536870913],[185,\"~:block/title\",\"Library\",536870913],[185,\"~:block/updated-at\",1786627375725,536870913],[185,\"^5\",\"~u00000004-1294-7765-6000-000000000000\",536870913],[185,\"~:logseq.property/built-in?\",true,536870913]]]", + "addresses": null + }, + { + "addr": 1000146, + "content": "[\"^ \",\"~:keys\",[[186,\"~:block/created-at\",1786627375725,536870913],[186,\"~:block/name\",\"contents\",536870913],[186,\"~:block/tags\",4,536870913],[186,\"~:block/title\",\"Contents\",536870913],[186,\"~:block/updated-at\",1786627375725,536870913],[186,\"~:block/uuid\",\"~u00000004-1690-2597-3200-000000000000\",536870913],[186,\"~:logseq.property/built-in?\",true,536870913],[187,\"^1\",1786627375725,536870913],[187,\"^2\",\"quick add\",536870913],[187,\"^3\",4,536870913],[187,\"^4\",\"Quick add\",536870913],[187,\"^5\",1786627375725,536870913],[187,\"^6\",\"~u00000004-7336-8251-3000-000000000000\",536870913],[187,\"^7\",true,536870913],[187,\"~:logseq.property/hide?\",true,536870913],[188,\"^1\",1786627375725,536870913]]]", + "addresses": null + }, + { + "addr": 1000147, + "content": "[\"^ \",\"~:keys\",[[188,\"~:block/name\",\"$$$views\",536870913],[188,\"~:block/tags\",4,536870913],[188,\"~:block/title\",\"$$$views\",536870913],[188,\"~:block/updated-at\",1786627375725,536870913],[188,\"~:block/uuid\",\"~u00000004-1906-3437-5800-000000000000\",536870913],[188,\"~:logseq.property/built-in?\",true,536870913],[188,\"~:logseq.property/hide?\",true,536870913],[189,\"~:block/created-at\",1786627375725,536870913],[189,\"^1\",\"$$$favorites\",536870913],[189,\"^2\",4,536870913],[189,\"^3\",\"$$$favorites\",536870913],[189,\"^4\",1786627375725,536870913],[189,\"^5\",\"~u00000004-1018-5888-4100-000000000000\",536870913],[189,\"^6\",true,536870913],[189,\"^7\",true,536870913],[190,\"^8\",1786627375725,536870913],[190,\"^1\",\"recycle\",536870913],[190,\"^2\",4,536870913],[190,\"^3\",\"Recycle\",536870913],[190,\"^4\",1786627375725,536870913],[190,\"^5\",\"~u00000004-7238-1304-3000-000000000000\",536870913],[190,\"^6\",true,536870913],[190,\"^7\",true,536870913]]]", + "addresses": null + }, + { + "addr": 1000148, + "content": "[\"^ \",\"~:keys\",[[155,\"~:block/uuid\",\"~u00000002-6852-1375-1000-000000000000\",536870913],[156,\"~:db/index\",true,536870913],[157,\"~:logseq.property/hide?\",true,536870913],[158,\"~:logseq.property/type\",\"~:node\",536870913],[160,\"~:block/order\",\"b1O\",536870913],[161,\"~:block/updated-at\",1786627375724,536870913],[162,\"~:db/cardinality\",\"~:db.cardinality/one\",536870913],[164,\"~:block/name\",\"used template\",536870913],[165,\"^6\",\"b1T\",536870913],[166,\"~:block/title\",\"Reference to large block title stored in remote object storage\",536870913],[167,\"~:db/ident\",\"~:logseq.class/Journal\",536870913],[169,\"^;\",\"Task\",536870913],[170,\"^<\",\"~:logseq.class/Comments\",536870913],[172,\"~:block/created-at\",1786627375725,536870913],[173,\"^1\",\"~u00000002-1358-2811-0900-000000000000\",536870913],[175,\"^?\",1786627375725,536870913],[176,\"~:block/tags\",2,536870913],[177,\"^<\",\"~:logseq.class/Quote-block\",536870913],[179,\"^?\",1786627375725,536870913],[179,\"~:logseq.property.class/properties\",64,536870913],[183,\"^1\",\"~u00000004-4049-4381-0000-000000000000\",536870913],[185,\"~:logseq.property/built-in?\",true,536870913],[188,\"^?\",1786627375725,536870913],[190,\"^3\",true,536870913]]]", + "addresses": "[1000124,1000125,1000126,1000127,1000128,1000129,1000130,1000131,1000132,1000133,1000134,1000135,1000136,1000137,1000138,1000139,1000140,1000141,1000142,1000143,1000144,1000145,1000146,1000147]" + }, + { + "addr": 1000149, + "content": "[\"^ \",\"~:keys\",[[30,\"~:block/order\",\"aI\",536870913],[49,\"~:logseq.property/type\",\"~:page\",536870913],[71,\"~:db/cardinality\",\"~:db.cardinality/many\",536870913],[92,\"~:logseq.property/public?\",false,536870913],[114,\"~:db/ident\",\"~:logseq.property.view/type.gallery\",536870913],[133,\"^6\",false,536870913],[154,\"~:block/tags\",3,536870913],[190,\"~:logseq.property/hide?\",true,536870913]]]", + "addresses": "[1000021,1000038,1000055,1000072,1000089,1000106,1000123,1000148]" + }, + { + "addr": 1000150, + "content": "[\"^ \",\"~:keys\",[[95,\"~:block/closed-value-property\",92,536870913],[96,\"^1\",92,536870913],[97,\"^1\",92,536870913],[98,\"^1\",92,536870913],[101,\"^1\",100,536870913],[102,\"^1\",100,536870913],[103,\"^1\",100,536870913],[112,\"^1\",111,536870913],[113,\"^1\",111,536870913],[114,\"^1\",111,536870913],[1,\"~:block/created-at\",1786627375725,536870913],[2,\"^2\",1786627375725,536870913],[3,\"^2\",1786627375725,536870913],[4,\"^2\",1786627375725,536870913],[5,\"^2\",1786627375725,536870913],[12,\"^2\",1786627375714,536870913],[13,\"^2\",1786627375714,536870913],[14,\"^2\",1786627375714,536870913],[15,\"^2\",1786627375714,536870913],[16,\"^2\",1786627375715,536870913],[17,\"^2\",1786627375715,536870913],[18,\"^2\",1786627375715,536870913],[19,\"^2\",1786627375715,536870913],[20,\"^2\",1786627375716,536870913],[21,\"^2\",1786627375716,536870913],[22,\"^2\",1786627375716,536870913],[23,\"^2\",1786627375716,536870913],[24,\"^2\",1786627375716,536870913],[25,\"^2\",1786627375716,536870913],[26,\"^2\",1786627375716,536870913],[27,\"^2\",1786627375716,536870913]]]", + "addresses": null + }, + { + "addr": 1000151, + "content": "[\"^ \",\"~:keys\",[[28,\"~:block/created-at\",1786627375716,536870913],[29,\"^1\",1786627375717,536870913],[30,\"^1\",1786627375717,536870913],[31,\"^1\",1786627375717,536870913],[32,\"^1\",1786627375717,536870913],[33,\"^1\",1786627375717,536870913],[34,\"^1\",1786627375717,536870913],[35,\"^1\",1786627375717,536870913],[36,\"^1\",1786627375717,536870913],[37,\"^1\",1786627375717,536870913],[38,\"^1\",1786627375717,536870913],[39,\"^1\",1786627375717,536870913],[40,\"^1\",1786627375717,536870913],[41,\"^1\",1786627375717,536870913],[42,\"^1\",1786627375717,536870913],[43,\"^1\",1786627375717,536870913]]]", + "addresses": null + }, + { + "addr": 1000152, + "content": "[\"^ \",\"~:keys\",[[44,\"~:block/created-at\",1786627375717,536870913],[45,\"^1\",1786627375717,536870913],[46,\"^1\",1786627375718,536870913],[47,\"^1\",1786627375718,536870913],[48,\"^1\",1786627375718,536870913],[49,\"^1\",1786627375718,536870913],[50,\"^1\",1786627375718,536870913],[51,\"^1\",1786627375718,536870913],[52,\"^1\",1786627375718,536870913],[53,\"^1\",1786627375718,536870913],[54,\"^1\",1786627375718,536870913],[55,\"^1\",1786627375718,536870913],[56,\"^1\",1786627375718,536870913],[57,\"^1\",1786627375718,536870913],[58,\"^1\",1786627375718,536870913],[59,\"^1\",1786627375718,536870913]]]", + "addresses": null + }, + { + "addr": 1000153, + "content": "[\"^ \",\"~:keys\",[[60,\"~:block/created-at\",1786627375718,536870913],[61,\"^1\",1786627375718,536870913],[62,\"^1\",1786627375719,536870913],[63,\"^1\",1786627375719,536870913],[64,\"^1\",1786627375719,536870913],[65,\"^1\",1786627375719,536870913],[66,\"^1\",1786627375719,536870913],[67,\"^1\",1786627375719,536870913],[68,\"^1\",1786627375719,536870913],[69,\"^1\",1786627375719,536870913],[70,\"^1\",1786627375719,536870913],[71,\"^1\",1786627375719,536870913],[72,\"^1\",1786627375719,536870913],[73,\"^1\",1786627375719,536870913],[74,\"^1\",1786627375719,536870913],[75,\"^1\",1786627375719,536870913]]]", + "addresses": null + }, + { + "addr": 1000154, + "content": "[\"^ \",\"~:keys\",[[76,\"~:block/created-at\",1786627375719,536870913],[77,\"^1\",1786627375719,536870913],[78,\"^1\",1786627375719,536870913],[79,\"^1\",1786627375719,536870913],[80,\"^1\",1786627375720,536870913],[81,\"^1\",1786627375720,536870913],[82,\"^1\",1786627375720,536870913],[83,\"^1\",1786627375720,536870913],[84,\"^1\",1786627375720,536870913],[85,\"^1\",1786627375720,536870913],[86,\"^1\",1786627375720,536870913],[87,\"^1\",1786627375720,536870913],[88,\"^1\",1786627375720,536870913],[89,\"^1\",1786627375720,536870913],[90,\"^1\",1786627375720,536870913],[91,\"^1\",1786627375720,536870913]]]", + "addresses": null + }, + { + "addr": 1000155, + "content": "[\"^ \",\"~:keys\",[[92,\"~:block/created-at\",1786627375720,536870913],[93,\"^1\",1786627375720,536870913],[94,\"^1\",1786627375720,536870913],[95,\"^1\",1786627375721,536870913],[96,\"^1\",1786627375721,536870913],[97,\"^1\",1786627375721,536870913],[98,\"^1\",1786627375721,536870913],[99,\"^1\",1786627375721,536870913],[100,\"^1\",1786627375721,536870913],[101,\"^1\",1786627375721,536870913],[102,\"^1\",1786627375721,536870913],[103,\"^1\",1786627375721,536870913],[104,\"^1\",1786627375721,536870913],[105,\"^1\",1786627375721,536870913],[106,\"^1\",1786627375721,536870913],[107,\"^1\",1786627375721,536870913]]]", + "addresses": null + }, + { + "addr": 1000156, + "content": "[\"^ \",\"~:keys\",[[108,\"~:block/created-at\",1786627375721,536870913],[109,\"^1\",1786627375721,536870913],[110,\"^1\",1786627375721,536870913],[111,\"^1\",1786627375721,536870913],[112,\"^1\",1786627375721,536870913],[113,\"^1\",1786627375721,536870913],[114,\"^1\",1786627375721,536870913],[115,\"^1\",1786627375721,536870913],[116,\"^1\",1786627375721,536870913],[117,\"^1\",1786627375721,536870913],[118,\"^1\",1786627375722,536870913],[119,\"^1\",1786627375722,536870913],[120,\"^1\",1786627375722,536870913],[121,\"^1\",1786627375722,536870913],[122,\"^1\",1786627375722,536870913],[123,\"^1\",1786627375722,536870913]]]", + "addresses": null + }, + { + "addr": 1000157, + "content": "[\"^ \",\"~:keys\",[[124,\"~:block/created-at\",1786627375722,536870913],[125,\"^1\",1786627375722,536870913],[126,\"^1\",1786627375722,536870913],[127,\"^1\",1786627375722,536870913],[128,\"^1\",1786627375722,536870913],[129,\"^1\",1786627375722,536870913],[130,\"^1\",1786627375722,536870913],[131,\"^1\",1786627375722,536870913],[132,\"^1\",1786627375722,536870913],[133,\"^1\",1786627375722,536870913],[134,\"^1\",1786627375722,536870913],[135,\"^1\",1786627375722,536870913],[136,\"^1\",1786627375722,536870913],[137,\"^1\",1786627375722,536870913],[138,\"^1\",1786627375722,536870913],[139,\"^1\",1786627375722,536870913]]]", + "addresses": null + }, + { + "addr": 1000158, + "content": "[\"^ \",\"~:keys\",[[140,\"~:block/created-at\",1786627375722,536870913],[141,\"^1\",1786627375722,536870913],[142,\"^1\",1786627375722,536870913],[143,\"^1\",1786627375723,536870913],[144,\"^1\",1786627375723,536870913],[145,\"^1\",1786627375723,536870913],[146,\"^1\",1786627375723,536870913],[147,\"^1\",1786627375723,536870913],[148,\"^1\",1786627375723,536870913],[149,\"^1\",1786627375723,536870913],[150,\"^1\",1786627375723,536870913],[151,\"^1\",1786627375723,536870913],[152,\"^1\",1786627375723,536870913],[153,\"^1\",1786627375723,536870913],[154,\"^1\",1786627375723,536870913],[155,\"^1\",1786627375723,536870913]]]", + "addresses": null + }, + { + "addr": 1000159, + "content": "[\"^ \",\"~:keys\",[[156,\"~:block/created-at\",1786627375723,536870913],[157,\"^1\",1786627375723,536870913],[158,\"^1\",1786627375723,536870913],[159,\"^1\",1786627375723,536870913],[160,\"^1\",1786627375723,536870913],[161,\"^1\",1786627375724,536870913],[162,\"^1\",1786627375724,536870913],[163,\"^1\",1786627375724,536870913],[164,\"^1\",1786627375724,536870913],[165,\"^1\",1786627375724,536870913],[166,\"^1\",1786627375724,536870913],[167,\"^1\",1786627375725,536870913],[168,\"^1\",1786627375725,536870913],[169,\"^1\",1786627375725,536870913],[170,\"^1\",1786627375725,536870913],[171,\"^1\",1786627375725,536870913]]]", + "addresses": null + }, + { + "addr": 1000160, + "content": "[\"^ \",\"~:keys\",[[172,\"~:block/created-at\",1786627375725,536870913],[173,\"^1\",1786627375725,536870913],[174,\"^1\",1786627375725,536870913],[175,\"^1\",1786627375725,536870913],[176,\"^1\",1786627375725,536870913],[177,\"^1\",1786627375725,536870913],[178,\"^1\",1786627375725,536870913],[179,\"^1\",1786627375725,536870913],[185,\"^1\",1786627375725,536870913],[186,\"^1\",1786627375725,536870913],[187,\"^1\",1786627375725,536870913],[188,\"^1\",1786627375725,536870913],[189,\"^1\",1786627375725,536870913],[190,\"^1\",1786627375725,536870913],[1,\"~:block/name\",\"root tag\",536870913],[2,\"^2\",\"tag\",536870913],[3,\"^2\",\"property\",536870913],[4,\"^2\",\"page\",536870913],[5,\"^2\",\"template\",536870913],[12,\"^2\",\"property type\",536870913]]]", + "addresses": null + }, + { + "addr": 1000161, + "content": "[\"^ \",\"~:keys\",[[13,\"~:block/name\",\"built in?\",536870913],[14,\"^1\",\"hide this property or page\",536870913],[15,\"^1\",\"created from property\",536870913],[16,\"^1\",\"property public?\",536870913],[17,\"^1\",\"property view context\",536870913],[18,\"^1\",\"property position\",536870913],[19,\"^1\",\"property classes\",536870913],[20,\"^1\",\"property value\",536870913],[21,\"^1\",\"alias\",536870913],[22,\"^1\",\"tags\",536870913],[23,\"^1\",\"node parent\",536870913],[24,\"^1\",\"node order\",536870913],[25,\"^1\",\"node collapsed?\",536870913],[26,\"^1\",\"node page\",536870913],[27,\"^1\",\"node references\",536870913],[28,\"^1\",\"node links to\",536870913]]]", + "addresses": null + }, + { + "addr": 1000162, + "content": "[\"^ \",\"~:keys\",[[29,\"~:block/name\",\"node title\",536870913],[30,\"^1\",\"closed value property\",536870913],[31,\"^1\",\"journal date\",536870913],[32,\"^1\",\"node created at\",536870913],[33,\"^1\",\"node updated at\",536870913],[34,\"^1\",\"node display type\",536870913],[35,\"^1\",\"description\",536870913],[36,\"^1\",\"code mode\",536870913],[37,\"^1\",\"default value\",536870913],[38,\"^1\",\"non ref type default value\",536870913],[39,\"^1\",\"extends\",536870913],[41,\"^1\",\"tag properties\",536870913],[42,\"^1\",\"bidirectional property title\",536870913],[43,\"^1\",\"enable bidirectional properties\",536870913],[45,\"^1\",\"hide empty value\",536870913],[47,\"^1\",\"hide from node\",536870913]]]", + "addresses": null + }, + { + "addr": 1000163, + "content": "[\"^ \",\"~:keys\",[[48,\"~:block/name\",\"query\",536870913],[49,\"^1\",\"page tags\",536870913],[51,\"^1\",\"background color\",536870913],[52,\"^1\",\"heading\",536870913],[53,\"^1\",\"asset\",536870913],[54,\"^1\",\"ls-type\",536870913],[55,\"^1\",\"annotation type\",536870913],[56,\"^1\",\"annotation color\",536870913],[62,\"^1\",\"annotation page\",536870913],[63,\"^1\",\"annotation image\",536870913],[64,\"^1\",\"annotation data\",536870913],[65,\"^1\",\"list type\",536870913],[66,\"^1\",\"included references\",536870913],[67,\"^1\",\"excluded references\",536870913],[68,\"^1\",\"commented blocks\",536870913],[69,\"^1\",\"title format\",536870913]]]", + "addresses": null + }, + { + "addr": 1000164, + "content": "[\"^ \",\"~:keys\",[[70,\"~:block/name\",\"choice checkbox state\",536870913],[71,\"^1\",\"choice classes\",536870913],[72,\"^1\",\"choice exclusions\",536870913],[73,\"^1\",\"properties displayed as checkbox\",536870913],[74,\"^1\",\"status\",536870913],[81,\"^1\",\"priority\",536870913],[86,\"^1\",\"deadline\",536870913],[88,\"^1\",\"scheduled\",536870913],[90,\"^1\",\"repeating recur frequency\",536870913],[92,\"^1\",\"repeating recur unit\",536870913],[99,\"^1\",\"node repeats?\",536870913],[100,\"^1\",\"repeating type\",536870913],[104,\"^1\",\"repeating temporal property\",536870913],[105,\"^1\",\"repeating checked property\",536870913],[106,\"^1\",\"assignee\",536870913],[107,\"^1\",\"icon\",536870913]]]", + "addresses": null + }, + { + "addr": 1000165, + "content": "[\"^ \",\"~:keys\",[[108,\"~:block/name\",\"publishing public?\",536870913],[109,\"^1\",\"published url\",536870913],[110,\"^1\",\"excluded from graph view?\",536870913],[111,\"^1\",\"view type\",536870913],[115,\"^1\",\"view feature type\",536870913],[116,\"^1\",\"view group by property\",536870913],[117,\"^1\",\"gallery asset property\",536870913],[118,\"^1\",\"gallery display properties\",536870913],[119,\"^1\",\"gallery card size\",536870913],[120,\"^1\",\"gallery card width\",536870913],[121,\"^1\",\"gallery card height\",536870913],[122,\"^1\",\"view sort groups by\",536870913],[123,\"^1\",\"view sort groups desc\",536870913],[124,\"^1\",\"view sorting\",536870913],[125,\"^1\",\"view filters\",536870913],[126,\"^1\",\"view hidden columns\",536870913]]]", + "addresses": null + }, + { + "addr": 1000166, + "content": "[\"^ \",\"~:keys\",[[127,\"~:block/name\",\"view ordered columns\",536870913],[128,\"^1\",\"view columns settings\",536870913],[129,\"^1\",\"table view pinned columns\",536870913],[130,\"^1\",\"this view belongs to\",536870913],[131,\"^1\",\"file type\",536870913],[132,\"^1\",\"external url\",536870913],[133,\"^1\",\"external file name\",536870913],[134,\"^1\",\"file size\",536870913],[135,\"^1\",\"image width\",536870913],[136,\"^1\",\"image height\",536870913],[137,\"^1\",\"file checksum\",536870913],[138,\"^1\",\"last visit page\",536870913],[139,\"^1\",\"file remote metadata\",536870913],[141,\"^1\",\"asset resize metadata\",536870913],[142,\"^1\",\"asset alignment\",536870913],[143,\"^1\",\"due\",536870913]]]", + "addresses": null + }, + { + "addr": 1000167, + "content": "[\"^ \",\"~:keys\",[[144,\"~:block/name\",\"state\",536870913],[145,\"^1\",\"user name\",536870913],[146,\"^1\",\"user email\",536870913],[147,\"^1\",\"user avatar\",536870913],[148,\"^1\",\"enable property history\",536870913],[150,\"^1\",\"history block\",536870913],[151,\"^1\",\"history property\",536870913],[152,\"^1\",\"history value\",536870913],[153,\"^1\",\"history scalar value\",536870913],[154,\"^1\",\"node created by\",536870913],[155,\"^1\",\"deleted at\",536870913],[156,\"^1\",\"deleted by\",536870913],[157,\"^1\",\"recycle original parent\",536870913],[158,\"^1\",\"recycle original page\",536870913],[159,\"^1\",\"recycle original order\",536870913],[160,\"^1\",\"reaction emoji\",536870913]]]", + "addresses": null + }, + { + "addr": 1000168, + "content": "[\"^ \",\"~:keys\",[[161,\"~:block/name\",\"reaction target\",536870913],[162,\"^1\",\"agent session id\",536870913],[164,\"^1\",\"used template\",536870913],[165,\"^1\",\"apply template to tags\",536870913],[166,\"^1\",\"reference to large block title stored in remote object storage\",536870913],[167,\"^1\",\"journal\",536870913],[168,\"^1\",\"whiteboard\",536870913],[169,\"^1\",\"task\",536870913],[170,\"^1\",\"comments\",536870913],[171,\"^1\",\"comment\",536870913],[172,\"^1\",\"query\",536870913],[173,\"^1\",\"card\",536870913],[174,\"^1\",\"cards\",536870913],[175,\"^1\",\"asset\",536870913],[176,\"^1\",\"code\",536870913],[177,\"^1\",\"quote\",536870913],[178,\"^1\",\"math\",536870913],[179,\"^1\",\"pdf annotation\",536870913],[185,\"^1\",\"library\",536870913],[186,\"^1\",\"contents\",536870913],[187,\"^1\",\"quick add\",536870913],[188,\"^1\",\"$$$views\",536870913],[189,\"^1\",\"$$$favorites\",536870913],[190,\"^1\",\"recycle\",536870913],[12,\"~:block/order\",\"a0\",536870913],[13,\"^2\",\"a1\",536870913],[14,\"^2\",\"a2\",536870913]]]", + "addresses": null + }, + { + "addr": 1000169, + "content": "[\"^ \",\"~:keys\",[[15,\"~:block/order\",\"a3\",536870913],[16,\"^1\",\"a4\",536870913],[17,\"^1\",\"a5\",536870913],[18,\"^1\",\"a6\",536870913],[19,\"^1\",\"a7\",536870913],[20,\"^1\",\"a8\",536870913],[21,\"^1\",\"a9\",536870913],[22,\"^1\",\"aA\",536870913],[23,\"^1\",\"aB\",536870913],[24,\"^1\",\"aC\",536870913],[25,\"^1\",\"aD\",536870913],[26,\"^1\",\"aE\",536870913],[27,\"^1\",\"aF\",536870913],[28,\"^1\",\"aG\",536870913],[29,\"^1\",\"aH\",536870913],[30,\"^1\",\"aI\",536870913]]]", + "addresses": null + }, + { + "addr": 1000170, + "content": "[\"^ \",\"~:keys\",[[31,\"~:block/order\",\"aJ\",536870913],[32,\"^1\",\"aK\",536870913],[33,\"^1\",\"aL\",536870913],[34,\"^1\",\"aM\",536870913],[35,\"^1\",\"aN\",536870913],[36,\"^1\",\"aO\",536870913],[37,\"^1\",\"aP\",536870913],[38,\"^1\",\"aQ\",536870913],[39,\"^1\",\"aR\",536870913],[40,\"^1\",\"aS\",536870913],[41,\"^1\",\"aT\",536870913],[42,\"^1\",\"aU\",536870913],[43,\"^1\",\"aV\",536870913],[44,\"^1\",\"aW\",536870913],[45,\"^1\",\"aX\",536870913],[46,\"^1\",\"aY\",536870913]]]", + "addresses": null + }, + { + "addr": 1000171, + "content": "[\"^ \",\"~:keys\",[[47,\"~:block/order\",\"aZ\",536870913],[48,\"^1\",\"aa\",536870913],[49,\"^1\",\"ab\",536870913],[50,\"^1\",\"ac\",536870913],[51,\"^1\",\"ad\",536870913],[52,\"^1\",\"ae\",536870913],[53,\"^1\",\"af\",536870913],[54,\"^1\",\"ag\",536870913],[55,\"^1\",\"ah\",536870913],[56,\"^1\",\"ai\",536870913],[57,\"^1\",\"aj\",536870913],[58,\"^1\",\"ak\",536870913],[59,\"^1\",\"al\",536870913],[60,\"^1\",\"am\",536870913],[61,\"^1\",\"an\",536870913],[62,\"^1\",\"ao\",536870913]]]", + "addresses": null + }, + { + "addr": 1000172, + "content": "[\"^ \",\"~:keys\",[[63,\"~:block/order\",\"ap\",536870913],[64,\"^1\",\"aq\",536870913],[65,\"^1\",\"ar\",536870913],[66,\"^1\",\"as\",536870913],[67,\"^1\",\"at\",536870913],[68,\"^1\",\"au\",536870913],[69,\"^1\",\"av\",536870913],[70,\"^1\",\"aw\",536870913],[71,\"^1\",\"ax\",536870913],[72,\"^1\",\"ay\",536870913],[73,\"^1\",\"az\",536870913],[74,\"^1\",\"b00\",536870913],[75,\"^1\",\"b01\",536870913],[76,\"^1\",\"b02\",536870913],[77,\"^1\",\"b03\",536870913],[78,\"^1\",\"b04\",536870913]]]", + "addresses": null + }, + { + "addr": 1000173, + "content": "[\"^ \",\"~:keys\",[[79,\"~:block/order\",\"b05\",536870913],[80,\"^1\",\"b06\",536870913],[81,\"^1\",\"b07\",536870913],[82,\"^1\",\"b08\",536870913],[83,\"^1\",\"b09\",536870913],[84,\"^1\",\"b0A\",536870913],[85,\"^1\",\"b0B\",536870913],[86,\"^1\",\"b0C\",536870913],[87,\"^1\",\"b0D\",536870913],[88,\"^1\",\"b0E\",536870913],[89,\"^1\",\"b0F\",536870913],[90,\"^1\",\"b0G\",536870913],[91,\"^1\",\"b0H\",536870913],[92,\"^1\",\"b0I\",536870913],[93,\"^1\",\"b0J\",536870913],[94,\"^1\",\"b0K\",536870913]]]", + "addresses": null + }, + { + "addr": 1000174, + "content": "[\"^ \",\"~:keys\",[[95,\"~:block/order\",\"b0L\",536870913],[96,\"^1\",\"b0M\",536870913],[97,\"^1\",\"b0N\",536870913],[98,\"^1\",\"b0O\",536870913],[99,\"^1\",\"b0P\",536870913],[100,\"^1\",\"b0Q\",536870913],[101,\"^1\",\"b0R\",536870913],[102,\"^1\",\"b0S\",536870913],[103,\"^1\",\"b0T\",536870913],[104,\"^1\",\"b0U\",536870913],[105,\"^1\",\"b0V\",536870913],[106,\"^1\",\"b0W\",536870913],[107,\"^1\",\"b0X\",536870913],[108,\"^1\",\"b0Y\",536870913],[109,\"^1\",\"b0Z\",536870913],[110,\"^1\",\"b0a\",536870913]]]", + "addresses": null + }, + { + "addr": 1000175, + "content": "[\"^ \",\"~:keys\",[[111,\"~:block/order\",\"b0b\",536870913],[112,\"^1\",\"b0c\",536870913],[113,\"^1\",\"b0d\",536870913],[114,\"^1\",\"b0e\",536870913],[115,\"^1\",\"b0f\",536870913],[116,\"^1\",\"b0g\",536870913],[117,\"^1\",\"b0h\",536870913],[118,\"^1\",\"b0i\",536870913],[119,\"^1\",\"b0j\",536870913],[120,\"^1\",\"b0k\",536870913],[121,\"^1\",\"b0l\",536870913],[122,\"^1\",\"b0m\",536870913],[123,\"^1\",\"b0n\",536870913],[124,\"^1\",\"b0o\",536870913],[125,\"^1\",\"b0p\",536870913],[126,\"^1\",\"b0q\",536870913]]]", + "addresses": null + }, + { + "addr": 1000176, + "content": "[\"^ \",\"~:keys\",[[127,\"~:block/order\",\"b0r\",536870913],[128,\"^1\",\"b0s\",536870913],[129,\"^1\",\"b0t\",536870913],[130,\"^1\",\"b0u\",536870913],[131,\"^1\",\"b0v\",536870913],[132,\"^1\",\"b0w\",536870913],[133,\"^1\",\"b0x\",536870913],[134,\"^1\",\"b0y\",536870913],[135,\"^1\",\"b0z\",536870913],[136,\"^1\",\"b10\",536870913],[137,\"^1\",\"b11\",536870913],[138,\"^1\",\"b12\",536870913],[139,\"^1\",\"b13\",536870913],[140,\"^1\",\"b14\",536870913],[141,\"^1\",\"b15\",536870913],[142,\"^1\",\"b16\",536870913]]]", + "addresses": null + }, + { + "addr": 1000177, + "content": "[\"^ \",\"~:keys\",[[143,\"~:block/order\",\"b17\",536870913],[144,\"^1\",\"b18\",536870913],[145,\"^1\",\"b19\",536870913],[146,\"^1\",\"b1A\",536870913],[147,\"^1\",\"b1B\",536870913],[148,\"^1\",\"b1C\",536870913],[149,\"^1\",\"b1D\",536870913],[150,\"^1\",\"b1E\",536870913],[151,\"^1\",\"b1F\",536870913],[152,\"^1\",\"b1G\",536870913],[153,\"^1\",\"b1H\",536870913],[154,\"^1\",\"b1I\",536870913],[155,\"^1\",\"b1J\",536870913],[156,\"^1\",\"b1K\",536870913],[157,\"^1\",\"b1L\",536870913],[158,\"^1\",\"b1M\",536870913],[159,\"^1\",\"b1N\",536870913],[160,\"^1\",\"b1O\",536870913],[161,\"^1\",\"b1P\",536870913],[162,\"^1\",\"b1Q\",536870913],[163,\"^1\",\"b1R\",536870913],[164,\"^1\",\"b1S\",536870913],[165,\"^1\",\"b1T\",536870913],[166,\"^1\",\"b1U\",536870913],[40,\"~:block/page\",39,536870913],[44,\"^2\",43,536870913],[46,\"^2\",45,536870913],[50,\"^2\",49,536870913]]]", + "addresses": null + }, + { + "addr": 1000178, + "content": "[\"^ \",\"~:keys\",[[57,\"~:block/page\",56,536870913],[58,\"^1\",56,536870913],[59,\"^1\",56,536870913],[60,\"^1\",56,536870913],[61,\"^1\",56,536870913],[75,\"^1\",74,536870913],[76,\"^1\",74,536870913],[77,\"^1\",74,536870913],[78,\"^1\",74,536870913],[79,\"^1\",74,536870913],[80,\"^1\",74,536870913],[82,\"^1\",81,536870913],[83,\"^1\",81,536870913],[84,\"^1\",81,536870913],[85,\"^1\",81,536870913],[87,\"^1\",86,536870913]]]", + "addresses": null + }, + { + "addr": 1000179, + "content": "[\"^ \",\"~:keys\",[[94,\"~:block/closed-value-property\",92,536870913],[27,\"~:block/created-at\",1786627375716,536870913],[43,\"^2\",1786627375717,536870913],[59,\"^2\",1786627375718,536870913],[75,\"^2\",1786627375719,536870913],[91,\"^2\",1786627375720,536870913],[107,\"^2\",1786627375721,536870913],[123,\"^2\",1786627375722,536870913],[139,\"^2\",1786627375722,536870913],[155,\"^2\",1786627375723,536870913],[171,\"^2\",1786627375725,536870913],[12,\"~:block/name\",\"property type\",536870913],[28,\"^3\",\"node links to\",536870913],[47,\"^3\",\"hide from node\",536870913],[69,\"^3\",\"title format\",536870913],[107,\"^3\",\"icon\",536870913],[126,\"^3\",\"view hidden columns\",536870913],[143,\"^3\",\"due\",536870913],[160,\"^3\",\"reaction emoji\",536870913],[14,\"~:block/order\",\"a2\",536870913],[30,\"^4\",\"aI\",536870913],[46,\"^4\",\"aY\",536870913],[62,\"^4\",\"ao\",536870913],[78,\"^4\",\"b04\",536870913],[94,\"^4\",\"b0K\",536870913],[110,\"^4\",\"b0a\",536870913],[126,\"^4\",\"b0q\",536870913],[142,\"^4\",\"b16\",536870913],[50,\"~:block/page\",49,536870913],[87,\"^5\",86,536870913]]]", + "addresses": "[1000002,1000150,1000151,1000152,1000153,1000154,1000155,1000156,1000157,1000158,1000159,1000160,1000161,1000162,1000163,1000164,1000165,1000166,1000167,1000168,1000169,1000170,1000171,1000172,1000173,1000174,1000175,1000176,1000177,1000178]" + }, + { + "addr": 1000180, + "content": "[\"^ \",\"~:keys\",[[89,\"~:block/page\",88,536870913],[91,\"^1\",90,536870913],[93,\"^1\",92,536870913],[94,\"^1\",92,536870913],[95,\"^1\",92,536870913],[96,\"^1\",92,536870913],[97,\"^1\",92,536870913],[98,\"^1\",92,536870913],[101,\"^1\",100,536870913],[102,\"^1\",100,536870913],[103,\"^1\",100,536870913],[112,\"^1\",111,536870913],[113,\"^1\",111,536870913],[114,\"^1\",111,536870913],[140,\"^1\",139,536870913],[149,\"^1\",148,536870913],[163,\"^1\",162,536870913],[40,\"~:block/parent\",39,536870913],[44,\"^2\",43,536870913],[46,\"^2\",45,536870913],[50,\"^2\",49,536870913],[57,\"^2\",56,536870913],[58,\"^2\",56,536870913]]]", + "addresses": null + }, + { + "addr": 1000181, + "content": "[\"^ \",\"~:keys\",[[59,\"~:block/parent\",56,536870913],[60,\"^1\",56,536870913],[61,\"^1\",56,536870913],[75,\"^1\",74,536870913],[76,\"^1\",74,536870913],[77,\"^1\",74,536870913],[78,\"^1\",74,536870913],[79,\"^1\",74,536870913],[80,\"^1\",74,536870913],[82,\"^1\",81,536870913],[83,\"^1\",81,536870913],[84,\"^1\",81,536870913],[85,\"^1\",81,536870913],[87,\"^1\",86,536870913],[89,\"^1\",88,536870913],[91,\"^1\",90,536870913]]]", + "addresses": null + }, + { + "addr": 1000182, + "content": "[\"^ \",\"~:keys\",[[93,\"~:block/parent\",92,536870913],[94,\"^1\",92,536870913],[95,\"^1\",92,536870913],[96,\"^1\",92,536870913],[97,\"^1\",92,536870913],[98,\"^1\",92,536870913],[101,\"^1\",100,536870913],[102,\"^1\",100,536870913],[103,\"^1\",100,536870913],[112,\"^1\",111,536870913],[113,\"^1\",111,536870913],[114,\"^1\",111,536870913],[140,\"^1\",139,536870913],[149,\"^1\",148,536870913],[163,\"^1\",162,536870913],[1,\"~:block/tags\",2,536870913],[2,\"^2\",2,536870913],[3,\"^2\",2,536870913],[4,\"^2\",2,536870913],[5,\"^2\",2,536870913],[12,\"^2\",3,536870913],[13,\"^2\",3,536870913],[14,\"^2\",3,536870913],[15,\"^2\",3,536870913],[16,\"^2\",3,536870913],[17,\"^2\",3,536870913],[18,\"^2\",3,536870913],[19,\"^2\",3,536870913]]]", + "addresses": null + }, + { + "addr": 1000183, + "content": "[\"^ \",\"~:keys\",[[20,\"~:block/tags\",3,536870913],[21,\"^1\",3,536870913],[22,\"^1\",3,536870913],[23,\"^1\",3,536870913],[24,\"^1\",3,536870913],[25,\"^1\",3,536870913],[26,\"^1\",3,536870913],[27,\"^1\",3,536870913],[28,\"^1\",3,536870913],[29,\"^1\",3,536870913],[30,\"^1\",3,536870913],[31,\"^1\",3,536870913],[32,\"^1\",3,536870913],[33,\"^1\",3,536870913],[34,\"^1\",3,536870913],[35,\"^1\",3,536870913]]]", + "addresses": null + }, + { + "addr": 1000184, + "content": "[\"^ \",\"~:keys\",[[36,\"~:block/tags\",3,536870913],[37,\"^1\",3,536870913],[38,\"^1\",3,536870913],[39,\"^1\",3,536870913],[41,\"^1\",3,536870913],[42,\"^1\",3,536870913],[43,\"^1\",3,536870913],[45,\"^1\",3,536870913],[47,\"^1\",3,536870913],[48,\"^1\",3,536870913],[49,\"^1\",3,536870913],[51,\"^1\",3,536870913],[52,\"^1\",3,536870913],[53,\"^1\",3,536870913],[54,\"^1\",3,536870913],[55,\"^1\",3,536870913]]]", + "addresses": null + }, + { + "addr": 1000185, + "content": "[\"^ \",\"~:keys\",[[56,\"~:block/tags\",3,536870913],[62,\"^1\",3,536870913],[63,\"^1\",3,536870913],[64,\"^1\",3,536870913],[65,\"^1\",3,536870913],[66,\"^1\",3,536870913],[67,\"^1\",3,536870913],[68,\"^1\",3,536870913],[69,\"^1\",3,536870913],[70,\"^1\",3,536870913],[71,\"^1\",3,536870913],[72,\"^1\",3,536870913],[73,\"^1\",3,536870913],[74,\"^1\",3,536870913],[81,\"^1\",3,536870913],[86,\"^1\",3,536870913]]]", + "addresses": null + }, + { + "addr": 1000186, + "content": "[\"^ \",\"~:keys\",[[88,\"~:block/tags\",3,536870913],[90,\"^1\",3,536870913],[92,\"^1\",3,536870913],[99,\"^1\",3,536870913],[100,\"^1\",3,536870913],[104,\"^1\",3,536870913],[105,\"^1\",3,536870913],[106,\"^1\",3,536870913],[107,\"^1\",3,536870913],[108,\"^1\",3,536870913],[109,\"^1\",3,536870913],[110,\"^1\",3,536870913],[111,\"^1\",3,536870913],[115,\"^1\",3,536870913],[116,\"^1\",3,536870913],[117,\"^1\",3,536870913]]]", + "addresses": null + }, + { + "addr": 1000187, + "content": "[\"^ \",\"~:keys\",[[118,\"~:block/tags\",3,536870913],[119,\"^1\",3,536870913],[120,\"^1\",3,536870913],[121,\"^1\",3,536870913],[122,\"^1\",3,536870913],[123,\"^1\",3,536870913],[124,\"^1\",3,536870913],[125,\"^1\",3,536870913],[126,\"^1\",3,536870913],[127,\"^1\",3,536870913],[128,\"^1\",3,536870913],[129,\"^1\",3,536870913],[130,\"^1\",3,536870913],[131,\"^1\",3,536870913],[132,\"^1\",3,536870913],[133,\"^1\",3,536870913]]]", + "addresses": null + }, + { + "addr": 1000188, + "content": "[\"^ \",\"~:keys\",[[134,\"~:block/tags\",3,536870913],[135,\"^1\",3,536870913],[136,\"^1\",3,536870913],[137,\"^1\",3,536870913],[138,\"^1\",3,536870913],[139,\"^1\",3,536870913],[141,\"^1\",3,536870913],[142,\"^1\",3,536870913],[143,\"^1\",3,536870913],[144,\"^1\",3,536870913],[145,\"^1\",3,536870913],[146,\"^1\",3,536870913],[147,\"^1\",3,536870913],[148,\"^1\",3,536870913],[150,\"^1\",3,536870913],[151,\"^1\",3,536870913]]]", + "addresses": null + }, + { + "addr": 1000189, + "content": "[\"^ \",\"~:keys\",[[152,\"~:block/tags\",3,536870913],[153,\"^1\",3,536870913],[154,\"^1\",3,536870913],[155,\"^1\",3,536870913],[156,\"^1\",3,536870913],[157,\"^1\",3,536870913],[158,\"^1\",3,536870913],[159,\"^1\",3,536870913],[160,\"^1\",3,536870913],[161,\"^1\",3,536870913],[162,\"^1\",3,536870913],[164,\"^1\",3,536870913],[165,\"^1\",3,536870913],[166,\"^1\",3,536870913],[167,\"^1\",2,536870913],[168,\"^1\",2,536870913]]]", + "addresses": null + }, + { + "addr": 1000190, + "content": "[\"^ \",\"~:keys\",[[169,\"~:block/tags\",2,536870913],[170,\"^1\",2,536870913],[171,\"^1\",2,536870913],[172,\"^1\",2,536870913],[173,\"^1\",2,536870913],[174,\"^1\",2,536870913],[175,\"^1\",2,536870913],[176,\"^1\",2,536870913],[177,\"^1\",2,536870913],[178,\"^1\",2,536870913],[179,\"^1\",2,536870913],[185,\"^1\",4,536870913],[186,\"^1\",4,536870913],[187,\"^1\",4,536870913],[188,\"^1\",4,536870913],[189,\"^1\",4,536870913],[190,\"^1\",4,536870913],[1,\"~:block/title\",\"Root Tag\",536870913],[2,\"^2\",\"Tag\",536870913],[3,\"^2\",\"Property\",536870913],[4,\"^2\",\"Page\",536870913],[5,\"^2\",\"Template\",536870913],[12,\"^2\",\"Property type\",536870913],[13,\"^2\",\"Built in?\",536870913],[14,\"^2\",\"Hide this property or page\",536870913],[15,\"^2\",\"Created from property\",536870913]]]", + "addresses": null + }, + { + "addr": 1000191, + "content": "[\"^ \",\"~:keys\",[[16,\"~:block/title\",\"Property public?\",536870913],[17,\"^1\",\"Property view context\",536870913],[18,\"^1\",\"Property position\",536870913],[19,\"^1\",\"Property classes\",536870913],[20,\"^1\",\"Property value\",536870913],[21,\"^1\",\"Alias\",536870913],[22,\"^1\",\"Tags\",536870913],[23,\"^1\",\"Node parent\",536870913],[24,\"^1\",\"Node order\",536870913],[25,\"^1\",\"Node collapsed?\",536870913],[26,\"^1\",\"Node page\",536870913],[27,\"^1\",\"Node references\",536870913],[28,\"^1\",\"Node links to\",536870913],[29,\"^1\",\"Node title\",536870913],[30,\"^1\",\"Closed value property\",536870913],[31,\"^1\",\"Journal date\",536870913]]]", + "addresses": null + }, + { + "addr": 1000192, + "content": "[\"^ \",\"~:keys\",[[32,\"~:block/title\",\"Node created at\",536870913],[33,\"^1\",\"Node updated at\",536870913],[34,\"^1\",\"Node Display Type\",536870913],[35,\"^1\",\"Description\",536870913],[36,\"^1\",\"Code Mode\",536870913],[37,\"^1\",\"Default value\",536870913],[38,\"^1\",\"Non ref type default value\",536870913],[39,\"^1\",\"Extends\",536870913],[40,\"^1\",\"This enables tags to inherit properties from other tags\",536870913],[41,\"^1\",\"Tag Properties\",536870913],[42,\"^1\",\"Bidirectional property title\",536870913],[43,\"^1\",\"Enable bidirectional properties\",536870913],[44,\"^1\",\"When enabled, this tag will show reverse nodes that link to the current node via properties.\",536870913],[45,\"^1\",\"Hide empty value\",536870913],[46,\"^1\",\"Hides a property's value on any node when empty e.g. when a property appears on a node through a tag.\",536870913],[47,\"^1\",\"Hide from Node\",536870913]]]", + "addresses": null + }, + { + "addr": 1000193, + "content": "[\"^ \",\"~:keys\",[[48,\"~:block/title\",\"Query\",536870913],[49,\"^1\",\"Page Tags\",536870913],[50,\"^1\",\"Provides a way for a page to associate to another page i.e. backward compatible tagging.\",536870913],[51,\"^1\",\"Background color\",536870913],[52,\"^1\",\"Heading\",536870913],[53,\"^1\",\"Asset\",536870913],[54,\"^1\",\"ls-type\",536870913],[55,\"^1\",\"Annotation type\",536870913],[56,\"^1\",\"Annotation color\",536870913],[57,\"^1\",\"yellow\",536870913],[58,\"^1\",\"red\",536870913],[59,\"^1\",\"green\",536870913],[60,\"^1\",\"blue\",536870913],[61,\"^1\",\"purple\",536870913],[62,\"^1\",\"Annotation page\",536870913],[63,\"^1\",\"Annotation image\",536870913]]]", + "addresses": null + }, + { + "addr": 1000194, + "content": "[\"^ \",\"~:keys\",[[64,\"~:block/title\",\"Annotation data\",536870913],[65,\"^1\",\"List type\",536870913],[66,\"^1\",\"Included references\",536870913],[67,\"^1\",\"Excluded references\",536870913],[68,\"^1\",\"Commented blocks\",536870913],[69,\"^1\",\"Title Format\",536870913],[70,\"^1\",\"Choice checkbox state\",536870913],[71,\"^1\",\"Choice classes\",536870913],[72,\"^1\",\"Choice exclusions\",536870913],[73,\"^1\",\"Properties displayed as checkbox\",536870913],[74,\"^1\",\"Status\",536870913],[75,\"^1\",\"Backlog\",536870913],[76,\"^1\",\"Todo\",536870913],[77,\"^1\",\"Doing\",536870913],[78,\"^1\",\"In Review\",536870913],[79,\"^1\",\"Done\",536870913]]]", + "addresses": null + }, + { + "addr": 1000195, + "content": "[\"^ \",\"~:keys\",[[80,\"~:block/title\",\"Canceled\",536870913],[81,\"^1\",\"Priority\",536870913],[82,\"^1\",\"Low\",536870913],[83,\"^1\",\"Medium\",536870913],[84,\"^1\",\"High\",536870913],[85,\"^1\",\"Urgent\",536870913],[86,\"^1\",\"Deadline\",536870913],[87,\"^1\",\"Use it to finish something at a specific date(time).\",536870913],[88,\"^1\",\"Scheduled\",536870913],[89,\"^1\",\"Use it to plan something to start at a specific date(time).\",536870913],[90,\"^1\",\"Repeating recur frequency\",536870913],[92,\"^1\",\"Repeating recur unit\",536870913],[93,\"^1\",\"Minute\",536870913],[94,\"^1\",\"Hour\",536870913],[95,\"^1\",\"Day\",536870913],[96,\"^1\",\"Week\",536870913]]]", + "addresses": null + }, + { + "addr": 1000196, + "content": "[\"^ \",\"~:keys\",[[97,\"~:block/title\",\"Month\",536870913],[98,\"^1\",\"Year\",536870913],[99,\"^1\",\"Node Repeats?\",536870913],[100,\"^1\",\"Repeating type\",536870913],[101,\"^1\",\"Advance from completion\",536870913],[102,\"^1\",\"Advance from scheduled\",536870913],[103,\"^1\",\"Advance from scheduled, skip to future\",536870913],[104,\"^1\",\"Repeating Temporal Property\",536870913],[105,\"^1\",\"Repeating Checked Property\",536870913],[106,\"^1\",\"Assignee\",536870913],[107,\"^1\",\"Icon\",536870913],[108,\"^1\",\"Publishing Public?\",536870913],[109,\"^1\",\"Published URL\",536870913],[110,\"^1\",\"Excluded from Graph view?\",536870913],[111,\"^1\",\"View Type\",536870913],[112,\"^1\",\"Table View\",536870913]]]", + "addresses": null + }, + { + "addr": 1000197, + "content": "[\"^ \",\"~:keys\",[[113,\"~:block/title\",\"List View\",536870913],[114,\"^1\",\"Gallery View\",536870913],[115,\"^1\",\"View Feature Type\",536870913],[116,\"^1\",\"View group by property\",536870913],[117,\"^1\",\"Gallery asset property\",536870913],[118,\"^1\",\"Gallery display properties\",536870913],[119,\"^1\",\"Gallery card size\",536870913],[120,\"^1\",\"Gallery card width\",536870913],[121,\"^1\",\"Gallery card height\",536870913],[122,\"^1\",\"View sort groups by\",536870913],[123,\"^1\",\"View sort groups DESC\",536870913],[124,\"^1\",\"View sorting\",536870913],[125,\"^1\",\"View filters\",536870913],[126,\"^1\",\"View hidden columns\",536870913],[127,\"^1\",\"View ordered columns\",536870913],[128,\"^1\",\"View columns settings\",536870913]]]", + "addresses": null + }, + { + "addr": 1000198, + "content": "[\"^ \",\"~:keys\",[[129,\"~:block/title\",\"Table view pinned columns\",536870913],[130,\"^1\",\"This view belongs to\",536870913],[131,\"^1\",\"File Type\",536870913],[132,\"^1\",\"External URL\",536870913],[133,\"^1\",\"External file name\",536870913],[134,\"^1\",\"File Size\",536870913],[135,\"^1\",\"Image width\",536870913],[136,\"^1\",\"Image height\",536870913],[137,\"^1\",\"File checksum\",536870913],[138,\"^1\",\"Last visit page\",536870913],[139,\"^1\",\"File remote metadata\",536870913],[140,\"^1\",\"Metadata of asset in remote storage\",536870913],[141,\"^1\",\"Asset resize metadata\",536870913],[142,\"^1\",\"Asset alignment\",536870913],[143,\"^1\",\"Due\",536870913],[144,\"^1\",\"State\",536870913]]]", + "addresses": null + }, + { + "addr": 1000199, + "content": "[\"^ \",\"~:keys\",[[145,\"~:block/title\",\"User Name\",536870913],[146,\"^1\",\"User Email\",536870913],[147,\"^1\",\"User Avatar\",536870913],[148,\"^1\",\"Enable property history\",536870913],[149,\"^1\",\"Records history anytime a property's value changes on a node.\",536870913],[150,\"^1\",\"History block\",536870913],[151,\"^1\",\"History property\",536870913],[152,\"^1\",\"History value\",536870913],[153,\"^1\",\"History scalar value\",536870913],[154,\"^1\",\"Node created by\",536870913],[155,\"^1\",\"Deleted at\",536870913],[156,\"^1\",\"Deleted by\",536870913],[157,\"^1\",\"Recycle original parent\",536870913],[158,\"^1\",\"Recycle original page\",536870913],[159,\"^1\",\"Recycle original order\",536870913],[160,\"^1\",\"Reaction emoji\",536870913]]]", + "addresses": null + }, + { + "addr": 1000200, + "content": "[\"^ \",\"~:keys\",[[161,\"~:block/title\",\"Reaction target\",536870913],[162,\"^1\",\"Agent Session ID\",536870913],[163,\"^1\",\"Stores the AgentBridge session ID for a routed task.\",536870913],[164,\"^1\",\"Used template\",536870913],[165,\"^1\",\"Apply template to tags\",536870913],[166,\"^1\",\"Reference to large block title stored in remote object storage\",536870913],[167,\"^1\",\"Journal\",536870913],[168,\"^1\",\"Whiteboard\",536870913],[169,\"^1\",\"Task\",536870913],[170,\"^1\",\"Comments\",536870913],[171,\"^1\",\"Comment\",536870913],[172,\"^1\",\"Query\",536870913],[173,\"^1\",\"Card\",536870913],[174,\"^1\",\"Cards\",536870913],[175,\"^1\",\"Asset\",536870913],[176,\"^1\",\"Code\",536870913],[177,\"^1\",\"Quote\",536870913],[178,\"^1\",\"Math\",536870913],[179,\"^1\",\"PDF Annotation\",536870913],[185,\"^1\",\"Library\",536870913],[186,\"^1\",\"Contents\",536870913],[187,\"^1\",\"Quick add\",536870913],[188,\"^1\",\"$$$views\",536870913],[189,\"^1\",\"$$$favorites\",536870913],[190,\"^1\",\"Recycle\",536870913],[1,\"~:block/updated-at\",1786627375725,536870913],[2,\"^2\",1786627375725,536870913],[3,\"^2\",1786627375725,536870913],[4,\"^2\",1786627375725,536870913],[5,\"^2\",1786627375725,536870913],[12,\"^2\",1786627375714,536870913]]]", + "addresses": null + }, + { + "addr": 1000201, + "content": "[\"^ \",\"~:keys\",[[13,\"~:block/updated-at\",1786627375714,536870913],[14,\"^1\",1786627375714,536870913],[15,\"^1\",1786627375714,536870913],[16,\"^1\",1786627375715,536870913],[17,\"^1\",1786627375715,536870913],[18,\"^1\",1786627375715,536870913],[19,\"^1\",1786627375715,536870913],[20,\"^1\",1786627375716,536870913],[21,\"^1\",1786627375716,536870913],[22,\"^1\",1786627375716,536870913],[23,\"^1\",1786627375716,536870913],[24,\"^1\",1786627375716,536870913],[25,\"^1\",1786627375716,536870913],[26,\"^1\",1786627375716,536870913],[27,\"^1\",1786627375716,536870913],[28,\"^1\",1786627375716,536870913]]]", + "addresses": null + }, + { + "addr": 1000202, + "content": "[\"^ \",\"~:keys\",[[29,\"~:block/updated-at\",1786627375717,536870913],[30,\"^1\",1786627375717,536870913],[31,\"^1\",1786627375717,536870913],[32,\"^1\",1786627375717,536870913],[33,\"^1\",1786627375717,536870913],[34,\"^1\",1786627375717,536870913],[35,\"^1\",1786627375717,536870913],[36,\"^1\",1786627375717,536870913],[37,\"^1\",1786627375717,536870913],[38,\"^1\",1786627375717,536870913],[39,\"^1\",1786627375717,536870913],[40,\"^1\",1786627375717,536870913],[41,\"^1\",1786627375717,536870913],[42,\"^1\",1786627375717,536870913],[43,\"^1\",1786627375717,536870913],[44,\"^1\",1786627375717,536870913]]]", + "addresses": null + }, + { + "addr": 1000203, + "content": "[\"^ \",\"~:keys\",[[45,\"~:block/updated-at\",1786627375717,536870913],[46,\"^1\",1786627375718,536870913],[47,\"^1\",1786627375718,536870913],[48,\"^1\",1786627375718,536870913],[49,\"^1\",1786627375718,536870913],[50,\"^1\",1786627375718,536870913],[51,\"^1\",1786627375718,536870913],[52,\"^1\",1786627375718,536870913],[53,\"^1\",1786627375718,536870913],[54,\"^1\",1786627375718,536870913],[55,\"^1\",1786627375718,536870913],[56,\"^1\",1786627375718,536870913],[57,\"^1\",1786627375718,536870913],[58,\"^1\",1786627375718,536870913],[59,\"^1\",1786627375718,536870913],[60,\"^1\",1786627375718,536870913]]]", + "addresses": null + }, + { + "addr": 1000204, + "content": "[\"^ \",\"~:keys\",[[61,\"~:block/updated-at\",1786627375718,536870913],[62,\"^1\",1786627375719,536870913],[63,\"^1\",1786627375719,536870913],[64,\"^1\",1786627375719,536870913],[65,\"^1\",1786627375719,536870913],[66,\"^1\",1786627375719,536870913],[67,\"^1\",1786627375719,536870913],[68,\"^1\",1786627375719,536870913],[69,\"^1\",1786627375719,536870913],[70,\"^1\",1786627375719,536870913],[71,\"^1\",1786627375719,536870913],[72,\"^1\",1786627375719,536870913],[73,\"^1\",1786627375719,536870913],[74,\"^1\",1786627375719,536870913],[75,\"^1\",1786627375719,536870913],[76,\"^1\",1786627375719,536870913]]]", + "addresses": null + }, + { + "addr": 1000205, + "content": "[\"^ \",\"~:keys\",[[77,\"~:block/updated-at\",1786627375719,536870913],[78,\"^1\",1786627375719,536870913],[79,\"^1\",1786627375719,536870913],[80,\"^1\",1786627375720,536870913],[81,\"^1\",1786627375720,536870913],[82,\"^1\",1786627375720,536870913],[83,\"^1\",1786627375720,536870913],[84,\"^1\",1786627375720,536870913],[85,\"^1\",1786627375720,536870913],[86,\"^1\",1786627375720,536870913],[87,\"^1\",1786627375720,536870913],[88,\"^1\",1786627375720,536870913],[89,\"^1\",1786627375720,536870913],[90,\"^1\",1786627375720,536870913],[91,\"^1\",1786627375720,536870913],[92,\"^1\",1786627375720,536870913]]]", + "addresses": null + }, + { + "addr": 1000206, + "content": "[\"^ \",\"~:keys\",[[93,\"~:block/updated-at\",1786627375720,536870913],[94,\"^1\",1786627375720,536870913],[95,\"^1\",1786627375721,536870913],[96,\"^1\",1786627375721,536870913],[97,\"^1\",1786627375721,536870913],[98,\"^1\",1786627375721,536870913],[99,\"^1\",1786627375721,536870913],[100,\"^1\",1786627375721,536870913],[101,\"^1\",1786627375721,536870913],[102,\"^1\",1786627375721,536870913],[103,\"^1\",1786627375721,536870913],[104,\"^1\",1786627375721,536870913],[105,\"^1\",1786627375721,536870913],[106,\"^1\",1786627375721,536870913],[107,\"^1\",1786627375721,536870913],[108,\"^1\",1786627375721,536870913]]]", + "addresses": null + }, + { + "addr": 1000207, + "content": "[\"^ \",\"~:keys\",[[109,\"~:block/updated-at\",1786627375721,536870913],[110,\"^1\",1786627375721,536870913],[111,\"^1\",1786627375721,536870913],[112,\"^1\",1786627375721,536870913],[113,\"^1\",1786627375721,536870913],[114,\"^1\",1786627375721,536870913],[115,\"^1\",1786627375721,536870913],[116,\"^1\",1786627375721,536870913],[117,\"^1\",1786627375721,536870913],[118,\"^1\",1786627375722,536870913],[119,\"^1\",1786627375722,536870913],[120,\"^1\",1786627375722,536870913],[121,\"^1\",1786627375722,536870913],[122,\"^1\",1786627375722,536870913],[123,\"^1\",1786627375722,536870913],[124,\"^1\",1786627375722,536870913]]]", + "addresses": null + }, + { + "addr": 1000208, + "content": "[\"^ \",\"~:keys\",[[125,\"~:block/updated-at\",1786627375722,536870913],[126,\"^1\",1786627375722,536870913],[127,\"^1\",1786627375722,536870913],[128,\"^1\",1786627375722,536870913],[129,\"^1\",1786627375722,536870913],[130,\"^1\",1786627375722,536870913],[131,\"^1\",1786627375722,536870913],[132,\"^1\",1786627375722,536870913],[133,\"^1\",1786627375722,536870913],[134,\"^1\",1786627375722,536870913],[135,\"^1\",1786627375722,536870913],[136,\"^1\",1786627375722,536870913],[137,\"^1\",1786627375722,536870913],[138,\"^1\",1786627375722,536870913],[139,\"^1\",1786627375722,536870913],[140,\"^1\",1786627375722,536870913]]]", + "addresses": null + }, + { + "addr": 1000209, + "content": "[\"^ \",\"~:keys\",[[141,\"~:block/updated-at\",1786627375722,536870913],[142,\"^1\",1786627375722,536870913],[143,\"^1\",1786627375723,536870913],[144,\"^1\",1786627375723,536870913],[145,\"^1\",1786627375723,536870913],[146,\"^1\",1786627375723,536870913],[147,\"^1\",1786627375723,536870913],[148,\"^1\",1786627375723,536870913],[149,\"^1\",1786627375723,536870913],[150,\"^1\",1786627375723,536870913],[151,\"^1\",1786627375723,536870913],[152,\"^1\",1786627375723,536870913],[153,\"^1\",1786627375723,536870913],[154,\"^1\",1786627375723,536870913],[155,\"^1\",1786627375723,536870913],[156,\"^1\",1786627375723,536870913]]]", + "addresses": null + }, + { + "addr": 1000210, + "content": "[\"^ \",\"~:keys\",[[157,\"~:block/updated-at\",1786627375723,536870913],[158,\"^1\",1786627375723,536870913],[159,\"^1\",1786627375723,536870913],[160,\"^1\",1786627375723,536870913],[161,\"^1\",1786627375724,536870913],[162,\"^1\",1786627375724,536870913],[163,\"^1\",1786627375724,536870913],[164,\"^1\",1786627375724,536870913],[165,\"^1\",1786627375724,536870913],[166,\"^1\",1786627375724,536870913],[167,\"^1\",1786627375725,536870913],[168,\"^1\",1786627375725,536870913],[169,\"^1\",1786627375725,536870913],[170,\"^1\",1786627375725,536870913],[171,\"^1\",1786627375725,536870913],[172,\"^1\",1786627375725,536870913]]]", + "addresses": null + }, + { + "addr": 1000211, + "content": "[\"^ \",\"~:keys\",[[173,\"~:block/updated-at\",1786627375725,536870913],[174,\"^1\",1786627375725,536870913],[175,\"^1\",1786627375725,536870913],[176,\"^1\",1786627375725,536870913],[177,\"^1\",1786627375725,536870913],[178,\"^1\",1786627375725,536870913],[179,\"^1\",1786627375725,536870913],[185,\"^1\",1786627375725,536870913],[186,\"^1\",1786627375725,536870913],[187,\"^1\",1786627375725,536870913],[188,\"^1\",1786627375725,536870913],[189,\"^1\",1786627375725,536870913],[190,\"^1\",1786627375725,536870913],[1,\"~:block/uuid\",\"~u00000002-2737-8382-7000-000000000000\",536870913],[2,\"^2\",\"~u00000002-5389-0208-3000-000000000000\",536870913],[3,\"^2\",\"~u00000002-1038-7670-4800-000000000000\",536870913],[4,\"^2\",\"~u00000002-1484-3403-2900-000000000000\",536870913],[5,\"^2\",\"~u00000002-1720-8548-4600-000000000000\",536870913],[10,\"^2\",\"~u00000004-1595-0218-3700-000000000000\",536870913],[12,\"^2\",\"~u00000002-8384-2404-0000-000000000000\",536870913],[13,\"^2\",\"~u00000002-1125-9581-6000-000000000000\",536870913],[14,\"^2\",\"~u00000002-6836-5746-0000-000000000000\",536870913]]]", + "addresses": null + }, + { + "addr": 1000212, + "content": "[\"^ \",\"~:keys\",[[58,\"~:block/parent\",56,536870913],[91,\"^1\",90,536870913],[19,\"~:block/tags\",3,536870913],[35,\"^2\",3,536870913],[55,\"^2\",3,536870913],[86,\"^2\",3,536870913],[117,\"^2\",3,536870913],[133,\"^2\",3,536870913],[151,\"^2\",3,536870913],[168,\"^2\",2,536870913],[15,\"~:block/title\",\"Created from property\",536870913],[31,\"^3\",\"Journal date\",536870913],[47,\"^3\",\"Hide from Node\",536870913],[63,\"^3\",\"Annotation image\",536870913],[79,\"^3\",\"Done\",536870913],[96,\"^3\",\"Week\",536870913],[112,\"^3\",\"Table View\",536870913],[128,\"^3\",\"View columns settings\",536870913],[144,\"^3\",\"State\",536870913],[160,\"^3\",\"Reaction emoji\",536870913],[12,\"~:block/updated-at\",1786627375714,536870913],[28,\"^4\",1786627375716,536870913],[44,\"^4\",1786627375717,536870913],[60,\"^4\",1786627375718,536870913],[76,\"^4\",1786627375719,536870913],[92,\"^4\",1786627375720,536870913],[108,\"^4\",1786627375721,536870913],[124,\"^4\",1786627375722,536870913],[140,\"^4\",1786627375722,536870913],[156,\"^4\",1786627375723,536870913],[172,\"^4\",1786627375725,536870913],[14,\"~:block/uuid\",\"~u00000002-6836-5746-0000-000000000000\",536870913]]]", + "addresses": "[1000180,1000181,1000182,1000183,1000184,1000185,1000186,1000187,1000188,1000189,1000190,1000191,1000192,1000193,1000194,1000195,1000196,1000197,1000198,1000199,1000200,1000201,1000202,1000203,1000204,1000205,1000206,1000207,1000208,1000209,1000210,1000211]" + }, + { + "addr": 1000213, + "content": "[\"^ \",\"~:keys\",[[15,\"~:block/uuid\",\"~u00000002-8618-9226-7000-000000000000\",536870913],[16,\"^1\",\"~u00000002-1843-0851-4900-000000000000\",536870913],[17,\"^1\",\"~u00000002-1547-3958-2800-000000000000\",536870913],[18,\"^1\",\"~u00000002-1869-2008-6400-000000000000\",536870913],[19,\"^1\",\"~u00000002-9137-5048-6000-000000000000\",536870913],[20,\"^1\",\"~u00000002-1396-5248-6500-000000000000\",536870913],[21,\"^1\",\"~u00000002-2112-6446-9900-000000000000\",536870913],[22,\"^1\",\"~u00000002-1814-9483-4000-000000000000\",536870913],[23,\"^1\",\"~u00000002-9183-0906-4000-000000000000\",536870913],[24,\"^1\",\"~u00000002-1429-2824-3700-000000000000\",536870913],[25,\"^1\",\"~u00000002-2140-2109-9100-000000000000\",536870913],[26,\"^1\",\"~u00000002-8223-1410-8000-000000000000\",536870913],[27,\"^1\",\"~u00000002-1214-4953-4900-000000000000\",536870913],[28,\"^1\",\"~u00000002-1872-3999-9300-000000000000\",536870913],[29,\"^1\",\"~u00000002-7104-4568-4000-000000000000\",536870913],[30,\"^1\",\"~u00000002-1157-7928-1300-000000000000\",536870913]]]", + "addresses": null + }, + { + "addr": 1000214, + "content": "[\"^ \",\"~:keys\",[[31,\"~:block/uuid\",\"~u00000002-1457-4836-6000-000000000000\",536870913],[32,\"^1\",\"~u00000002-1440-0150-0000-000000000000\",536870913],[33,\"^1\",\"~u00000002-1516-5505-5100-000000000000\",536870913],[34,\"^1\",\"~u00000002-4424-4618-9000-000000000000\",536870913],[35,\"^1\",\"~u00000002-3362-3620-0000-000000000000\",536870913],[36,\"^1\",\"~u00000002-8578-9716-5000-000000000000\",536870913],[37,\"^1\",\"~u00000002-8920-7966-2000-000000000000\",536870913],[38,\"^1\",\"~u00000002-1595-7230-1400-000000000000\",536870913],[39,\"^1\",\"~u00000002-7475-9380-3000-000000000000\",536870913],[40,\"^1\",\"~u00000004-4485-6574-6000-000000000000\",536870913],[41,\"^1\",\"~u00000002-2123-7120-5000-000000000000\",536870913],[42,\"^1\",\"~u00000002-6050-5418-7000-000000000000\",536870913],[43,\"^1\",\"~u00000002-2917-8613-8000-000000000000\",536870913],[44,\"^1\",\"~u00000004-1830-0481-2500-000000000000\",536870913],[45,\"^1\",\"~u00000002-2062-3258-9900-000000000000\",536870913],[46,\"^1\",\"~u00000004-1118-8585-8000-000000000000\",536870913]]]", + "addresses": null + }, + { + "addr": 1000215, + "content": "[\"^ \",\"~:keys\",[[47,\"~:block/uuid\",\"~u00000002-2610-3727-0000-000000000000\",536870913],[48,\"^1\",\"~u00000002-9741-4126-0000-000000000000\",536870913],[49,\"^1\",\"~u00000002-2133-5311-8500-000000000000\",536870913],[50,\"^1\",\"~u00000004-1223-8671-8000-000000000000\",536870913],[51,\"^1\",\"~u00000002-5191-2660-6000-000000000000\",536870913],[52,\"^1\",\"~u00000002-1858-7494-1500-000000000000\",536870913],[53,\"^1\",\"~u00000002-8768-5679-0000-000000000000\",536870913],[54,\"^1\",\"~u00000002-3269-7934-5000-000000000000\",536870913],[55,\"^1\",\"~u00000002-9984-3783-2000-000000000000\",536870913],[56,\"^1\",\"~u00000002-6747-9388-7000-000000000000\",536870913],[57,\"^1\",\"~u00000002-1752-1030-0200-000000000000\",536870913],[58,\"^1\",\"~u00000002-4136-2216-2000-000000000000\",536870913],[59,\"^1\",\"~u00000002-1992-2016-2600-000000000000\",536870913],[60,\"^1\",\"~u00000002-1836-0512-4100-000000000000\",536870913],[61,\"^1\",\"~u00000002-1104-3848-5600-000000000000\",536870913],[62,\"^1\",\"~u00000002-7532-8459-6000-000000000000\",536870913]]]", + "addresses": null + }, + { + "addr": 1000216, + "content": "[\"^ \",\"~:keys\",[[63,\"~:block/uuid\",\"~u00000002-1377-6700-9000-000000000000\",536870913],[64,\"^1\",\"~u00000002-5458-2940-2000-000000000000\",536870913],[65,\"^1\",\"~u00000002-6078-1711-1000-000000000000\",536870913],[66,\"^1\",\"~u00000002-1680-5777-0300-000000000000\",536870913],[67,\"^1\",\"~u00000002-2426-7588-9000-000000000000\",536870913],[68,\"^1\",\"~u00000002-2143-9561-0900-000000000000\",536870913],[69,\"^1\",\"~u00000002-1536-4979-5400-000000000000\",536870913],[70,\"^1\",\"~u00000002-1272-4228-6300-000000000000\",536870913],[71,\"^1\",\"~u00000002-7629-6621-9000-000000000000\",536870913],[72,\"^1\",\"~u00000002-1233-5229-4600-000000000000\",536870913],[73,\"^1\",\"~u00000002-3215-3256-9000-000000000000\",536870913],[74,\"^1\",\"~u00000002-9072-1685-3000-000000000000\",536870913],[75,\"^1\",\"~u00000002-7233-3491-0000-000000000000\",536870913],[76,\"^1\",\"~u00000002-1615-5853-7700-000000000000\",536870913],[77,\"^1\",\"~u00000002-1840-1229-0800-000000000000\",536870913],[78,\"^1\",\"~u00000002-1870-0014-4300-000000000000\",536870913]]]", + "addresses": null + }, + { + "addr": 1000217, + "content": "[\"^ \",\"~:keys\",[[79,\"~:block/uuid\",\"~u00000002-1827-5820-8200-000000000000\",536870913],[80,\"^1\",\"~u00000002-7526-4326-2000-000000000000\",536870913],[81,\"^1\",\"~u00000002-2392-2841-1000-000000000000\",536870913],[82,\"^1\",\"~u00000002-2107-4537-4800-000000000000\",536870913],[83,\"^1\",\"~u00000002-1829-3222-7800-000000000000\",536870913],[84,\"^1\",\"~u00000002-5672-2766-8000-000000000000\",536870913],[85,\"^1\",\"~u00000002-1996-4346-6700-000000000000\",536870913],[86,\"^1\",\"~u00000002-1685-9016-0400-000000000000\",536870913],[87,\"^1\",\"~u00000004-1356-3664-2200-000000000000\",536870913],[88,\"^1\",\"~u00000002-1644-5209-4300-000000000000\",536870913],[89,\"^1\",\"~u00000004-9817-6380-0000-000000000000\",536870913],[90,\"^1\",\"~u00000002-8716-1592-2000-000000000000\",536870913],[91,\"^1\",\"~u00000004-1654-1034-2600-000000000000\",536870913],[92,\"^1\",\"~u00000002-6903-0624-7000-000000000000\",536870913],[93,\"^1\",\"~u00000002-1513-6550-8500-000000000000\",536870913],[94,\"^1\",\"~u00000002-1438-8849-5400-000000000000\",536870913]]]", + "addresses": null + }, + { + "addr": 1000218, + "content": "[\"^ \",\"~:keys\",[[95,\"~:block/uuid\",\"~u00000002-3924-1785-8000-000000000000\",536870913],[96,\"^1\",\"~u00000002-2130-9244-4900-000000000000\",536870913],[97,\"^1\",\"~u00000002-2073-3937-9700-000000000000\",536870913],[98,\"^1\",\"~u00000002-1520-4385-2400-000000000000\",536870913],[99,\"^1\",\"~u00000002-1908-1217-8900-000000000000\",536870913],[100,\"^1\",\"~u00000002-1984-2972-8100-000000000000\",536870913],[101,\"^1\",\"~u00000002-1406-5331-0600-000000000000\",536870913],[102,\"^1\",\"~u00000002-1646-4063-5900-000000000000\",536870913],[103,\"^1\",\"~u00000002-7441-9615-3000-000000000000\",536870913],[104,\"^1\",\"~u00000002-8346-1078-4000-000000000000\",536870913],[105,\"^1\",\"~u00000002-1866-3655-5300-000000000000\",536870913],[106,\"^1\",\"~u00000002-2190-1503-4000-000000000000\",536870913],[107,\"^1\",\"~u00000002-5891-2328-5000-000000000000\",536870913],[108,\"^1\",\"~u00000002-1094-6579-3900-000000000000\",536870913],[109,\"^1\",\"~u00000002-2045-1825-0100-000000000000\",536870913],[110,\"^1\",\"~u00000002-4524-3306-5000-000000000000\",536870913]]]", + "addresses": null + }, + { + "addr": 1000219, + "content": "[\"^ \",\"~:keys\",[[111,\"~:block/uuid\",\"~u00000002-2182-3760-7000-000000000000\",536870913],[112,\"^1\",\"~u00000002-1942-5424-0000-000000000000\",536870913],[113,\"^1\",\"~u00000002-1164-8285-0200-000000000000\",536870913],[114,\"^1\",\"~u00000002-1506-0511-2000-000000000000\",536870913],[115,\"^1\",\"~u00000002-9391-4187-1000-000000000000\",536870913],[116,\"^1\",\"~u00000002-8092-1623-6000-000000000000\",536870913],[117,\"^1\",\"~u00000002-1668-3468-9000-000000000000\",536870913],[118,\"^1\",\"~u00000002-1806-7066-8800-000000000000\",536870913],[119,\"^1\",\"~u00000002-2012-8168-3300-000000000000\",536870913],[120,\"^1\",\"~u00000002-1403-2963-9500-000000000000\",536870913],[121,\"^1\",\"~u00000002-4342-2083-2000-000000000000\",536870913],[122,\"^1\",\"~u00000002-6253-4002-1000-000000000000\",536870913],[123,\"^1\",\"~u00000002-1081-4073-8700-000000000000\",536870913],[124,\"^1\",\"~u00000002-2081-0259-4000-000000000000\",536870913],[125,\"^1\",\"~u00000002-1702-3936-3300-000000000000\",536870913],[126,\"^1\",\"~u00000002-9750-5719-2000-000000000000\",536870913]]]", + "addresses": null + }, + { + "addr": 1000220, + "content": "[\"^ \",\"~:keys\",[[127,\"~:block/uuid\",\"~u00000002-1485-5871-0000-000000000000\",536870913],[128,\"^1\",\"~u00000002-1675-5105-5500-000000000000\",536870913],[129,\"^1\",\"~u00000002-2673-7513-8000-000000000000\",536870913],[130,\"^1\",\"~u00000002-3627-4319-0000-000000000000\",536870913],[131,\"^1\",\"~u00000002-1142-0830-9800-000000000000\",536870913],[132,\"^1\",\"~u00000002-1364-7751-6300-000000000000\",536870913],[133,\"^1\",\"~u00000002-3995-2972-0000-000000000000\",536870913],[134,\"^1\",\"~u00000002-1167-8621-9000-000000000000\",536870913],[135,\"^1\",\"~u00000002-1857-8658-3900-000000000000\",536870913],[136,\"^1\",\"~u00000002-3306-1650-4000-000000000000\",536870913],[137,\"^1\",\"~u00000002-1011-4169-7900-000000000000\",536870913],[138,\"^1\",\"~u00000002-2107-8035-3500-000000000000\",536870913],[139,\"^1\",\"~u00000002-9907-5046-9000-000000000000\",536870913],[140,\"^1\",\"~u00000004-5450-2102-9000-000000000000\",536870913],[141,\"^1\",\"~u00000002-1297-5230-5500-000000000000\",536870913],[142,\"^1\",\"~u00000002-7135-0412-8000-000000000000\",536870913]]]", + "addresses": null + }, + { + "addr": 1000221, + "content": "[\"^ \",\"~:keys\",[[143,\"~:block/uuid\",\"~u00000002-1089-0805-4900-000000000000\",536870913],[144,\"^1\",\"~u00000002-1165-1650-8700-000000000000\",536870913],[145,\"^1\",\"~u00000002-1360-0260-1600-000000000000\",536870913],[146,\"^1\",\"~u00000002-1655-2060-6300-000000000000\",536870913],[147,\"^1\",\"~u00000002-4165-4885-8000-000000000000\",536870913],[148,\"^1\",\"~u00000002-8058-5960-2000-000000000000\",536870913],[149,\"^1\",\"~u00000004-1438-1481-0000-000000000000\",536870913],[150,\"^1\",\"~u00000002-1142-5541-6000-000000000000\",536870913],[151,\"^1\",\"~u00000002-1600-4090-8200-000000000000\",536870913],[152,\"^1\",\"~u00000002-5131-3603-7000-000000000000\",536870913],[153,\"^1\",\"~u00000002-2393-3777-5000-000000000000\",536870913],[154,\"^1\",\"~u00000002-8544-3390-8000-000000000000\",536870913],[155,\"^1\",\"~u00000002-6852-1375-1000-000000000000\",536870913],[156,\"^1\",\"~u00000002-1998-0431-1200-000000000000\",536870913],[157,\"^1\",\"~u00000002-4970-0399-4000-000000000000\",536870913],[158,\"^1\",\"~u00000002-1658-7897-9900-000000000000\",536870913]]]", + "addresses": null + }, + { + "addr": 1000222, + "content": "[\"^ \",\"~:keys\",[[159,\"~:block/uuid\",\"~u00000002-8699-2537-0000-000000000000\",536870913],[160,\"^1\",\"~u00000002-9877-5864-5000-000000000000\",536870913],[161,\"^1\",\"~u00000002-1127-5718-5000-000000000000\",536870913],[162,\"^1\",\"~u00000002-1783-0312-6200-000000000000\",536870913],[163,\"^1\",\"~u00000004-1209-4784-1500-000000000000\",536870913],[164,\"^1\",\"~u00000002-9803-6990-6000-000000000000\",536870913],[165,\"^1\",\"~u00000002-4291-2432-2000-000000000000\",536870913],[166,\"^1\",\"~u00000002-1044-8271-6500-000000000000\",536870913],[167,\"^1\",\"~u00000002-1979-7410-8100-000000000000\",536870913],[168,\"^1\",\"~u00000002-1013-6984-5200-000000000000\",536870913],[169,\"^1\",\"~u00000002-1282-1814-5700-000000000000\",536870913],[170,\"^1\",\"~u00000002-2556-9161-5000-000000000000\",536870913],[171,\"^1\",\"~u00000002-1323-0695-4000-000000000000\",536870913],[172,\"^1\",\"~u00000002-2324-8016-6000-000000000000\",536870913],[173,\"^1\",\"~u00000002-1358-2811-0900-000000000000\",536870913],[174,\"^1\",\"~u00000002-1284-2651-6700-000000000000\",536870913]]]", + "addresses": null + }, + { + "addr": 1000223, + "content": "[\"^ \",\"~:keys\",[[175,\"~:block/uuid\",\"~u00000002-7975-0297-0000-000000000000\",536870913],[176,\"^1\",\"~u00000002-1454-9866-4100-000000000000\",536870913],[177,\"^1\",\"~u00000002-1176-1666-1700-000000000000\",536870913],[178,\"^1\",\"~u00000002-2038-9631-2100-000000000000\",536870913],[179,\"^1\",\"~u00000002-5049-5962-0000-000000000000\",536870913],[180,\"^1\",\"~u00000004-3919-3813-3000-000000000000\",536870913],[181,\"^1\",\"~u00000004-1713-4660-3800-000000000000\",536870913],[182,\"^1\",\"~u00000004-1335-6485-2300-000000000000\",536870913],[183,\"^1\",\"~u00000004-4049-4381-0000-000000000000\",536870913],[184,\"^1\",\"~u00000004-2116-2824-5200-000000000000\",536870913],[185,\"^1\",\"~u00000004-1294-7765-6000-000000000000\",536870913],[186,\"^1\",\"~u00000004-1690-2597-3200-000000000000\",536870913],[187,\"^1\",\"~u00000004-7336-8251-3000-000000000000\",536870913],[188,\"^1\",\"~u00000004-1906-3437-5800-000000000000\",536870913],[189,\"^1\",\"~u00000004-1018-5888-4100-000000000000\",536870913],[190,\"^1\",\"~u00000004-7238-1304-3000-000000000000\",536870913],[12,\"~:db/cardinality\",\"~:db.cardinality/one\",536870913],[13,\"^2\",\"^3\",536870913]]]", + "addresses": null + }, + { + "addr": 1000224, + "content": "[\"^ \",\"~:keys\",[[14,\"~:db/cardinality\",\"~:db.cardinality/one\",536870913],[15,\"^1\",\"^2\",536870913],[16,\"^1\",\"^2\",536870913],[17,\"^1\",\"^2\",536870913],[18,\"^1\",\"^2\",536870913],[19,\"^1\",\"~:db.cardinality/many\",536870913],[20,\"^1\",\"^2\",536870913],[21,\"^1\",\"^3\",536870913],[22,\"^1\",\"^3\",536870913],[23,\"^1\",\"^2\",536870913],[24,\"^1\",\"^2\",536870913],[25,\"^1\",\"^2\",536870913],[26,\"^1\",\"^2\",536870913],[27,\"^1\",\"^3\",536870913],[28,\"^1\",\"^2\",536870913],[29,\"^1\",\"^2\",536870913]]]", + "addresses": null + }, + { + "addr": 1000225, + "content": "[\"^ \",\"~:keys\",[[30,\"~:db/cardinality\",\"~:db.cardinality/one\",536870913],[31,\"^1\",\"^2\",536870913],[32,\"^1\",\"^2\",536870913],[33,\"^1\",\"^2\",536870913],[34,\"^1\",\"^2\",536870913],[35,\"^1\",\"^2\",536870913],[36,\"^1\",\"^2\",536870913],[37,\"^1\",\"^2\",536870913],[38,\"^1\",\"^2\",536870913],[39,\"^1\",\"~:db.cardinality/many\",536870913],[41,\"^1\",\"^3\",536870913],[42,\"^1\",\"^2\",536870913],[43,\"^1\",\"^2\",536870913],[45,\"^1\",\"^2\",536870913],[47,\"^1\",\"^2\",536870913],[48,\"^1\",\"^2\",536870913]]]", + "addresses": null + }, + { + "addr": 1000226, + "content": "[\"^ \",\"~:keys\",[[49,\"~:db/cardinality\",\"~:db.cardinality/many\",536870913],[51,\"^1\",\"~:db.cardinality/one\",536870913],[52,\"^1\",\"^3\",536870913],[53,\"^1\",\"^3\",536870913],[54,\"^1\",\"^3\",536870913],[55,\"^1\",\"^3\",536870913],[56,\"^1\",\"^3\",536870913],[62,\"^1\",\"^3\",536870913],[63,\"^1\",\"^3\",536870913],[64,\"^1\",\"^3\",536870913],[65,\"^1\",\"^3\",536870913],[66,\"^1\",\"^2\",536870913],[67,\"^1\",\"^2\",536870913],[68,\"^1\",\"^2\",536870913],[69,\"^1\",\"^3\",536870913],[70,\"^1\",\"^3\",536870913]]]", + "addresses": null + }, + { + "addr": 1000227, + "content": "[\"^ \",\"~:keys\",[[71,\"~:db/cardinality\",\"~:db.cardinality/many\",536870913],[72,\"^1\",\"^2\",536870913],[73,\"^1\",\"^2\",536870913],[74,\"^1\",\"~:db.cardinality/one\",536870913],[81,\"^1\",\"^3\",536870913],[86,\"^1\",\"^3\",536870913],[88,\"^1\",\"^3\",536870913],[90,\"^1\",\"^3\",536870913],[92,\"^1\",\"^3\",536870913],[99,\"^1\",\"^3\",536870913],[100,\"^1\",\"^3\",536870913],[104,\"^1\",\"^3\",536870913],[105,\"^1\",\"^3\",536870913],[106,\"^1\",\"^2\",536870913],[107,\"^1\",\"^3\",536870913],[108,\"^1\",\"^3\",536870913]]]", + "addresses": null + }, + { + "addr": 1000228, + "content": "[\"^ \",\"~:keys\",[[109,\"~:db/cardinality\",\"~:db.cardinality/one\",536870913],[110,\"^1\",\"^2\",536870913],[111,\"^1\",\"^2\",536870913],[115,\"^1\",\"^2\",536870913],[116,\"^1\",\"^2\",536870913],[117,\"^1\",\"^2\",536870913],[118,\"^1\",\"~:db.cardinality/many\",536870913],[119,\"^1\",\"^2\",536870913],[120,\"^1\",\"^2\",536870913],[121,\"^1\",\"^2\",536870913],[122,\"^1\",\"^2\",536870913],[123,\"^1\",\"^2\",536870913],[124,\"^1\",\"^2\",536870913],[125,\"^1\",\"^2\",536870913],[126,\"^1\",\"^3\",536870913],[127,\"^1\",\"^2\",536870913]]]", + "addresses": null + }, + { + "addr": 1000229, + "content": "[\"^ \",\"~:keys\",[[128,\"~:db/cardinality\",\"~:db.cardinality/one\",536870913],[129,\"^1\",\"~:db.cardinality/many\",536870913],[130,\"^1\",\"^2\",536870913],[131,\"^1\",\"^2\",536870913],[132,\"^1\",\"^2\",536870913],[133,\"^1\",\"^2\",536870913],[134,\"^1\",\"^2\",536870913],[135,\"^1\",\"^2\",536870913],[136,\"^1\",\"^2\",536870913],[137,\"^1\",\"^2\",536870913],[138,\"^1\",\"^2\",536870913],[139,\"^1\",\"^2\",536870913],[141,\"^1\",\"^2\",536870913],[142,\"^1\",\"^2\",536870913],[143,\"^1\",\"^2\",536870913],[144,\"^1\",\"^2\",536870913]]]", + "addresses": null + }, + { + "addr": 1000230, + "content": "[\"^ \",\"~:keys\",[[145,\"~:db/cardinality\",\"~:db.cardinality/one\",536870913],[146,\"^1\",\"^2\",536870913],[147,\"^1\",\"^2\",536870913],[148,\"^1\",\"^2\",536870913],[150,\"^1\",\"^2\",536870913],[151,\"^1\",\"^2\",536870913],[152,\"^1\",\"^2\",536870913],[153,\"^1\",\"^2\",536870913],[154,\"^1\",\"^2\",536870913],[155,\"^1\",\"^2\",536870913],[156,\"^1\",\"^2\",536870913],[157,\"^1\",\"^2\",536870913],[158,\"^1\",\"^2\",536870913],[159,\"^1\",\"^2\",536870913],[160,\"^1\",\"^2\",536870913],[161,\"^1\",\"^2\",536870913],[162,\"^1\",\"^2\",536870913],[164,\"^1\",\"^2\",536870913],[165,\"^1\",\"~:db.cardinality/many\",536870913],[166,\"^1\",\"^2\",536870913],[1,\"~:db/ident\",\"~:logseq.class/Root\",536870913],[2,\"^4\",\"~:logseq.class/Tag\",536870913]]]", + "addresses": null + }, + { + "addr": 1000231, + "content": "[\"^ \",\"~:keys\",[[3,\"~:db/ident\",\"~:logseq.class/Property\",536870913],[4,\"^1\",\"~:logseq.class/Page\",536870913],[5,\"^1\",\"~:logseq.class/Template\",536870913],[6,\"^1\",\"~:logseq.kv/db-type\",536870913],[7,\"^1\",\"~:logseq.kv/schema-version\",536870913],[8,\"^1\",\"~:logseq.kv/graph-initial-schema-version\",536870913],[9,\"^1\",\"~:logseq.kv/graph-created-at\",536870913],[10,\"^1\",\"~:logseq.property/empty-placeholder\",536870913],[11,\"^1\",\"~:logseq.kv/local-graph-uuid\",536870913],[12,\"^1\",\"~:logseq.property/type\",536870913],[13,\"^1\",\"~:logseq.property/built-in?\",536870913],[14,\"^1\",\"~:logseq.property/hide?\",536870913],[15,\"^1\",\"~:logseq.property/created-from-property\",536870913],[16,\"^1\",\"~:logseq.property/public?\",536870913],[17,\"^1\",\"~:logseq.property/view-context\",536870913],[18,\"^1\",\"~:logseq.property/ui-position\",536870913]]]", + "addresses": null + }, + { + "addr": 1000232, + "content": "[\"^ \",\"~:keys\",[[19,\"~:db/ident\",\"~:logseq.property/classes\",536870913],[20,\"^1\",\"~:logseq.property/value\",536870913],[21,\"^1\",\"~:block/alias\",536870913],[22,\"^1\",\"~:block/tags\",536870913],[23,\"^1\",\"~:block/parent\",536870913],[24,\"^1\",\"~:block/order\",536870913],[25,\"^1\",\"~:block/collapsed?\",536870913],[26,\"^1\",\"~:block/page\",536870913],[27,\"^1\",\"~:block/refs\",536870913],[28,\"^1\",\"~:block/link\",536870913],[29,\"^1\",\"~:block/title\",536870913],[30,\"^1\",\"~:block/closed-value-property\",536870913],[31,\"^1\",\"~:block/journal-day\",536870913],[32,\"^1\",\"~:block/created-at\",536870913],[33,\"^1\",\"~:block/updated-at\",536870913],[34,\"^1\",\"~:logseq.property.node/display-type\",536870913]]]", + "addresses": null + }, + { + "addr": 1000233, + "content": "[\"^ \",\"~:keys\",[[35,\"~:db/ident\",\"~:logseq.property/description\",536870913],[36,\"^1\",\"~:logseq.property.code/lang\",536870913],[37,\"^1\",\"~:logseq.property/default-value\",536870913],[38,\"^1\",\"~:logseq.property/scalar-default-value\",536870913],[39,\"^1\",\"~:logseq.property.class/extends\",536870913],[41,\"^1\",\"~:logseq.property.class/properties\",536870913],[42,\"^1\",\"~:logseq.property.class/bidirectional-property-title\",536870913],[43,\"^1\",\"~:logseq.property.class/enable-bidirectional?\",536870913],[45,\"^1\",\"~:logseq.property/hide-empty-value\",536870913],[47,\"^1\",\"~:logseq.property.class/hide-from-node\",536870913],[48,\"^1\",\"~:logseq.property/query\",536870913],[49,\"^1\",\"~:logseq.property/page-tags\",536870913],[51,\"^1\",\"~:logseq.property/background-color\",536870913],[52,\"^1\",\"~:logseq.property/heading\",536870913],[53,\"^1\",\"~:logseq.property/asset\",536870913],[54,\"^1\",\"~:logseq.property/ls-type\",536870913]]]", + "addresses": null + }, + { + "addr": 1000234, + "content": "[\"^ \",\"~:keys\",[[55,\"~:db/ident\",\"~:logseq.property.pdf/hl-type\",536870913],[56,\"^1\",\"~:logseq.property.pdf/hl-color\",536870913],[57,\"^1\",\"~:logseq.property/color.yellow\",536870913],[58,\"^1\",\"~:logseq.property/color.red\",536870913],[59,\"^1\",\"~:logseq.property/color.green\",536870913],[60,\"^1\",\"~:logseq.property/color.blue\",536870913],[61,\"^1\",\"~:logseq.property/color.purple\",536870913],[62,\"^1\",\"~:logseq.property.pdf/hl-page\",536870913],[63,\"^1\",\"~:logseq.property.pdf/hl-image\",536870913],[64,\"^1\",\"~:logseq.property.pdf/hl-value\",536870913],[65,\"^1\",\"~:logseq.property/order-list-type\",536870913],[66,\"^1\",\"~:logseq.property.linked-references/includes\",536870913],[67,\"^1\",\"~:logseq.property.linked-references/excludes\",536870913],[68,\"^1\",\"~:logseq.property.comments/blocks\",536870913],[69,\"^1\",\"~:logseq.property.journal/title-format\",536870913],[70,\"^1\",\"~:logseq.property/choice-checkbox-state\",536870913]]]", + "addresses": null + }, + { + "addr": 1000235, + "content": "[\"^ \",\"~:keys\",[[71,\"~:db/ident\",\"~:logseq.property/choice-classes\",536870913],[72,\"^1\",\"~:logseq.property/choice-exclusions\",536870913],[73,\"^1\",\"~:logseq.property/checkbox-display-properties\",536870913],[74,\"^1\",\"~:logseq.property/status\",536870913],[75,\"^1\",\"~:logseq.property/status.backlog\",536870913],[76,\"^1\",\"~:logseq.property/status.todo\",536870913],[77,\"^1\",\"~:logseq.property/status.doing\",536870913],[78,\"^1\",\"~:logseq.property/status.in-review\",536870913],[79,\"^1\",\"~:logseq.property/status.done\",536870913],[80,\"^1\",\"~:logseq.property/status.canceled\",536870913],[81,\"^1\",\"~:logseq.property/priority\",536870913],[82,\"^1\",\"~:logseq.property/priority.low\",536870913],[83,\"^1\",\"~:logseq.property/priority.medium\",536870913],[84,\"^1\",\"~:logseq.property/priority.high\",536870913],[85,\"^1\",\"~:logseq.property/priority.urgent\",536870913],[86,\"^1\",\"~:logseq.property/deadline\",536870913]]]", + "addresses": null + }, + { + "addr": 1000236, + "content": "[\"^ \",\"~:keys\",[[88,\"~:db/ident\",\"~:logseq.property/scheduled\",536870913],[90,\"^1\",\"~:logseq.property.repeat/recur-frequency\",536870913],[92,\"^1\",\"~:logseq.property.repeat/recur-unit\",536870913],[93,\"^1\",\"~:logseq.property.repeat/recur-unit.minute\",536870913],[94,\"^1\",\"~:logseq.property.repeat/recur-unit.hour\",536870913],[95,\"^1\",\"~:logseq.property.repeat/recur-unit.day\",536870913],[96,\"^1\",\"~:logseq.property.repeat/recur-unit.week\",536870913],[97,\"^1\",\"~:logseq.property.repeat/recur-unit.month\",536870913],[98,\"^1\",\"~:logseq.property.repeat/recur-unit.year\",536870913],[99,\"^1\",\"~:logseq.property.repeat/repeated?\",536870913],[100,\"^1\",\"~:logseq.property.repeat/repeat-type\",536870913],[101,\"^1\",\"~:logseq.property.repeat/repeat-type.dotted-plus\",536870913],[102,\"^1\",\"~:logseq.property.repeat/repeat-type.plus\",536870913],[103,\"^1\",\"~:logseq.property.repeat/repeat-type.double-plus\",536870913],[104,\"^1\",\"~:logseq.property.repeat/temporal-property\",536870913],[105,\"^1\",\"~:logseq.property.repeat/checked-property\",536870913]]]", + "addresses": null + }, + { + "addr": 1000237, + "content": "[\"^ \",\"~:keys\",[[106,\"~:db/ident\",\"~:logseq.property/assignee\",536870913],[107,\"^1\",\"~:logseq.property/icon\",536870913],[108,\"^1\",\"~:logseq.property/publishing-public?\",536870913],[109,\"^1\",\"~:logseq.property.publish/published-url\",536870913],[110,\"^1\",\"~:logseq.property/exclude-from-graph-view\",536870913],[111,\"^1\",\"~:logseq.property.view/type\",536870913],[112,\"^1\",\"~:logseq.property.view/type.table\",536870913],[113,\"^1\",\"~:logseq.property.view/type.list\",536870913],[114,\"^1\",\"~:logseq.property.view/type.gallery\",536870913],[115,\"^1\",\"~:logseq.property.view/feature-type\",536870913],[116,\"^1\",\"~:logseq.property.view/group-by-property\",536870913],[117,\"^1\",\"~:logseq.property.view/gallery-asset-property\",536870913],[118,\"^1\",\"~:logseq.property.view/gallery-display-properties\",536870913],[119,\"^1\",\"~:logseq.property.view/gallery-card-size\",536870913],[120,\"^1\",\"~:logseq.property.view/gallery-card-width\",536870913],[121,\"^1\",\"~:logseq.property.view/gallery-card-height\",536870913]]]", + "addresses": null + }, + { + "addr": 1000238, + "content": "[\"^ \",\"~:keys\",[[122,\"~:db/ident\",\"~:logseq.property.view/sort-groups-by-property\",536870913],[123,\"^1\",\"~:logseq.property.view/sort-groups-desc?\",536870913],[124,\"^1\",\"~:logseq.property.table/sorting\",536870913],[125,\"^1\",\"~:logseq.property.table/filters\",536870913],[126,\"^1\",\"~:logseq.property.table/hidden-columns\",536870913],[127,\"^1\",\"~:logseq.property.table/ordered-columns\",536870913],[128,\"^1\",\"~:logseq.property.table/sized-columns\",536870913],[129,\"^1\",\"~:logseq.property.table/pinned-columns\",536870913],[130,\"^1\",\"~:logseq.property/view-for\",536870913],[131,\"^1\",\"~:logseq.property.asset/type\",536870913],[132,\"^1\",\"~:logseq.property.asset/external-url\",536870913],[133,\"^1\",\"~:logseq.property.asset/external-file-name\",536870913],[134,\"^1\",\"~:logseq.property.asset/size\",536870913],[135,\"^1\",\"~:logseq.property.asset/width\",536870913],[136,\"^1\",\"~:logseq.property.asset/height\",536870913],[137,\"^1\",\"~:logseq.property.asset/checksum\",536870913]]]", + "addresses": null + }, + { + "addr": 1000239, + "content": "[\"^ \",\"~:keys\",[[138,\"~:db/ident\",\"~:logseq.property.asset/last-visit-page\",536870913],[139,\"^1\",\"~:logseq.property.asset/remote-metadata\",536870913],[141,\"^1\",\"~:logseq.property.asset/resize-metadata\",536870913],[142,\"^1\",\"~:logseq.property.asset/align\",536870913],[143,\"^1\",\"~:logseq.property.fsrs/due\",536870913],[144,\"^1\",\"~:logseq.property.fsrs/state\",536870913],[145,\"^1\",\"~:logseq.property.user/name\",536870913],[146,\"^1\",\"~:logseq.property.user/email\",536870913],[147,\"^1\",\"~:logseq.property.user/avatar\",536870913],[148,\"^1\",\"~:logseq.property/enable-history?\",536870913],[150,\"^1\",\"~:logseq.property.history/block\",536870913],[151,\"^1\",\"~:logseq.property.history/property\",536870913],[152,\"^1\",\"~:logseq.property.history/ref-value\",536870913],[153,\"^1\",\"~:logseq.property.history/scalar-value\",536870913],[154,\"^1\",\"~:logseq.property/created-by-ref\",536870913],[155,\"^1\",\"~:logseq.property/deleted-at\",536870913]]]", + "addresses": null + }, + { + "addr": 1000240, + "content": "[\"^ \",\"~:keys\",[[156,\"~:db/ident\",\"~:logseq.property/deleted-by-ref\",536870913],[157,\"^1\",\"~:logseq.property.recycle/original-parent\",536870913],[158,\"^1\",\"~:logseq.property.recycle/original-page\",536870913],[159,\"^1\",\"~:logseq.property.recycle/original-order\",536870913],[160,\"^1\",\"~:logseq.property.reaction/emoji-id\",536870913],[161,\"^1\",\"~:logseq.property.reaction/target\",536870913],[162,\"^1\",\"~:logseq.property.agent/session-id\",536870913],[164,\"^1\",\"~:logseq.property/used-template\",536870913],[165,\"^1\",\"~:logseq.property/template-applied-to\",536870913],[166,\"^1\",\"~:logseq.property.sync/large-title-object\",536870913],[167,\"^1\",\"~:logseq.class/Journal\",536870913],[168,\"^1\",\"~:logseq.class/Whiteboard\",536870913],[169,\"^1\",\"~:logseq.class/Task\",536870913],[170,\"^1\",\"~:logseq.class/Comments\",536870913],[171,\"^1\",\"~:logseq.class/Comment\",536870913],[172,\"^1\",\"~:logseq.class/Query\",536870913],[173,\"^1\",\"~:logseq.class/Card\",536870913],[174,\"^1\",\"~:logseq.class/Cards\",536870913],[175,\"^1\",\"~:logseq.class/Asset\",536870913],[176,\"^1\",\"~:logseq.class/Code-block\",536870913],[177,\"^1\",\"~:logseq.class/Quote-block\",536870913],[178,\"^1\",\"~:logseq.class/Math-block\",536870913],[179,\"^1\",\"~:logseq.class/Pdf-annotation\",536870913],[12,\"~:db/index\",true,536870913],[13,\"^I\",true,536870913],[14,\"^I\",true,536870913]]]", + "addresses": null + }, + { + "addr": 1000241, + "content": "[\"^ \",\"~:keys\",[[30,\"~:block/uuid\",\"~u00000002-1157-7928-1300-000000000000\",536870913],[46,\"^1\",\"~u00000004-1118-8585-8000-000000000000\",536870913],[62,\"^1\",\"~u00000002-7532-8459-6000-000000000000\",536870913],[78,\"^1\",\"~u00000002-1870-0014-4300-000000000000\",536870913],[94,\"^1\",\"~u00000002-1438-8849-5400-000000000000\",536870913],[110,\"^1\",\"~u00000002-4524-3306-5000-000000000000\",536870913],[126,\"^1\",\"~u00000002-9750-5719-2000-000000000000\",536870913],[142,\"^1\",\"~u00000002-7135-0412-8000-000000000000\",536870913],[158,\"^1\",\"~u00000002-1658-7897-9900-000000000000\",536870913],[174,\"^1\",\"~u00000002-1284-2651-6700-000000000000\",536870913],[13,\"~:db/cardinality\",\"~:db.cardinality/one\",536870913],[29,\"^2\",\"^3\",536870913],[48,\"^2\",\"^3\",536870913],[70,\"^2\",\"^3\",536870913],[108,\"^2\",\"^3\",536870913],[127,\"^2\",\"^3\",536870913],[144,\"^2\",\"^3\",536870913],[2,\"~:db/ident\",\"~:logseq.class/Tag\",536870913],[18,\"^4\",\"~:logseq.property/ui-position\",536870913],[34,\"^4\",\"~:logseq.property.node/display-type\",536870913],[54,\"^4\",\"~:logseq.property/ls-type\",536870913],[70,\"^4\",\"~:logseq.property/choice-checkbox-state\",536870913],[86,\"^4\",\"~:logseq.property/deadline\",536870913],[105,\"^4\",\"~:logseq.property.repeat/checked-property\",536870913],[121,\"^4\",\"~:logseq.property.view/gallery-card-height\",536870913],[137,\"^4\",\"~:logseq.property.asset/checksum\",536870913],[155,\"^4\",\"~:logseq.property/deleted-at\",536870913],[14,\"~:db/index\",true,536870913]]]", + "addresses": "[1000213,1000214,1000215,1000216,1000217,1000218,1000219,1000220,1000221,1000222,1000223,1000224,1000225,1000226,1000227,1000228,1000229,1000230,1000231,1000232,1000233,1000234,1000235,1000236,1000237,1000238,1000239,1000240]" + }, + { + "addr": 1000242, + "content": "[\"^ \",\"~:keys\",[[15,\"~:db/index\",true,536870913],[16,\"^1\",true,536870913],[17,\"^1\",true,536870913],[18,\"^1\",true,536870913],[19,\"^1\",true,536870913],[20,\"^1\",true,536870913],[21,\"^1\",true,536870913],[22,\"^1\",true,536870913],[23,\"^1\",true,536870913],[24,\"^1\",true,536870913],[25,\"^1\",true,536870913],[26,\"^1\",true,536870913],[27,\"^1\",true,536870913],[28,\"^1\",true,536870913],[29,\"^1\",true,536870913],[30,\"^1\",true,536870913]]]", + "addresses": null + }, + { + "addr": 1000243, + "content": "[\"^ \",\"~:keys\",[[31,\"~:db/index\",true,536870913],[32,\"^1\",true,536870913],[33,\"^1\",true,536870913],[34,\"^1\",true,536870913],[35,\"^1\",true,536870913],[36,\"^1\",true,536870913],[37,\"^1\",true,536870913],[38,\"^1\",true,536870913],[39,\"^1\",true,536870913],[41,\"^1\",true,536870913],[42,\"^1\",true,536870913],[43,\"^1\",true,536870913],[45,\"^1\",true,536870913],[47,\"^1\",true,536870913],[48,\"^1\",true,536870913],[49,\"^1\",true,536870913]]]", + "addresses": null + }, + { + "addr": 1000244, + "content": "[\"^ \",\"~:keys\",[[51,\"~:db/index\",true,536870913],[52,\"^1\",true,536870913],[53,\"^1\",true,536870913],[54,\"^1\",true,536870913],[55,\"^1\",true,536870913],[56,\"^1\",true,536870913],[62,\"^1\",true,536870913],[63,\"^1\",true,536870913],[64,\"^1\",true,536870913],[65,\"^1\",true,536870913],[66,\"^1\",true,536870913],[67,\"^1\",true,536870913],[68,\"^1\",true,536870913],[69,\"^1\",true,536870913],[70,\"^1\",true,536870913],[71,\"^1\",true,536870913]]]", + "addresses": null + }, + { + "addr": 1000245, + "content": "[\"^ \",\"~:keys\",[[72,\"~:db/index\",true,536870913],[73,\"^1\",true,536870913],[74,\"^1\",true,536870913],[81,\"^1\",true,536870913],[86,\"^1\",true,536870913],[88,\"^1\",true,536870913],[90,\"^1\",true,536870913],[92,\"^1\",true,536870913],[99,\"^1\",true,536870913],[100,\"^1\",true,536870913],[104,\"^1\",true,536870913],[105,\"^1\",true,536870913],[106,\"^1\",true,536870913],[107,\"^1\",true,536870913],[108,\"^1\",true,536870913],[109,\"^1\",true,536870913]]]", + "addresses": null + }, + { + "addr": 1000246, + "content": "[\"^ \",\"~:keys\",[[110,\"~:db/index\",true,536870913],[111,\"^1\",true,536870913],[115,\"^1\",true,536870913],[116,\"^1\",true,536870913],[117,\"^1\",true,536870913],[118,\"^1\",true,536870913],[119,\"^1\",true,536870913],[120,\"^1\",true,536870913],[121,\"^1\",true,536870913],[122,\"^1\",true,536870913],[123,\"^1\",true,536870913],[124,\"^1\",true,536870913],[125,\"^1\",true,536870913],[126,\"^1\",true,536870913],[127,\"^1\",true,536870913],[128,\"^1\",true,536870913]]]", + "addresses": null + }, + { + "addr": 1000247, + "content": "[\"^ \",\"~:keys\",[[129,\"~:db/index\",true,536870913],[130,\"^1\",true,536870913],[131,\"^1\",true,536870913],[132,\"^1\",true,536870913],[133,\"^1\",true,536870913],[134,\"^1\",true,536870913],[135,\"^1\",true,536870913],[136,\"^1\",true,536870913],[137,\"^1\",true,536870913],[138,\"^1\",true,536870913],[139,\"^1\",true,536870913],[141,\"^1\",true,536870913],[142,\"^1\",true,536870913],[143,\"^1\",true,536870913],[144,\"^1\",true,536870913],[145,\"^1\",true,536870913]]]", + "addresses": null + }, + { + "addr": 1000248, + "content": "[\"^ \",\"~:keys\",[[146,\"~:db/index\",true,536870913],[147,\"^1\",true,536870913],[148,\"^1\",true,536870913],[150,\"^1\",true,536870913],[151,\"^1\",true,536870913],[152,\"^1\",true,536870913],[153,\"^1\",true,536870913],[154,\"^1\",true,536870913],[155,\"^1\",true,536870913],[156,\"^1\",true,536870913],[157,\"^1\",true,536870913],[158,\"^1\",true,536870913],[159,\"^1\",true,536870913],[160,\"^1\",true,536870913],[161,\"^1\",true,536870913],[162,\"^1\",true,536870913],[164,\"^1\",true,536870913],[165,\"^1\",true,536870913],[166,\"^1\",true,536870913],[15,\"~:db/valueType\",\"~:db.type/ref\",536870913],[19,\"^2\",\"^3\",536870913],[21,\"^2\",\"^3\",536870913],[22,\"^2\",\"^3\",536870913],[23,\"^2\",\"^3\",536870913],[26,\"^2\",\"^3\",536870913]]]", + "addresses": null + }, + { + "addr": 1000249, + "content": "[\"^ \",\"~:keys\",[[27,\"~:db/valueType\",\"~:db.type/ref\",536870913],[28,\"^1\",\"^2\",536870913],[30,\"^1\",\"^2\",536870913],[35,\"^1\",\"^2\",536870913],[37,\"^1\",\"^2\",536870913],[39,\"^1\",\"^2\",536870913],[41,\"^1\",\"^2\",536870913],[48,\"^1\",\"^2\",536870913],[49,\"^1\",\"^2\",536870913],[51,\"^1\",\"^2\",536870913],[53,\"^1\",\"^2\",536870913],[56,\"^1\",\"^2\",536870913],[63,\"^1\",\"^2\",536870913],[65,\"^1\",\"^2\",536870913],[66,\"^1\",\"^2\",536870913],[67,\"^1\",\"^2\",536870913]]]", + "addresses": null + }, + { + "addr": 1000250, + "content": "[\"^ \",\"~:keys\",[[68,\"~:db/valueType\",\"~:db.type/ref\",536870913],[71,\"^1\",\"^2\",536870913],[72,\"^1\",\"^2\",536870913],[73,\"^1\",\"^2\",536870913],[74,\"^1\",\"^2\",536870913],[81,\"^1\",\"^2\",536870913],[90,\"^1\",\"^2\",536870913],[92,\"^1\",\"^2\",536870913],[100,\"^1\",\"^2\",536870913],[104,\"^1\",\"^2\",536870913],[105,\"^1\",\"^2\",536870913],[106,\"^1\",\"^2\",536870913],[109,\"^1\",\"^2\",536870913],[111,\"^1\",\"^2\",536870913],[116,\"^1\",\"^2\",536870913],[117,\"^1\",\"^2\",536870913]]]", + "addresses": null + }, + { + "addr": 1000251, + "content": "[\"^ \",\"~:keys\",[[118,\"~:db/valueType\",\"~:db.type/ref\",536870913],[122,\"^1\",\"^2\",536870913],[129,\"^1\",\"^2\",536870913],[130,\"^1\",\"^2\",536870913],[150,\"^1\",\"^2\",536870913],[151,\"^1\",\"^2\",536870913],[152,\"^1\",\"^2\",536870913],[154,\"^1\",\"^2\",536870913],[156,\"^1\",\"^2\",536870913],[157,\"^1\",\"^2\",536870913],[158,\"^1\",\"^2\",536870913],[161,\"^1\",\"^2\",536870913],[164,\"^1\",\"^2\",536870913],[165,\"^1\",\"^2\",536870913],[180,\"~:file/content\",\"{:meta/version 1\\n\\n ;; Hide empty block properties\\n ;; Default value: false\\n ;; :ui/hide-empty-properties? false\\n\\n ;; Enable tooltip preview on hover.\\n ;; Default value: true\\n :ui/enable-tooltip? true\\n\\n ;; Display brackets [[]] around page references.\\n ;; Default value: true\\n ;; :ui/show-brackets? true\\n\\n ;; Display all lines of a block when referencing ((block)).\\n ;; Default value: false\\n :ui/show-full-blocks? false\\n\\n ;; Automatically expand block references when zooming in.\\n ;; Default value: true\\n :ui/auto-expand-block-refs? true\\n\\n ;; Disable accent marks when searching.\\n ;; After changing this setting, rebuild the search index by pressing (^C ^S).\\n ;; Default value: true\\n :feature/enable-search-remove-accents? true\\n\\n ;; Enable flashcards.\\n ;; Default value: true\\n ;; :feature/enable-flashcards? true\\n\\n ;; Disable the journal's built-in 'Scheduled tasks and deadlines' query.\\n ;; Default value: false\\n ;; :feature/disable-scheduled-and-deadline-query? false\\n\\n ;; Specify the number of days displayed in the future for\\n ;; the 'scheduled tasks and deadlines' query.\\n ;; Example usage:\\n ;; Display all scheduled and deadline blocks for the next 14 days:\\n ;; :scheduled/future-days 14\\n ;; Default value: 7\\n ;; :scheduled/future-days 7\\n\\n ;; Specify the first day of the week.\\n ;; Available options:\\n ;; - integer from 0 to 6 (Monday to Sunday)\\n ;; Default value: 6 (Sunday)\\n :start-of-week 6\\n\\n ;; Specify a custom CSS import.\\n ;; This option takes precedence over the local `logseq/custom.css` file.\\n ;; Example usage:\\n ;; :custom-css-url \\\"@import url('https://cdn.jsdelivr.net/gh/dracula/logseq@master/custom.css');\\\"\\n\\n ;; Specify a custom JS import.\\n ;; This option takes precedence over the local `logseq/custom.js` file.\\n ;; Example usage:\\n ;; :custom-js-url \\\"https://cdn.logseq.com/custom.js\\\"\\n\\n ;; Set bullet indentation when exporting\\n ;; Available options:\\n ;; - `:eight-spaces` as eight spaces\\n ;; - `:four-spaces` as four spaces\\n ;; - `:two-spaces` as two spaces\\n ;; - `:tab` as a tab character (default)\\n ;; :export/bullet-indentation :tab\\n\\n ;; Publish all pages within the Graph\\n ;; Regardless of whether individual pages have been marked as public.\\n ;; Default value: false\\n ;; :publishing/all-pages-public? false\\n\\n ;; Define the default home page and sidebar status.\\n ;; If unspecified, the journal page will be loaded on startup and the right sidebar will stay hidden.\\n ;; The `:page` value represents the name of the page displayed at startup.\\n ;; Available options for `:sidebar` are:\\n ;; - \\\"Contents\\\" to display the Contents page in the right sidebar.\\n ;; - A specific page name to display in the right sidebar.\\n ;; - An array of multiple pages, e.g., [\\\"Contents\\\" \\\"Page A\\\" \\\"Page B\\\"].\\n ;; If `:sidebar` remains unset, the right sidebar will stay hidden.\\n ;; Examples:\\n ;; 1. Set \\\"Changelog\\\" as the home page and display \\\"Contents\\\" in the right sidebar:\\n ;; :default-home {:page \\\"Changelog\\\", :sidebar \\\"Contents\\\"}\\n ;; 2. Set \\\"Jun 3rd, 2021\\\" as the home page without the right sidebar:\\n ;; :default-home {:page \\\"Jun 3rd, 2021\\\"}\\n ;; 3. Set \\\"home\\\" as the home page and display multiple pages in the right sidebar:\\n ;; :default-home {:page \\\"home\\\", :sidebar [\\\"Page A\\\" \\\"Page B\\\"]}\\n\\n ;; Configure custom shortcuts.\\n ;; Syntax:\\n ;; 1. + indicates simultaneous key presses, e.g., `Ctrl+Shift+a`.\\n ;; 2. A space between keys represents key chords, e.g., `t s` means\\n ;; pressing `t` followed by `s`.\\n ;; 3. mod refers to `Ctrl` for Windows/Linux and `Command` for Mac.\\n ;; 4. Use false to disable a specific shortcut.\\n ;; 5. You can define multiple bindings for a single action, e.g., [\\\"ctrl+j\\\" \\\"down\\\"].\\n ;; The full list of configurable shortcuts is available at:\\n ;; https://github.com/logseq/logseq/blob/master/src/main/frontend/modules/shortcut/config.cljs\\n ;; Example:\\n ;; :shortcuts\\n ;; {:editor/new-block \\\"enter\\\"\\n ;; :editor/new-line \\\"shift+enter\\\"\\n ;; :editor/insert-link \\\"mod+shift+k\\\"\\n ;; :editor/highlight false\\n ;; :ui/toggle-settings \\\"t s\\\"\\n ;; :editor/up [\\\"ctrl+k\\\" \\\"up\\\"]\\n ;; :editor/down [\\\"ctrl+j\\\" \\\"down\\\"]\\n ;; :editor/left [\\\"ctrl+h\\\" \\\"left\\\"]\\n ;; :editor/right [\\\"ctrl+l\\\" \\\"right\\\"]}\\n :shortcuts {}\\n\\n ;; Configure the behavior of pressing Enter in document mode.\\n ;; if set to true, pressing Enter will create a new block.\\n ;; Default value: false\\n :shortcut/doc-mode-enter-for-new-block? false\\n\\n ;; Block content larger than `block/title-max-length` will not be searchable\\n ;; or editable for performance.\\n ;; Default value: 10000\\n :block/title-max-length 10000\\n\\n ;; Display command documentation on hover.\\n ;; Default value: true\\n :ui/show-command-doc? true\\n\\n ;; Display empty bullet points.\\n ;; Default value: false\\n :ui/show-empty-bullets? false\\n\\n ;; Pre-defined :view function to use with advanced queries.\\n :query/views\\n {:pprint\\n (fn [r] [:pre.code (pprint r)])}\\n\\n ;; Advanced queries `:result-transform` function.\\n ;; Transform the query result before displaying it.\\n ;; Example usage for DB graphs:\\n;; :query/result-transforms\\n;; {:sort-by-priority\\n;; (fn [result] (sort-by (fn [h] (get h :logseq.property/priority \\\"Z\\\")) result))}\\n\\n;; Queries will be displayed at the bottom of today's journal page.\\n;; Example usage:\\n;; :default-queries\\n;; {:journals []}\\n\\n ;; Add custom commands to the command palette\\n ;; Example usage:\\n ;; :commands\\n ;; [\\n ;; [\\\"js\\\" \\\"Javascript\\\"]\\n ;; [\\\"md\\\" \\\"Markdown\\\"]\\n ;; ]\\n :commands []\\n\\n ;; Enable collapsing blocks with titles but no children.\\n ;; By default, only blocks with children can be collapsed.\\n ;; Setting `:outliner/block-title-collapse-enabled?` to true allows collapsing\\n ;; blocks with titles (multiple lines) and content. For example:\\n ;; - block title\\n ;; block content\\n ;; Default value: false\\n :outliner/block-title-collapse-enabled? false\\n\\n ;; Macros replace texts and will make you more productive.\\n ;; Example usage:\\n ;; Change the :macros value below to:\\n ;; {\\\"poem\\\" \\\"Rose is $1, violet's $2. Life's ordered: Org assists you.\\\"}\\n ;; input \\\"{{poem red,blue}}\\\"\\n ;; becomes\\n ;; Rose is red, violet's blue. Life's ordered: Org assists you.\\n :macros {}\\n\\n ;; Configure the default expansion level for linked references.\\n ;; For example, consider the following block hierarchy:\\n ;; - a [[page]] (level 1)\\n ;; - b (level 2)\\n ;; - c (level 3)\\n ;; - d (level 4)\\n ;;\\n ;; With the default value of level 2, block b will be collapsed.\\n ;; If the level's value is set to 3, block c will be collapsed.\\n ;; Default value: 2\\n :ref/default-open-blocks-level 2\\n\\n ;; Mobile photo upload configuration.\\n ;; :mobile/photo\\n ;; {:allow-editing? true\\n ;; :quality 80}\\n\\n ;; Mobile features options\\n ;; Gestures\\n ;; Example usage:\\n ;; :mobile\\n ;; {:gestures/disabled-in-block-with-tags [\\\"kanban\\\"]}\\n\\n ;; Extra CodeMirror options\\n ;; See https://codemirror.net/5/doc/manual.html#config for possible options\\n ;; Example usage:\\n ;; :editor/extra-codemirror-options\\n ;; {:lineWrapping false ; Default value: false\\n ;; :lineNumbers true ; Default value: true\\n ;; :readOnly false} ; Default value: false\\n\\n ;; Enable logical outdenting\\n ;; Default value: false\\n ;; :editor/logical-outdenting? false\\n\\n ;; Prefer pasting the file when text and a file are in the clipboard.\\n ;; Default value: false\\n ;; :editor/preferred-pasting-file? false\\n\\n ;; Quick capture templates for receiving content from other apps.\\n ;; Each template contains three elements {time}, {text} and {url}, which can be auto-expanded\\n ;; by receiving content from other apps. Note: the {} cannot be omitted.\\n ;; - {time}: capture time\\n ;; - {date}: capture date using current date format, use `[[{date}]]` to get a page reference\\n ;; - {text}: text that users selected before sharing.\\n ;; - {url}: URL or assets path for media files stored in Logseq.\\n ;; You can also reorder them or use only one or two of them in the template.\\n ;; You can also insert or format any text in the template, as shown in the following examples.\\n ;; :quick-capture-templates\\n ;; {:text \\\"[[quick capture]] **{time}**: {text} from {url}\\\"\\n ;; :media \\\"[[quick capture]] **{time}**: {url}\\\"}\\n\\n ;; Quick capture options.\\n ;; - insert-today? Insert the capture at the end of today's journal page (boolean).\\n ;; - redirect-page? Redirect to the quick capture page after capturing (boolean).\\n ;; - default-page The default page to capture to if insert-today? is false (string).\\n ;; - prettify-url? Whether urls to YouTube or X should be wrapped with {{video ...}} or {{twitter ...}}\\n ;; :quick-capture-options\\n ;; {:insert-today? false ;; Default value: true\\n ;; :redirect-page? false ;; Default value: false\\n ;; :default-page \\\"quick capture\\\" ;; Default page: \\\"quick capture\\\"\\n ;; :prettify-url? true} ;; Default value: true\\n\\n }\\n\",536870913],[181,\"^3\",\"\",536870913]]]", + "addresses": null + }, + { + "addr": 1000252, + "content": "[\"^ \",\"~:keys\",[[182,\"~:file/content\",\"\",536870913],[183,\"^1\",\"\",536870913],[184,\"^1\",\"\",536870913],[180,\"~:file/created-at\",\"~m1786627375714\",536870913],[181,\"^2\",\"~m1786627375714\",536870913],[182,\"^2\",\"~m1786627375714\",536870913],[183,\"^2\",\"~m1786627375714\",536870913],[184,\"^2\",\"~m1786627375714\",536870913],[180,\"~:file/last-modified-at\",\"~m1786627375714\",536870913],[181,\"^3\",\"~m1786627375714\",536870913],[182,\"^3\",\"~m1786627375714\",536870913],[183,\"^3\",\"~m1786627375714\",536870913],[184,\"^3\",\"~m1786627375714\",536870913],[180,\"~:file/path\",\"logseq/config.edn\",536870913],[181,\"^4\",\"logseq/custom.css\",536870913],[182,\"^4\",\"logseq/custom.js\",536870913],[183,\"^4\",\"logseq/publish.css\",536870913],[184,\"^4\",\"logseq/publish.js\",536870913],[6,\"~:kv/value\",\"db\",536870913],[7,\"^5\",[\"^ \",\"~:major\",65,\"~:minor\",33],536870913],[8,\"^5\",[\"^ \",\"^6\",65,\"^7\",33],536870913],[9,\"^5\",1786627375714,536870913]]]", + "addresses": null + }, + { + "addr": 1000253, + "content": "[\"^ \",\"~:keys\",[[11,\"~:kv/value\",\"~u00000000-7b10-4374-96ba-e7f6ea9cbb32\",536870913],[1,\"~:logseq.property/built-in?\",true,536870913],[2,\"^2\",true,536870913],[3,\"^2\",true,536870913],[4,\"^2\",true,536870913],[5,\"^2\",true,536870913],[12,\"^2\",true,536870913],[13,\"^2\",true,536870913],[14,\"^2\",true,536870913],[15,\"^2\",true,536870913],[16,\"^2\",true,536870913],[17,\"^2\",true,536870913],[18,\"^2\",true,536870913],[19,\"^2\",true,536870913],[20,\"^2\",true,536870913],[21,\"^2\",true,536870913],[22,\"^2\",true,536870913],[23,\"^2\",true,536870913],[24,\"^2\",true,536870913],[25,\"^2\",true,536870913],[26,\"^2\",true,536870913]]]", + "addresses": null + }, + { + "addr": 1000254, + "content": "[\"^ \",\"~:keys\",[[27,\"~:logseq.property/built-in?\",true,536870913],[28,\"^1\",true,536870913],[29,\"^1\",true,536870913],[30,\"^1\",true,536870913],[31,\"^1\",true,536870913],[32,\"^1\",true,536870913],[33,\"^1\",true,536870913],[34,\"^1\",true,536870913],[35,\"^1\",true,536870913],[36,\"^1\",true,536870913],[37,\"^1\",true,536870913],[38,\"^1\",true,536870913],[39,\"^1\",true,536870913],[40,\"^1\",true,536870913],[41,\"^1\",true,536870913],[42,\"^1\",true,536870913]]]", + "addresses": null + }, + { + "addr": 1000255, + "content": "[\"^ \",\"~:keys\",[[43,\"~:logseq.property/built-in?\",true,536870913],[44,\"^1\",true,536870913],[45,\"^1\",true,536870913],[46,\"^1\",true,536870913],[47,\"^1\",true,536870913],[48,\"^1\",true,536870913],[49,\"^1\",true,536870913],[50,\"^1\",true,536870913],[51,\"^1\",true,536870913],[52,\"^1\",true,536870913],[53,\"^1\",true,536870913],[54,\"^1\",true,536870913],[55,\"^1\",true,536870913],[56,\"^1\",true,536870913],[57,\"^1\",true,536870913],[58,\"^1\",true,536870913]]]", + "addresses": null + }, + { + "addr": 1000256, + "content": "[\"^ \",\"~:keys\",[[59,\"~:logseq.property/built-in?\",true,536870913],[60,\"^1\",true,536870913],[61,\"^1\",true,536870913],[62,\"^1\",true,536870913],[63,\"^1\",true,536870913],[64,\"^1\",true,536870913],[65,\"^1\",true,536870913],[66,\"^1\",true,536870913],[67,\"^1\",true,536870913],[68,\"^1\",true,536870913],[69,\"^1\",true,536870913],[70,\"^1\",true,536870913],[71,\"^1\",true,536870913],[72,\"^1\",true,536870913],[73,\"^1\",true,536870913],[74,\"^1\",true,536870913]]]", + "addresses": null + }, + { + "addr": 1000257, + "content": "[\"^ \",\"~:keys\",[[75,\"~:logseq.property/built-in?\",true,536870913],[76,\"^1\",true,536870913],[77,\"^1\",true,536870913],[78,\"^1\",true,536870913],[79,\"^1\",true,536870913],[80,\"^1\",true,536870913],[81,\"^1\",true,536870913],[82,\"^1\",true,536870913],[83,\"^1\",true,536870913],[84,\"^1\",true,536870913],[85,\"^1\",true,536870913],[86,\"^1\",true,536870913],[87,\"^1\",true,536870913],[88,\"^1\",true,536870913],[89,\"^1\",true,536870913],[90,\"^1\",true,536870913]]]", + "addresses": null + }, + { + "addr": 1000258, + "content": "[\"^ \",\"~:keys\",[[91,\"~:logseq.property/built-in?\",true,536870913],[92,\"^1\",true,536870913],[93,\"^1\",true,536870913],[94,\"^1\",true,536870913],[95,\"^1\",true,536870913],[96,\"^1\",true,536870913],[97,\"^1\",true,536870913],[98,\"^1\",true,536870913],[99,\"^1\",true,536870913],[100,\"^1\",true,536870913],[101,\"^1\",true,536870913],[102,\"^1\",true,536870913],[103,\"^1\",true,536870913],[104,\"^1\",true,536870913],[105,\"^1\",true,536870913],[106,\"^1\",true,536870913]]]", + "addresses": null + }, + { + "addr": 1000259, + "content": "[\"^ \",\"~:keys\",[[107,\"~:logseq.property/built-in?\",true,536870913],[108,\"^1\",true,536870913],[109,\"^1\",true,536870913],[110,\"^1\",true,536870913],[111,\"^1\",true,536870913],[112,\"^1\",true,536870913],[113,\"^1\",true,536870913],[114,\"^1\",true,536870913],[115,\"^1\",true,536870913],[116,\"^1\",true,536870913],[117,\"^1\",true,536870913],[118,\"^1\",true,536870913],[119,\"^1\",true,536870913],[120,\"^1\",true,536870913],[121,\"^1\",true,536870913],[122,\"^1\",true,536870913]]]", + "addresses": null + }, + { + "addr": 1000260, + "content": "[\"^ \",\"~:keys\",[[123,\"~:logseq.property/built-in?\",true,536870913],[124,\"^1\",true,536870913],[125,\"^1\",true,536870913],[126,\"^1\",true,536870913],[127,\"^1\",true,536870913],[128,\"^1\",true,536870913],[129,\"^1\",true,536870913],[130,\"^1\",true,536870913],[131,\"^1\",true,536870913],[132,\"^1\",true,536870913],[133,\"^1\",true,536870913],[134,\"^1\",true,536870913],[135,\"^1\",true,536870913],[136,\"^1\",true,536870913],[137,\"^1\",true,536870913],[138,\"^1\",true,536870913]]]", + "addresses": null + }, + { + "addr": 1000261, + "content": "[\"^ \",\"~:keys\",[[139,\"~:logseq.property/built-in?\",true,536870913],[140,\"^1\",true,536870913],[141,\"^1\",true,536870913],[142,\"^1\",true,536870913],[143,\"^1\",true,536870913],[144,\"^1\",true,536870913],[145,\"^1\",true,536870913],[146,\"^1\",true,536870913],[147,\"^1\",true,536870913],[148,\"^1\",true,536870913],[149,\"^1\",true,536870913],[150,\"^1\",true,536870913],[151,\"^1\",true,536870913],[152,\"^1\",true,536870913],[153,\"^1\",true,536870913],[154,\"^1\",true,536870913]]]", + "addresses": null + }, + { + "addr": 1000262, + "content": "[\"^ \",\"~:keys\",[[155,\"~:logseq.property/built-in?\",true,536870913],[156,\"^1\",true,536870913],[157,\"^1\",true,536870913],[158,\"^1\",true,536870913],[159,\"^1\",true,536870913],[160,\"^1\",true,536870913],[161,\"^1\",true,536870913],[162,\"^1\",true,536870913],[163,\"^1\",true,536870913],[164,\"^1\",true,536870913],[165,\"^1\",true,536870913],[166,\"^1\",true,536870913],[167,\"^1\",true,536870913],[168,\"^1\",true,536870913],[169,\"^1\",true,536870913],[170,\"^1\",true,536870913],[171,\"^1\",true,536870913],[172,\"^1\",true,536870913],[173,\"^1\",true,536870913],[174,\"^1\",true,536870913],[175,\"^1\",true,536870913],[176,\"^1\",true,536870913],[177,\"^1\",true,536870913],[178,\"^1\",true,536870913],[179,\"^1\",true,536870913],[185,\"^1\",true,536870913],[186,\"^1\",true,536870913],[187,\"^1\",true,536870913],[188,\"^1\",true,536870913],[189,\"^1\",true,536870913],[190,\"^1\",true,536870913],[76,\"~:logseq.property/choice-checkbox-state\",false,536870913]]]", + "addresses": null + }, + { + "addr": 1000263, + "content": "[\"^ \",\"~:keys\",[[79,\"~:logseq.property/choice-checkbox-state\",true,536870913],[106,\"~:logseq.property/classes\",4,536870913],[164,\"^2\",5,536870913],[40,\"~:logseq.property/created-from-property\",35,536870913],[44,\"^3\",35,536870913],[46,\"^3\",35,536870913],[50,\"^3\",35,536870913],[57,\"^3\",56,536870913],[58,\"^3\",56,536870913],[59,\"^3\",56,536870913],[60,\"^3\",56,536870913],[61,\"^3\",56,536870913],[75,\"^3\",74,536870913],[76,\"^3\",74,536870913],[77,\"^3\",74,536870913],[78,\"^3\",74,536870913]]]", + "addresses": null + }, + { + "addr": 1000264, + "content": "[\"^ \",\"~:keys\",[[79,\"~:logseq.property/created-from-property\",74,536870913],[80,\"^1\",74,536870913],[82,\"^1\",81,536870913],[83,\"^1\",81,536870913],[84,\"^1\",81,536870913],[85,\"^1\",81,536870913],[87,\"^1\",35,536870913],[89,\"^1\",35,536870913],[91,\"^1\",90,536870913],[93,\"^1\",92,536870913],[94,\"^1\",92,536870913],[95,\"^1\",92,536870913],[96,\"^1\",92,536870913],[97,\"^1\",92,536870913],[98,\"^1\",92,536870913],[101,\"^1\",100,536870913],[102,\"^1\",100,536870913],[103,\"^1\",100,536870913],[112,\"^1\",111,536870913],[113,\"^1\",111,536870913],[114,\"^1\",111,536870913],[140,\"^1\",35,536870913],[149,\"^1\",35,536870913],[163,\"^1\",35,536870913],[74,\"~:logseq.property/default-value\",76,536870913],[90,\"^2\",91,536870913]]]", + "addresses": null + }, + { + "addr": 1000265, + "content": "[\"^ \",\"~:keys\",[[92,\"~:logseq.property/default-value\",95,536870913],[100,\"^1\",103,536870913],[111,\"^1\",112,536870913],[39,\"~:logseq.property/description\",40,536870913],[43,\"^2\",44,536870913],[45,\"^2\",46,536870913],[49,\"^2\",50,536870913],[86,\"^2\",87,536870913],[88,\"^2\",89,536870913],[139,\"^2\",140,536870913],[148,\"^2\",149,536870913],[162,\"^2\",163,536870913],[74,\"~:logseq.property/enable-history?\",true,536870913],[81,\"^3\",true,536870913],[74,\"~:logseq.property/hide-empty-value\",true,536870913],[81,\"^4\",true,536870913],[86,\"^4\",true,536870913],[88,\"^4\",true,536870913],[90,\"^4\",true,536870913],[92,\"^4\",true,536870913],[100,\"^4\",true,536870913],[106,\"^4\",true,536870913],[12,\"~:logseq.property/hide?\",true,536870913],[13,\"^5\",true,536870913],[14,\"^5\",true,536870913]]]", + "addresses": null + }, + { + "addr": 1000266, + "content": "[\"^ \",\"~:keys\",[[15,\"~:logseq.property/hide?\",true,536870913],[16,\"^1\",true,536870913],[17,\"^1\",true,536870913],[18,\"^1\",true,536870913],[19,\"^1\",true,536870913],[20,\"^1\",true,536870913],[23,\"^1\",true,536870913],[24,\"^1\",true,536870913],[25,\"^1\",true,536870913],[26,\"^1\",true,536870913],[27,\"^1\",true,536870913],[28,\"^1\",true,536870913],[29,\"^1\",true,536870913],[30,\"^1\",true,536870913],[31,\"^1\",true,536870913],[32,\"^1\",true,536870913]]]", + "addresses": null + }, + { + "addr": 1000267, + "content": "[\"^ \",\"~:keys\",[[30,\"~:db/index\",true,536870913],[49,\"^1\",true,536870913],[71,\"^1\",true,536870913],[109,\"^1\",true,536870913],[128,\"^1\",true,536870913],[145,\"^1\",true,536870913],[26,\"~:db/valueType\",\"~:db.type/ref\",536870913],[67,\"^2\",\"^3\",536870913],[117,\"^2\",\"^3\",536870913],[181,\"~:file/content\",\"\",536870913],[9,\"~:kv/value\",1786627375714,536870913],[26,\"~:logseq.property/built-in?\",true,536870913],[42,\"^6\",true,536870913],[58,\"^6\",true,536870913],[74,\"^6\",true,536870913],[90,\"^6\",true,536870913],[106,\"^6\",true,536870913],[122,\"^6\",true,536870913],[138,\"^6\",true,536870913],[154,\"^6\",true,536870913],[76,\"~:logseq.property/choice-checkbox-state\",false,536870913],[78,\"~:logseq.property/created-from-property\",74,536870913],[90,\"~:logseq.property/default-value\",91,536870913],[14,\"~:logseq.property/hide?\",true,536870913],[32,\"^:\",true,536870913]]]", + "addresses": "[1000242,1000243,1000244,1000245,1000246,1000247,1000248,1000249,1000250,1000251,1000252,1000253,1000254,1000255,1000256,1000257,1000258,1000259,1000260,1000261,1000262,1000263,1000264,1000265,1000266]" + }, + { + "addr": 1000268, + "content": "[\"^ \",\"~:keys\",[[33,\"~:logseq.property/hide?\",true,536870913],[34,\"^1\",true,536870913],[36,\"^1\",true,536870913],[37,\"^1\",true,536870913],[38,\"^1\",true,536870913],[48,\"^1\",true,536870913],[51,\"^1\",true,536870913],[52,\"^1\",true,536870913],[53,\"^1\",true,536870913],[54,\"^1\",true,536870913],[55,\"^1\",true,536870913],[56,\"^1\",true,536870913],[62,\"^1\",true,536870913],[63,\"^1\",true,536870913],[64,\"^1\",true,536870913],[65,\"^1\",true,536870913]]]", + "addresses": null + }, + { + "addr": 1000269, + "content": "[\"^ \",\"~:keys\",[[66,\"~:logseq.property/hide?\",true,536870913],[67,\"^1\",true,536870913],[68,\"^1\",true,536870913],[70,\"^1\",true,536870913],[71,\"^1\",true,536870913],[72,\"^1\",true,536870913],[73,\"^1\",true,536870913],[99,\"^1\",true,536870913],[104,\"^1\",true,536870913],[105,\"^1\",true,536870913],[108,\"^1\",true,536870913],[110,\"^1\",true,536870913],[111,\"^1\",true,536870913],[115,\"^1\",true,536870913],[116,\"^1\",true,536870913],[117,\"^1\",true,536870913]]]", + "addresses": null + }, + { + "addr": 1000270, + "content": "[\"^ \",\"~:keys\",[[118,\"~:logseq.property/hide?\",true,536870913],[119,\"^1\",true,536870913],[120,\"^1\",true,536870913],[121,\"^1\",true,536870913],[122,\"^1\",true,536870913],[123,\"^1\",true,536870913],[124,\"^1\",true,536870913],[125,\"^1\",true,536870913],[126,\"^1\",true,536870913],[127,\"^1\",true,536870913],[128,\"^1\",true,536870913],[129,\"^1\",true,536870913],[130,\"^1\",true,536870913],[131,\"^1\",true,536870913],[132,\"^1\",false,536870913],[133,\"^1\",true,536870913]]]", + "addresses": null + }, + { + "addr": 1000271, + "content": "[\"^ \",\"~:keys\",[[134,\"~:logseq.property/hide?\",true,536870913],[135,\"^1\",true,536870913],[136,\"^1\",true,536870913],[137,\"^1\",true,536870913],[138,\"^1\",true,536870913],[139,\"^1\",true,536870913],[141,\"^1\",true,536870913],[142,\"^1\",true,536870913],[143,\"^1\",false,536870913],[144,\"^1\",false,536870913],[145,\"^1\",false,536870913],[146,\"^1\",false,536870913],[147,\"^1\",false,536870913],[150,\"^1\",true,536870913],[151,\"^1\",true,536870913],[152,\"^1\",true,536870913]]]", + "addresses": null + }, + { + "addr": 1000272, + "content": "[\"^ \",\"~:keys\",[[153,\"~:logseq.property/hide?\",true,536870913],[154,\"^1\",true,536870913],[155,\"^1\",true,536870913],[156,\"^1\",true,536870913],[157,\"^1\",true,536870913],[158,\"^1\",true,536870913],[159,\"^1\",true,536870913],[160,\"^1\",true,536870913],[161,\"^1\",true,536870913],[162,\"^1\",true,536870913],[164,\"^1\",true,536870913],[166,\"^1\",true,536870913],[187,\"^1\",true,536870913],[188,\"^1\",true,536870913],[189,\"^1\",true,536870913],[190,\"^1\",true,536870913],[75,\"~:logseq.property/icon\",[\"^ \",\"~:type\",\"~:tabler-icon\",\"~:id\",\"Backlog\"],536870913],[76,\"^2\",[\"^ \",\"^3\",\"^4\",\"^5\",\"Todo\"],536870913],[77,\"^2\",[\"^ \",\"^3\",\"^4\",\"^5\",\"InProgress50\"],536870913]]]", + "addresses": null + }, + { + "addr": 1000273, + "content": "[\"^ \",\"~:keys\",[[78,\"~:logseq.property/icon\",[\"^ \",\"~:type\",\"~:tabler-icon\",\"~:id\",\"InReview\"],536870913],[79,\"^1\",[\"^ \",\"^2\",\"^3\",\"^4\",\"Done\"],536870913],[80,\"^1\",[\"^ \",\"^2\",\"^3\",\"^4\",\"Cancelled\"],536870913],[82,\"^1\",[\"^ \",\"^2\",\"^3\",\"^4\",\"priorityLvlLow\"],536870913],[83,\"^1\",[\"^ \",\"^2\",\"^3\",\"^4\",\"priorityLvlMedium\"],536870913],[84,\"^1\",[\"^ \",\"^2\",\"^3\",\"^4\",\"priorityLvlHigh\"],536870913],[85,\"^1\",[\"^ \",\"^2\",\"^3\",\"^4\",\"priorityLvlUrgent\"],536870913],[112,\"^1\",[\"^ \",\"^2\",\"^3\",\"^4\",\"table\"],536870913],[113,\"^1\",[\"^ \",\"^2\",\"^3\",\"^4\",\"list\"],536870913],[114,\"^1\",[\"^ \",\"^2\",\"^3\",\"^4\",\"layout-grid\"],536870913],[170,\"^1\",[\"^ \",\"^2\",\"^3\",\"^4\",\"message-circle\"],536870913],[172,\"^1\",[\"^ \",\"^2\",\"^3\",\"^4\",\"search\"],536870913],[174,\"^1\",[\"^ \",\"^2\",\"^3\",\"^4\",\"search\"],536870913],[19,\"~:logseq.property/public?\",false,536870913],[20,\"^5\",false,536870913],[21,\"^5\",true,536870913],[22,\"^5\",true,536870913],[23,\"^5\",false,536870913],[24,\"^5\",false,536870913],[25,\"^5\",false,536870913]]]", + "addresses": null + }, + { + "addr": 1000274, + "content": "[\"^ \",\"~:keys\",[[26,\"~:logseq.property/public?\",false,536870913],[27,\"^1\",false,536870913],[28,\"^1\",false,536870913],[29,\"^1\",false,536870913],[30,\"^1\",false,536870913],[31,\"^1\",false,536870913],[32,\"^1\",false,536870913],[33,\"^1\",false,536870913],[34,\"^1\",false,536870913],[35,\"^1\",true,536870913],[36,\"^1\",false,536870913],[37,\"^1\",false,536870913],[38,\"^1\",false,536870913],[39,\"^1\",true,536870913],[41,\"^1\",true,536870913],[42,\"^1\",true,536870913]]]", + "addresses": null + }, + { + "addr": 1000275, + "content": "[\"^ \",\"~:keys\",[[43,\"~:logseq.property/public?\",true,536870913],[45,\"^1\",true,536870913],[47,\"^1\",true,536870913],[48,\"^1\",true,536870913],[49,\"^1\",true,536870913],[68,\"^1\",false,536870913],[69,\"^1\",false,536870913],[71,\"^1\",false,536870913],[72,\"^1\",false,536870913],[74,\"^1\",true,536870913],[81,\"^1\",true,536870913],[86,\"^1\",true,536870913],[88,\"^1\",true,536870913],[90,\"^1\",false,536870913],[92,\"^1\",false,536870913],[100,\"^1\",false,536870913]]]", + "addresses": null + }, + { + "addr": 1000276, + "content": "[\"^ \",\"~:keys\",[[106,\"~:logseq.property/public?\",true,536870913],[108,\"^1\",true,536870913],[109,\"^1\",true,536870913],[110,\"^1\",true,536870913],[111,\"^1\",false,536870913],[115,\"^1\",false,536870913],[116,\"^1\",false,536870913],[117,\"^1\",false,536870913],[118,\"^1\",false,536870913],[119,\"^1\",false,536870913],[120,\"^1\",false,536870913],[121,\"^1\",false,536870913],[122,\"^1\",false,536870913],[123,\"^1\",false,536870913],[124,\"^1\",false,536870913],[125,\"^1\",false,536870913]]]", + "addresses": null + }, + { + "addr": 1000277, + "content": "[\"^ \",\"~:keys\",[[126,\"~:logseq.property/public?\",false,536870913],[127,\"^1\",false,536870913],[128,\"^1\",false,536870913],[129,\"^1\",false,536870913],[130,\"^1\",false,536870913],[131,\"^1\",false,536870913],[132,\"^1\",true,536870913],[133,\"^1\",false,536870913],[134,\"^1\",false,536870913],[135,\"^1\",false,536870913],[136,\"^1\",false,536870913],[137,\"^1\",false,536870913],[138,\"^1\",false,536870913],[139,\"^1\",false,536870913],[141,\"^1\",false,536870913],[142,\"^1\",false,536870913]]]", + "addresses": null + }, + { + "addr": 1000278, + "content": "[\"^ \",\"~:keys\",[[143,\"~:logseq.property/public?\",false,536870913],[144,\"^1\",false,536870913],[145,\"^1\",true,536870913],[146,\"^1\",true,536870913],[147,\"^1\",true,536870913],[148,\"^1\",true,536870913],[155,\"^1\",false,536870913],[156,\"^1\",false,536870913],[157,\"^1\",false,536870913],[158,\"^1\",false,536870913],[159,\"^1\",false,536870913],[160,\"^1\",false,536870913],[161,\"^1\",false,536870913],[162,\"^1\",true,536870913],[164,\"^1\",false,536870913],[165,\"^1\",true,536870913],[166,\"^1\",false,536870913],[119,\"~:logseq.property/scalar-default-value\",\"~:default\",536870913],[123,\"^2\",true,536870913],[12,\"~:logseq.property/type\",\"~:keyword\",536870913],[13,\"^4\",\"~:checkbox\",536870913],[14,\"^4\",\"^6\",536870913],[15,\"^4\",\"~:entity\",536870913],[16,\"^4\",\"^6\",536870913],[17,\"^4\",\"^5\",536870913],[18,\"^4\",\"^5\",536870913],[19,\"^4\",\"^7\",536870913]]]", + "addresses": null + }, + { + "addr": 1000279, + "content": "[\"^ \",\"~:keys\",[[20,\"~:logseq.property/type\",\"~:any\",536870913],[21,\"^1\",\"~:page\",536870913],[22,\"^1\",\"~:class\",536870913],[23,\"^1\",\"~:entity\",536870913],[24,\"^1\",\"~:string\",536870913],[25,\"^1\",\"~:checkbox\",536870913],[26,\"^1\",\"^5\",536870913],[27,\"^1\",\"^5\",536870913],[28,\"^1\",\"^5\",536870913],[29,\"^1\",\"^6\",536870913],[30,\"^1\",\"^5\",536870913],[31,\"^1\",\"~:raw-number\",536870913],[32,\"^1\",\"~:datetime\",536870913],[33,\"^1\",\"^9\",536870913],[34,\"^1\",\"~:keyword\",536870913],[35,\"^1\",\"~:default\",536870913]]]", + "addresses": null + }, + { + "addr": 1000280, + "content": "[\"^ \",\"~:keys\",[[36,\"~:logseq.property/type\",\"~:string\",536870913],[37,\"^1\",\"~:entity\",536870913],[38,\"^1\",\"~:any\",536870913],[39,\"^1\",\"~:class\",536870913],[41,\"^1\",\"~:property\",536870913],[42,\"^1\",\"^2\",536870913],[43,\"^1\",\"~:checkbox\",536870913],[45,\"^1\",\"^7\",536870913],[47,\"^1\",\"^7\",536870913],[48,\"^1\",\"~:default\",536870913],[49,\"^1\",\"~:page\",536870913],[51,\"^1\",\"^8\",536870913],[52,\"^1\",\"^4\",536870913],[53,\"^1\",\"^3\",536870913],[54,\"^1\",\"~:keyword\",536870913],[55,\"^1\",\"^:\",536870913]]]", + "addresses": null + }, + { + "addr": 1000281, + "content": "[\"^ \",\"~:keys\",[[56,\"~:logseq.property/type\",\"~:default\",536870913],[62,\"^1\",\"~:raw-number\",536870913],[63,\"^1\",\"~:entity\",536870913],[64,\"^1\",\"~:map\",536870913],[65,\"^1\",\"^2\",536870913],[66,\"^1\",\"~:node\",536870913],[67,\"^1\",\"^6\",536870913],[68,\"^1\",\"^6\",536870913],[69,\"^1\",\"~:string\",536870913],[70,\"^1\",\"~:checkbox\",536870913],[71,\"^1\",\"~:class\",536870913],[72,\"^1\",\"^6\",536870913],[73,\"^1\",\"~:property\",536870913],[74,\"^1\",\"^2\",536870913],[81,\"^1\",\"^2\",536870913],[86,\"^1\",\"~:datetime\",536870913]]]", + "addresses": null + }, + { + "addr": 1000282, + "content": "[\"^ \",\"~:keys\",[[88,\"~:logseq.property/type\",\"~:datetime\",536870913],[90,\"^1\",\"~:number\",536870913],[92,\"^1\",\"~:default\",536870913],[99,\"^1\",\"~:checkbox\",536870913],[100,\"^1\",\"^4\",536870913],[104,\"^1\",\"~:property\",536870913],[105,\"^1\",\"^6\",536870913],[106,\"^1\",\"~:node\",536870913],[107,\"^1\",\"~:map\",536870913],[108,\"^1\",\"^5\",536870913],[109,\"^1\",\"~:url\",536870913],[110,\"^1\",\"^5\",536870913],[111,\"^1\",\"^4\",536870913],[115,\"^1\",\"~:keyword\",536870913],[116,\"^1\",\"^6\",536870913],[117,\"^1\",\"^6\",536870913]]]", + "addresses": null + }, + { + "addr": 1000283, + "content": "[\"^ \",\"~:keys\",[[118,\"~:logseq.property/type\",\"~:property\",536870913],[119,\"^1\",\"~:keyword\",536870913],[120,\"^1\",\"~:raw-number\",536870913],[121,\"^1\",\"^4\",536870913],[122,\"^1\",\"^2\",536870913],[123,\"^1\",\"~:checkbox\",536870913],[124,\"^1\",\"~:coll\",536870913],[125,\"^1\",\"~:map\",536870913],[126,\"^1\",\"^3\",536870913],[127,\"^1\",\"^6\",536870913],[128,\"^1\",\"^7\",536870913],[129,\"^1\",\"^2\",536870913],[130,\"^1\",\"~:node\",536870913],[131,\"^1\",\"~:string\",536870913],[132,\"^1\",\"^9\",536870913],[133,\"^1\",\"^9\",536870913]]]", + "addresses": null + }, + { + "addr": 1000284, + "content": "[\"^ \",\"~:keys\",[[134,\"~:logseq.property/type\",\"~:raw-number\",536870913],[135,\"^1\",\"^2\",536870913],[136,\"^1\",\"^2\",536870913],[137,\"^1\",\"~:string\",536870913],[138,\"^1\",\"^2\",536870913],[139,\"^1\",\"~:map\",536870913],[141,\"^1\",\"^4\",536870913],[142,\"^1\",\"~:keyword\",536870913],[143,\"^1\",\"~:datetime\",536870913],[144,\"^1\",\"^4\",536870913],[145,\"^1\",\"^3\",536870913],[146,\"^1\",\"^3\",536870913],[147,\"^1\",\"^3\",536870913],[148,\"^1\",\"~:checkbox\",536870913],[150,\"^1\",\"~:entity\",536870913],[151,\"^1\",\"~:property\",536870913],[152,\"^1\",\"^8\",536870913],[153,\"^1\",\"~:any\",536870913],[154,\"^1\",\"^8\",536870913],[155,\"^1\",\"^6\",536870913],[156,\"^1\",\"^8\",536870913],[157,\"^1\",\"~:node\",536870913],[158,\"^1\",\"^;\",536870913],[159,\"^1\",\"^3\",536870913],[160,\"^1\",\"^3\",536870913],[161,\"^1\",\"^;\",536870913],[162,\"^1\",\"^3\",536870913],[164,\"^1\",\"^;\",536870913],[165,\"^1\",\"~:class\",536870913],[166,\"^1\",\"^4\",536870913],[74,\"~:logseq.property/ui-position\",\"~:block-left\",536870913]]]", + "addresses": null + }, + { + "addr": 1000285, + "content": "[\"^ \",\"~:keys\",[[81,\"~:logseq.property/ui-position\",\"~:block-left\",536870913],[86,\"^1\",\"~:block-below\",536870913],[88,\"^1\",\"^3\",536870913],[106,\"^1\",\"^3\",536870913],[91,\"~:logseq.property/value\",1,536870913],[21,\"~:logseq.property/view-context\",\"~:page\",536870913],[34,\"^5\",\"~:block\",536870913],[36,\"^5\",\"^7\",536870913],[37,\"^5\",\"~:property\",536870913],[38,\"^5\",\"^8\",536870913],[39,\"^5\",\"~:class\",536870913],[41,\"^5\",\"~:never\",536870913],[42,\"^5\",\"^9\",536870913],[43,\"^5\",\"^9\",536870913],[45,\"^5\",\"^8\",536870913],[47,\"^5\",\"^9\",536870913]]]", + "addresses": null + }, + { + "addr": 1000286, + "content": "[\"^ \",\"~:keys\",[[48,\"~:logseq.property/view-context\",\"~:block\",536870913],[49,\"^1\",\"~:page\",536870913],[71,\"^1\",\"~:never\",536870913],[72,\"^1\",\"^4\",536870913],[108,\"^1\",\"^3\",536870913],[109,\"^1\",\"^3\",536870913],[110,\"^1\",\"^3\",536870913],[148,\"^1\",\"~:property\",536870913],[2,\"~:logseq.property.class/extends\",1,536870913],[3,\"^6\",1,536870913],[4,\"^6\",1,536870913],[5,\"^6\",1,536870913],[167,\"^6\",4,536870913],[168,\"^6\",4,536870913],[169,\"^6\",1,536870913],[170,\"^6\",1,536870913]]]", + "addresses": null + }, + { + "addr": 1000287, + "content": "[\"^ \",\"~:keys\",[[171,\"~:logseq.property.class/extends\",1,536870913],[172,\"^1\",1,536870913],[173,\"^1\",1,536870913],[174,\"^1\",172,536870913],[175,\"^1\",1,536870913],[176,\"^1\",1,536870913],[177,\"^1\",1,536870913],[178,\"^1\",1,536870913],[179,\"^1\",1,536870913],[170,\"~:logseq.property.class/hide-from-node\",true,536870913],[171,\"^2\",true,536870913],[175,\"^2\",true,536870913],[176,\"^2\",true,536870913],[177,\"^2\",true,536870913],[178,\"^2\",true,536870913],[179,\"^2\",true,536870913],[5,\"~:logseq.property.class/properties\",165,536870913],[169,\"^3\",74,536870913]]]", + "addresses": null + }, + { + "addr": 1000288, + "content": "[\"^ \",\"~:keys\",[[169,\"~:logseq.property.class/properties\",81,536870913],[169,\"^1\",86,536870913],[169,\"^1\",88,536870913],[170,\"^1\",68,536870913],[172,\"^1\",48,536870913],[173,\"^1\",143,536870913],[173,\"^1\",144,536870913],[175,\"^1\",131,536870913],[175,\"^1\",134,536870913],[175,\"^1\",137,536870913],[176,\"^1\",34,536870913],[176,\"^1\",36,536870913],[177,\"^1\",34,536870913],[178,\"^1\",34,536870913],[179,\"^1\",53,536870913],[179,\"^1\",54,536870913],[179,\"^1\",55,536870913],[179,\"^1\",56,536870913],[179,\"^1\",62,536870913],[179,\"^1\",63,536870913],[179,\"^1\",64,536870913],[167,\"~:logseq.property.journal/title-format\",\"MMM do, yyyy\",536870913],[175,\"~:logseq.property.view/type\",114,536870913]]]", + "addresses": null + }, + { + "addr": 1000289, + "content": "[\"^ \",\"~:keys\",[[65,\"~:logseq.property/hide?\",true,536870913],[117,\"^1\",true,536870913],[133,\"^1\",true,536870913],[152,\"^1\",true,536870913],[77,\"~:logseq.property/icon\",[\"^ \",\"~:type\",\"~:tabler-icon\",\"~:id\",\"InProgress50\"],536870913],[25,\"~:logseq.property/public?\",false,536870913],[42,\"^6\",true,536870913],[100,\"^6\",false,536870913],[125,\"^6\",false,536870913],[142,\"^6\",false,536870913],[19,\"~:logseq.property/type\",\"~:entity\",536870913],[35,\"^7\",\"~:default\",536870913],[55,\"^7\",\"~:keyword\",536870913],[86,\"^7\",\"~:datetime\",536870913],[117,\"^7\",\"~:property\",536870913],[133,\"^7\",\"~:string\",536870913],[74,\"~:logseq.property/ui-position\",\"~:block-left\",536870913],[47,\"~:logseq.property/view-context\",\"~:class\",536870913],[170,\"~:logseq.property.class/extends\",1,536870913],[169,\"~:logseq.property.class/properties\",74,536870913],[175,\"~:logseq.property.view/type\",114,536870913]]]", + "addresses": "[1000268,1000269,1000270,1000271,1000272,1000273,1000274,1000275,1000276,1000277,1000278,1000279,1000280,1000281,1000282,1000283,1000284,1000285,1000286,1000287,1000288]" + }, + { + "addr": 1000290, + "content": "[\"^ \",\"~:keys\",[[87,\"~:block/page\",86,536870913],[14,\"~:block/uuid\",\"~u00000002-6836-5746-0000-000000000000\",536870913],[14,\"~:db/index\",true,536870913],[32,\"~:logseq.property/hide?\",true,536870913],[175,\"~:logseq.property.view/type\",114,536870913]]]", + "addresses": "[1000179,1000212,1000241,1000267,1000289]" + }, + { + "addr": 1000291, + "content": "[\"^ \",\"~:keys\",[[95,\"~:block/closed-value-property\",92,536870913],[96,\"^1\",92,536870913],[97,\"^1\",92,536870913],[98,\"^1\",92,536870913],[101,\"^1\",100,536870913],[102,\"^1\",100,536870913],[103,\"^1\",100,536870913],[112,\"^1\",111,536870913],[113,\"^1\",111,536870913],[114,\"^1\",111,536870913],[12,\"~:block/created-at\",1786627375714,536870913],[13,\"^2\",1786627375714,536870913],[14,\"^2\",1786627375714,536870913],[15,\"^2\",1786627375714,536870913],[16,\"^2\",1786627375715,536870913],[17,\"^2\",1786627375715,536870913],[18,\"^2\",1786627375715,536870913],[19,\"^2\",1786627375715,536870913],[20,\"^2\",1786627375716,536870913],[21,\"^2\",1786627375716,536870913],[22,\"^2\",1786627375716,536870913],[23,\"^2\",1786627375716,536870913],[24,\"^2\",1786627375716,536870913],[25,\"^2\",1786627375716,536870913],[26,\"^2\",1786627375716,536870913],[27,\"^2\",1786627375716,536870913]]]", + "addresses": null + }, + { + "addr": 1000292, + "content": "[\"^ \",\"~:keys\",[[28,\"~:block/created-at\",1786627375716,536870913],[29,\"^1\",1786627375717,536870913],[30,\"^1\",1786627375717,536870913],[31,\"^1\",1786627375717,536870913],[32,\"^1\",1786627375717,536870913],[33,\"^1\",1786627375717,536870913],[34,\"^1\",1786627375717,536870913],[35,\"^1\",1786627375717,536870913],[36,\"^1\",1786627375717,536870913],[37,\"^1\",1786627375717,536870913],[38,\"^1\",1786627375717,536870913],[39,\"^1\",1786627375717,536870913],[40,\"^1\",1786627375717,536870913],[41,\"^1\",1786627375717,536870913],[42,\"^1\",1786627375717,536870913],[43,\"^1\",1786627375717,536870913]]]", + "addresses": null + }, + { + "addr": 1000293, + "content": "[\"^ \",\"~:keys\",[[44,\"~:block/created-at\",1786627375717,536870913],[45,\"^1\",1786627375717,536870913],[46,\"^1\",1786627375718,536870913],[47,\"^1\",1786627375718,536870913],[48,\"^1\",1786627375718,536870913],[49,\"^1\",1786627375718,536870913],[50,\"^1\",1786627375718,536870913],[51,\"^1\",1786627375718,536870913],[52,\"^1\",1786627375718,536870913],[53,\"^1\",1786627375718,536870913],[54,\"^1\",1786627375718,536870913],[55,\"^1\",1786627375718,536870913],[56,\"^1\",1786627375718,536870913],[57,\"^1\",1786627375718,536870913],[58,\"^1\",1786627375718,536870913],[59,\"^1\",1786627375718,536870913]]]", + "addresses": null + }, + { + "addr": 1000294, + "content": "[\"^ \",\"~:keys\",[[60,\"~:block/created-at\",1786627375718,536870913],[61,\"^1\",1786627375718,536870913],[62,\"^1\",1786627375719,536870913],[63,\"^1\",1786627375719,536870913],[64,\"^1\",1786627375719,536870913],[65,\"^1\",1786627375719,536870913],[66,\"^1\",1786627375719,536870913],[67,\"^1\",1786627375719,536870913],[68,\"^1\",1786627375719,536870913],[69,\"^1\",1786627375719,536870913],[70,\"^1\",1786627375719,536870913],[71,\"^1\",1786627375719,536870913],[72,\"^1\",1786627375719,536870913],[73,\"^1\",1786627375719,536870913],[74,\"^1\",1786627375719,536870913],[75,\"^1\",1786627375719,536870913]]]", + "addresses": null + }, + { + "addr": 1000295, + "content": "[\"^ \",\"~:keys\",[[76,\"~:block/created-at\",1786627375719,536870913],[77,\"^1\",1786627375719,536870913],[78,\"^1\",1786627375719,536870913],[79,\"^1\",1786627375719,536870913],[80,\"^1\",1786627375720,536870913],[81,\"^1\",1786627375720,536870913],[82,\"^1\",1786627375720,536870913],[83,\"^1\",1786627375720,536870913],[84,\"^1\",1786627375720,536870913],[85,\"^1\",1786627375720,536870913],[86,\"^1\",1786627375720,536870913],[87,\"^1\",1786627375720,536870913],[88,\"^1\",1786627375720,536870913],[89,\"^1\",1786627375720,536870913],[90,\"^1\",1786627375720,536870913],[91,\"^1\",1786627375720,536870913]]]", + "addresses": null + }, + { + "addr": 1000296, + "content": "[\"^ \",\"~:keys\",[[92,\"~:block/created-at\",1786627375720,536870913],[93,\"^1\",1786627375720,536870913],[94,\"^1\",1786627375720,536870913],[95,\"^1\",1786627375721,536870913],[96,\"^1\",1786627375721,536870913],[97,\"^1\",1786627375721,536870913],[98,\"^1\",1786627375721,536870913],[99,\"^1\",1786627375721,536870913],[100,\"^1\",1786627375721,536870913],[101,\"^1\",1786627375721,536870913],[102,\"^1\",1786627375721,536870913],[103,\"^1\",1786627375721,536870913],[104,\"^1\",1786627375721,536870913],[105,\"^1\",1786627375721,536870913],[106,\"^1\",1786627375721,536870913],[107,\"^1\",1786627375721,536870913]]]", + "addresses": null + }, + { + "addr": 1000297, + "content": "[\"^ \",\"~:keys\",[[108,\"~:block/created-at\",1786627375721,536870913],[109,\"^1\",1786627375721,536870913],[110,\"^1\",1786627375721,536870913],[111,\"^1\",1786627375721,536870913],[112,\"^1\",1786627375721,536870913],[113,\"^1\",1786627375721,536870913],[114,\"^1\",1786627375721,536870913],[115,\"^1\",1786627375721,536870913],[116,\"^1\",1786627375721,536870913],[117,\"^1\",1786627375721,536870913],[118,\"^1\",1786627375722,536870913],[119,\"^1\",1786627375722,536870913],[120,\"^1\",1786627375722,536870913],[121,\"^1\",1786627375722,536870913],[122,\"^1\",1786627375722,536870913],[123,\"^1\",1786627375722,536870913]]]", + "addresses": null + }, + { + "addr": 1000298, + "content": "[\"^ \",\"~:keys\",[[124,\"~:block/created-at\",1786627375722,536870913],[125,\"^1\",1786627375722,536870913],[126,\"^1\",1786627375722,536870913],[127,\"^1\",1786627375722,536870913],[128,\"^1\",1786627375722,536870913],[129,\"^1\",1786627375722,536870913],[130,\"^1\",1786627375722,536870913],[131,\"^1\",1786627375722,536870913],[132,\"^1\",1786627375722,536870913],[133,\"^1\",1786627375722,536870913],[134,\"^1\",1786627375722,536870913],[135,\"^1\",1786627375722,536870913],[136,\"^1\",1786627375722,536870913],[137,\"^1\",1786627375722,536870913],[138,\"^1\",1786627375722,536870913],[139,\"^1\",1786627375722,536870913]]]", + "addresses": null + }, + { + "addr": 1000299, + "content": "[\"^ \",\"~:keys\",[[140,\"~:block/created-at\",1786627375722,536870913],[141,\"^1\",1786627375722,536870913],[142,\"^1\",1786627375722,536870913],[143,\"^1\",1786627375723,536870913],[144,\"^1\",1786627375723,536870913],[145,\"^1\",1786627375723,536870913],[146,\"^1\",1786627375723,536870913],[147,\"^1\",1786627375723,536870913],[148,\"^1\",1786627375723,536870913],[149,\"^1\",1786627375723,536870913],[150,\"^1\",1786627375723,536870913],[151,\"^1\",1786627375723,536870913],[152,\"^1\",1786627375723,536870913],[153,\"^1\",1786627375723,536870913],[154,\"^1\",1786627375723,536870913],[155,\"^1\",1786627375723,536870913]]]", + "addresses": null + }, + { + "addr": 1000300, + "content": "[\"^ \",\"~:keys\",[[156,\"~:block/created-at\",1786627375723,536870913],[157,\"^1\",1786627375723,536870913],[158,\"^1\",1786627375723,536870913],[159,\"^1\",1786627375723,536870913],[160,\"^1\",1786627375723,536870913],[161,\"^1\",1786627375724,536870913],[162,\"^1\",1786627375724,536870913],[163,\"^1\",1786627375724,536870913],[164,\"^1\",1786627375724,536870913],[165,\"^1\",1786627375724,536870913],[166,\"^1\",1786627375724,536870913],[1,\"^1\",1786627375725,536870913],[2,\"^1\",1786627375725,536870913],[3,\"^1\",1786627375725,536870913],[4,\"^1\",1786627375725,536870913],[5,\"^1\",1786627375725,536870913]]]", + "addresses": null + }, + { + "addr": 1000301, + "content": "[\"^ \",\"~:keys\",[[167,\"~:block/created-at\",1786627375725,536870913],[168,\"^1\",1786627375725,536870913],[169,\"^1\",1786627375725,536870913],[170,\"^1\",1786627375725,536870913],[171,\"^1\",1786627375725,536870913],[172,\"^1\",1786627375725,536870913],[173,\"^1\",1786627375725,536870913],[174,\"^1\",1786627375725,536870913],[175,\"^1\",1786627375725,536870913],[176,\"^1\",1786627375725,536870913],[177,\"^1\",1786627375725,536870913],[178,\"^1\",1786627375725,536870913],[179,\"^1\",1786627375725,536870913],[185,\"^1\",1786627375725,536870913],[186,\"^1\",1786627375725,536870913],[187,\"^1\",1786627375725,536870913],[188,\"^1\",1786627375725,536870913],[189,\"^1\",1786627375725,536870913],[190,\"^1\",1786627375725,536870913],[189,\"~:block/name\",\"$$$favorites\",536870913],[188,\"^2\",\"$$$views\",536870913],[162,\"^2\",\"agent session id\",536870913],[21,\"^2\",\"alias\",536870913],[56,\"^2\",\"annotation color\",536870913]]]", + "addresses": null + }, + { + "addr": 1000302, + "content": "[\"^ \",\"~:keys\",[[64,\"~:block/name\",\"annotation data\",536870913],[63,\"^1\",\"annotation image\",536870913],[62,\"^1\",\"annotation page\",536870913],[55,\"^1\",\"annotation type\",536870913],[165,\"^1\",\"apply template to tags\",536870913],[53,\"^1\",\"asset\",536870913],[175,\"^1\",\"asset\",536870913],[142,\"^1\",\"asset alignment\",536870913],[141,\"^1\",\"asset resize metadata\",536870913],[106,\"^1\",\"assignee\",536870913],[51,\"^1\",\"background color\",536870913],[42,\"^1\",\"bidirectional property title\",536870913],[13,\"^1\",\"built in?\",536870913],[173,\"^1\",\"card\",536870913],[174,\"^1\",\"cards\",536870913],[70,\"^1\",\"choice checkbox state\",536870913],[71,\"^1\",\"choice classes\",536870913],[72,\"^1\",\"choice exclusions\",536870913],[30,\"^1\",\"closed value property\",536870913],[176,\"^1\",\"code\",536870913],[36,\"^1\",\"code mode\",536870913],[171,\"^1\",\"comment\",536870913],[68,\"^1\",\"commented blocks\",536870913],[170,\"^1\",\"comments\",536870913],[186,\"^1\",\"contents\",536870913],[15,\"^1\",\"created from property\",536870913],[86,\"^1\",\"deadline\",536870913],[37,\"^1\",\"default value\",536870913]]]", + "addresses": null + }, + { + "addr": 1000303, + "content": "[\"^ \",\"~:keys\",[[155,\"~:block/name\",\"deleted at\",536870913],[156,\"^1\",\"deleted by\",536870913],[35,\"^1\",\"description\",536870913],[143,\"^1\",\"due\",536870913],[43,\"^1\",\"enable bidirectional properties\",536870913],[148,\"^1\",\"enable property history\",536870913],[110,\"^1\",\"excluded from graph view?\",536870913],[67,\"^1\",\"excluded references\",536870913],[39,\"^1\",\"extends\",536870913],[133,\"^1\",\"external file name\",536870913],[132,\"^1\",\"external url\",536870913],[137,\"^1\",\"file checksum\",536870913],[139,\"^1\",\"file remote metadata\",536870913],[134,\"^1\",\"file size\",536870913],[131,\"^1\",\"file type\",536870913],[117,\"^1\",\"gallery asset property\",536870913],[121,\"^1\",\"gallery card height\",536870913],[119,\"^1\",\"gallery card size\",536870913]]]", + "addresses": null + }, + { + "addr": 1000304, + "content": "[\"^ \",\"~:keys\",[[120,\"~:block/name\",\"gallery card width\",536870913],[118,\"^1\",\"gallery display properties\",536870913],[52,\"^1\",\"heading\",536870913],[45,\"^1\",\"hide empty value\",536870913],[47,\"^1\",\"hide from node\",536870913],[14,\"^1\",\"hide this property or page\",536870913],[150,\"^1\",\"history block\",536870913],[151,\"^1\",\"history property\",536870913],[153,\"^1\",\"history scalar value\",536870913],[152,\"^1\",\"history value\",536870913],[107,\"^1\",\"icon\",536870913],[136,\"^1\",\"image height\",536870913],[135,\"^1\",\"image width\",536870913],[66,\"^1\",\"included references\",536870913],[167,\"^1\",\"journal\",536870913],[31,\"^1\",\"journal date\",536870913],[138,\"^1\",\"last visit page\",536870913],[185,\"^1\",\"library\",536870913],[65,\"^1\",\"list type\",536870913],[54,\"^1\",\"ls-type\",536870913],[178,\"^1\",\"math\",536870913],[25,\"^1\",\"node collapsed?\",536870913],[32,\"^1\",\"node created at\",536870913]]]", + "addresses": null + }, + { + "addr": 1000305, + "content": "[\"^ \",\"~:keys\",[[154,\"~:block/name\",\"node created by\",536870913],[34,\"^1\",\"node display type\",536870913],[28,\"^1\",\"node links to\",536870913],[24,\"^1\",\"node order\",536870913],[26,\"^1\",\"node page\",536870913],[23,\"^1\",\"node parent\",536870913],[27,\"^1\",\"node references\",536870913],[99,\"^1\",\"node repeats?\",536870913],[29,\"^1\",\"node title\",536870913],[33,\"^1\",\"node updated at\",536870913],[38,\"^1\",\"non ref type default value\",536870913],[4,\"^1\",\"page\",536870913],[49,\"^1\",\"page tags\",536870913],[179,\"^1\",\"pdf annotation\",536870913],[81,\"^1\",\"priority\",536870913],[73,\"^1\",\"properties displayed as checkbox\",536870913],[3,\"^1\",\"property\",536870913],[19,\"^1\",\"property classes\",536870913],[18,\"^1\",\"property position\",536870913],[16,\"^1\",\"property public?\",536870913]]]", + "addresses": null + }, + { + "addr": 1000306, + "content": "[\"^ \",\"~:keys\",[[12,\"~:block/name\",\"property type\",536870913],[20,\"^1\",\"property value\",536870913],[17,\"^1\",\"property view context\",536870913],[109,\"^1\",\"published url\",536870913],[108,\"^1\",\"publishing public?\",536870913],[48,\"^1\",\"query\",536870913],[172,\"^1\",\"query\",536870913],[187,\"^1\",\"quick add\",536870913],[177,\"^1\",\"quote\",536870913],[160,\"^1\",\"reaction emoji\",536870913],[161,\"^1\",\"reaction target\",536870913],[190,\"^1\",\"recycle\",536870913],[159,\"^1\",\"recycle original order\",536870913],[158,\"^1\",\"recycle original page\",536870913],[157,\"^1\",\"recycle original parent\",536870913],[166,\"^1\",\"reference to large block title stored in remote object storage\",536870913],[105,\"^1\",\"repeating checked property\",536870913]]]", + "addresses": null + }, + { + "addr": 1000307, + "content": "[\"^ \",\"~:keys\",[[90,\"~:block/name\",\"repeating recur frequency\",536870913],[92,\"^1\",\"repeating recur unit\",536870913],[104,\"^1\",\"repeating temporal property\",536870913],[100,\"^1\",\"repeating type\",536870913],[1,\"^1\",\"root tag\",536870913],[88,\"^1\",\"scheduled\",536870913],[144,\"^1\",\"state\",536870913],[74,\"^1\",\"status\",536870913],[129,\"^1\",\"table view pinned columns\",536870913],[2,\"^1\",\"tag\",536870913],[41,\"^1\",\"tag properties\",536870913],[22,\"^1\",\"tags\",536870913],[169,\"^1\",\"task\",536870913],[5,\"^1\",\"template\",536870913],[130,\"^1\",\"this view belongs to\",536870913],[69,\"^1\",\"title format\",536870913]]]", + "addresses": null + }, + { + "addr": 1000308, + "content": "[\"^ \",\"~:keys\",[[164,\"~:block/name\",\"used template\",536870913],[147,\"^1\",\"user avatar\",536870913],[146,\"^1\",\"user email\",536870913],[145,\"^1\",\"user name\",536870913],[128,\"^1\",\"view columns settings\",536870913],[115,\"^1\",\"view feature type\",536870913],[125,\"^1\",\"view filters\",536870913],[116,\"^1\",\"view group by property\",536870913],[126,\"^1\",\"view hidden columns\",536870913],[127,\"^1\",\"view ordered columns\",536870913],[122,\"^1\",\"view sort groups by\",536870913],[123,\"^1\",\"view sort groups desc\",536870913],[124,\"^1\",\"view sorting\",536870913],[111,\"^1\",\"view type\",536870913],[168,\"^1\",\"whiteboard\",536870913],[12,\"~:block/order\",\"a0\",536870913],[13,\"^2\",\"a1\",536870913],[14,\"^2\",\"a2\",536870913],[15,\"^2\",\"a3\",536870913],[16,\"^2\",\"a4\",536870913],[17,\"^2\",\"a5\",536870913],[18,\"^2\",\"a6\",536870913],[19,\"^2\",\"a7\",536870913],[20,\"^2\",\"a8\",536870913],[21,\"^2\",\"a9\",536870913],[22,\"^2\",\"aA\",536870913],[23,\"^2\",\"aB\",536870913]]]", + "addresses": null + }, + { + "addr": 1000309, + "content": "[\"^ \",\"~:keys\",[[24,\"~:block/order\",\"aC\",536870913],[25,\"^1\",\"aD\",536870913],[26,\"^1\",\"aE\",536870913],[27,\"^1\",\"aF\",536870913],[28,\"^1\",\"aG\",536870913],[29,\"^1\",\"aH\",536870913],[30,\"^1\",\"aI\",536870913],[31,\"^1\",\"aJ\",536870913],[32,\"^1\",\"aK\",536870913],[33,\"^1\",\"aL\",536870913],[34,\"^1\",\"aM\",536870913],[35,\"^1\",\"aN\",536870913],[36,\"^1\",\"aO\",536870913],[37,\"^1\",\"aP\",536870913],[38,\"^1\",\"aQ\",536870913],[39,\"^1\",\"aR\",536870913]]]", + "addresses": null + }, + { + "addr": 1000310, + "content": "[\"^ \",\"~:keys\",[[40,\"~:block/order\",\"aS\",536870913],[41,\"^1\",\"aT\",536870913],[42,\"^1\",\"aU\",536870913],[43,\"^1\",\"aV\",536870913],[44,\"^1\",\"aW\",536870913],[45,\"^1\",\"aX\",536870913],[46,\"^1\",\"aY\",536870913],[47,\"^1\",\"aZ\",536870913],[48,\"^1\",\"aa\",536870913],[49,\"^1\",\"ab\",536870913],[50,\"^1\",\"ac\",536870913],[51,\"^1\",\"ad\",536870913],[52,\"^1\",\"ae\",536870913],[53,\"^1\",\"af\",536870913],[54,\"^1\",\"ag\",536870913],[55,\"^1\",\"ah\",536870913]]]", + "addresses": null + }, + { + "addr": 1000311, + "content": "[\"^ \",\"~:keys\",[[56,\"~:block/order\",\"ai\",536870913],[57,\"^1\",\"aj\",536870913],[58,\"^1\",\"ak\",536870913],[59,\"^1\",\"al\",536870913],[60,\"^1\",\"am\",536870913],[61,\"^1\",\"an\",536870913],[62,\"^1\",\"ao\",536870913],[63,\"^1\",\"ap\",536870913],[64,\"^1\",\"aq\",536870913],[65,\"^1\",\"ar\",536870913],[66,\"^1\",\"as\",536870913],[67,\"^1\",\"at\",536870913],[68,\"^1\",\"au\",536870913],[69,\"^1\",\"av\",536870913],[70,\"^1\",\"aw\",536870913],[71,\"^1\",\"ax\",536870913]]]", + "addresses": null + }, + { + "addr": 1000312, + "content": "[\"^ \",\"~:keys\",[[94,\"~:block/closed-value-property\",92,536870913],[27,\"~:block/created-at\",1786627375716,536870913],[43,\"^2\",1786627375717,536870913],[59,\"^2\",1786627375718,536870913],[75,\"^2\",1786627375719,536870913],[91,\"^2\",1786627375720,536870913],[107,\"^2\",1786627375721,536870913],[123,\"^2\",1786627375722,536870913],[139,\"^2\",1786627375722,536870913],[155,\"^2\",1786627375723,536870913],[5,\"^2\",1786627375725,536870913],[56,\"~:block/name\",\"annotation color\",536870913],[37,\"^3\",\"default value\",536870913],[119,\"^3\",\"gallery card size\",536870913],[32,\"^3\",\"node created at\",536870913],[16,\"^3\",\"property public?\",536870913],[105,\"^3\",\"repeating checked property\",536870913],[69,\"^3\",\"title format\",536870913],[23,\"~:block/order\",\"aB\",536870913],[39,\"^4\",\"aR\",536870913],[55,\"^4\",\"ah\",536870913],[71,\"^4\",\"ax\",536870913]]]", + "addresses": "[1000003,1000291,1000292,1000293,1000294,1000295,1000296,1000297,1000298,1000299,1000300,1000301,1000302,1000303,1000304,1000305,1000306,1000307,1000308,1000309,1000310,1000311]" + }, + { + "addr": 1000313, + "content": "[\"^ \",\"~:keys\",[[72,\"~:block/order\",\"ay\",536870913],[73,\"^1\",\"az\",536870913],[74,\"^1\",\"b00\",536870913],[75,\"^1\",\"b01\",536870913],[76,\"^1\",\"b02\",536870913],[77,\"^1\",\"b03\",536870913],[78,\"^1\",\"b04\",536870913],[79,\"^1\",\"b05\",536870913],[80,\"^1\",\"b06\",536870913],[81,\"^1\",\"b07\",536870913],[82,\"^1\",\"b08\",536870913],[83,\"^1\",\"b09\",536870913],[84,\"^1\",\"b0A\",536870913],[85,\"^1\",\"b0B\",536870913],[86,\"^1\",\"b0C\",536870913],[87,\"^1\",\"b0D\",536870913]]]", + "addresses": null + }, + { + "addr": 1000314, + "content": "[\"^ \",\"~:keys\",[[88,\"~:block/order\",\"b0E\",536870913],[89,\"^1\",\"b0F\",536870913],[90,\"^1\",\"b0G\",536870913],[91,\"^1\",\"b0H\",536870913],[92,\"^1\",\"b0I\",536870913],[93,\"^1\",\"b0J\",536870913],[94,\"^1\",\"b0K\",536870913],[95,\"^1\",\"b0L\",536870913],[96,\"^1\",\"b0M\",536870913],[97,\"^1\",\"b0N\",536870913],[98,\"^1\",\"b0O\",536870913],[99,\"^1\",\"b0P\",536870913],[100,\"^1\",\"b0Q\",536870913],[101,\"^1\",\"b0R\",536870913],[102,\"^1\",\"b0S\",536870913],[103,\"^1\",\"b0T\",536870913]]]", + "addresses": null + }, + { + "addr": 1000315, + "content": "[\"^ \",\"~:keys\",[[104,\"~:block/order\",\"b0U\",536870913],[105,\"^1\",\"b0V\",536870913],[106,\"^1\",\"b0W\",536870913],[107,\"^1\",\"b0X\",536870913],[108,\"^1\",\"b0Y\",536870913],[109,\"^1\",\"b0Z\",536870913],[110,\"^1\",\"b0a\",536870913],[111,\"^1\",\"b0b\",536870913],[112,\"^1\",\"b0c\",536870913],[113,\"^1\",\"b0d\",536870913],[114,\"^1\",\"b0e\",536870913],[115,\"^1\",\"b0f\",536870913],[116,\"^1\",\"b0g\",536870913],[117,\"^1\",\"b0h\",536870913],[118,\"^1\",\"b0i\",536870913],[119,\"^1\",\"b0j\",536870913]]]", + "addresses": null + }, + { + "addr": 1000316, + "content": "[\"^ \",\"~:keys\",[[120,\"~:block/order\",\"b0k\",536870913],[121,\"^1\",\"b0l\",536870913],[122,\"^1\",\"b0m\",536870913],[123,\"^1\",\"b0n\",536870913],[124,\"^1\",\"b0o\",536870913],[125,\"^1\",\"b0p\",536870913],[126,\"^1\",\"b0q\",536870913],[127,\"^1\",\"b0r\",536870913],[128,\"^1\",\"b0s\",536870913],[129,\"^1\",\"b0t\",536870913],[130,\"^1\",\"b0u\",536870913],[131,\"^1\",\"b0v\",536870913],[132,\"^1\",\"b0w\",536870913],[133,\"^1\",\"b0x\",536870913],[134,\"^1\",\"b0y\",536870913],[135,\"^1\",\"b0z\",536870913]]]", + "addresses": null + }, + { + "addr": 1000317, + "content": "[\"^ \",\"~:keys\",[[136,\"~:block/order\",\"b10\",536870913],[137,\"^1\",\"b11\",536870913],[138,\"^1\",\"b12\",536870913],[139,\"^1\",\"b13\",536870913],[140,\"^1\",\"b14\",536870913],[141,\"^1\",\"b15\",536870913],[142,\"^1\",\"b16\",536870913],[143,\"^1\",\"b17\",536870913],[144,\"^1\",\"b18\",536870913],[145,\"^1\",\"b19\",536870913],[146,\"^1\",\"b1A\",536870913],[147,\"^1\",\"b1B\",536870913],[148,\"^1\",\"b1C\",536870913],[149,\"^1\",\"b1D\",536870913],[150,\"^1\",\"b1E\",536870913],[151,\"^1\",\"b1F\",536870913]]]", + "addresses": null + }, + { + "addr": 1000318, + "content": "[\"^ \",\"~:keys\",[[152,\"~:block/order\",\"b1G\",536870913],[153,\"^1\",\"b1H\",536870913],[154,\"^1\",\"b1I\",536870913],[155,\"^1\",\"b1J\",536870913],[156,\"^1\",\"b1K\",536870913],[157,\"^1\",\"b1L\",536870913],[158,\"^1\",\"b1M\",536870913],[159,\"^1\",\"b1N\",536870913],[160,\"^1\",\"b1O\",536870913],[161,\"^1\",\"b1P\",536870913],[162,\"^1\",\"b1Q\",536870913],[163,\"^1\",\"b1R\",536870913],[164,\"^1\",\"b1S\",536870913],[165,\"^1\",\"b1T\",536870913],[166,\"^1\",\"b1U\",536870913],[40,\"~:block/page\",39,536870913],[44,\"^2\",43,536870913],[46,\"^2\",45,536870913],[50,\"^2\",49,536870913],[57,\"^2\",56,536870913]]]", + "addresses": null + }, + { + "addr": 1000319, + "content": "[\"^ \",\"~:keys\",[[58,\"~:block/page\",56,536870913],[59,\"^1\",56,536870913],[60,\"^1\",56,536870913],[61,\"^1\",56,536870913],[75,\"^1\",74,536870913],[76,\"^1\",74,536870913],[77,\"^1\",74,536870913],[78,\"^1\",74,536870913],[79,\"^1\",74,536870913],[80,\"^1\",74,536870913],[82,\"^1\",81,536870913],[83,\"^1\",81,536870913],[84,\"^1\",81,536870913],[85,\"^1\",81,536870913],[87,\"^1\",86,536870913],[89,\"^1\",88,536870913]]]", + "addresses": null + }, + { + "addr": 1000320, + "content": "[\"^ \",\"~:keys\",[[91,\"~:block/page\",90,536870913],[93,\"^1\",92,536870913],[94,\"^1\",92,536870913],[95,\"^1\",92,536870913],[96,\"^1\",92,536870913],[97,\"^1\",92,536870913],[98,\"^1\",92,536870913],[101,\"^1\",100,536870913],[102,\"^1\",100,536870913],[103,\"^1\",100,536870913],[112,\"^1\",111,536870913],[113,\"^1\",111,536870913],[114,\"^1\",111,536870913],[140,\"^1\",139,536870913],[149,\"^1\",148,536870913],[163,\"^1\",162,536870913],[40,\"~:block/parent\",39,536870913],[44,\"^2\",43,536870913],[46,\"^2\",45,536870913],[50,\"^2\",49,536870913]]]", + "addresses": null + }, + { + "addr": 1000321, + "content": "[\"^ \",\"~:keys\",[[57,\"~:block/parent\",56,536870913],[58,\"^1\",56,536870913],[59,\"^1\",56,536870913],[60,\"^1\",56,536870913],[61,\"^1\",56,536870913],[75,\"^1\",74,536870913],[76,\"^1\",74,536870913],[77,\"^1\",74,536870913],[78,\"^1\",74,536870913],[79,\"^1\",74,536870913],[80,\"^1\",74,536870913],[82,\"^1\",81,536870913],[83,\"^1\",81,536870913],[84,\"^1\",81,536870913],[85,\"^1\",81,536870913],[87,\"^1\",86,536870913]]]", + "addresses": null + }, + { + "addr": 1000322, + "content": "[\"^ \",\"~:keys\",[[89,\"~:block/parent\",88,536870913],[91,\"^1\",90,536870913],[93,\"^1\",92,536870913],[94,\"^1\",92,536870913],[95,\"^1\",92,536870913],[96,\"^1\",92,536870913],[97,\"^1\",92,536870913],[98,\"^1\",92,536870913],[101,\"^1\",100,536870913],[102,\"^1\",100,536870913],[103,\"^1\",100,536870913],[112,\"^1\",111,536870913],[113,\"^1\",111,536870913],[114,\"^1\",111,536870913],[140,\"^1\",139,536870913],[149,\"^1\",148,536870913]]]", + "addresses": null + }, + { + "addr": 1000323, + "content": "[\"^ \",\"~:keys\",[[163,\"~:block/parent\",162,536870913],[1,\"~:block/tags\",2,536870913],[2,\"^2\",2,536870913],[3,\"^2\",2,536870913],[4,\"^2\",2,536870913],[5,\"^2\",2,536870913],[167,\"^2\",2,536870913],[168,\"^2\",2,536870913],[169,\"^2\",2,536870913],[170,\"^2\",2,536870913],[171,\"^2\",2,536870913],[172,\"^2\",2,536870913],[173,\"^2\",2,536870913],[174,\"^2\",2,536870913],[175,\"^2\",2,536870913],[176,\"^2\",2,536870913],[177,\"^2\",2,536870913],[178,\"^2\",2,536870913],[179,\"^2\",2,536870913],[12,\"^2\",3,536870913],[13,\"^2\",3,536870913],[14,\"^2\",3,536870913]]]", + "addresses": null + }, + { + "addr": 1000324, + "content": "[\"^ \",\"~:keys\",[[15,\"~:block/tags\",3,536870913],[16,\"^1\",3,536870913],[17,\"^1\",3,536870913],[18,\"^1\",3,536870913],[19,\"^1\",3,536870913],[20,\"^1\",3,536870913],[21,\"^1\",3,536870913],[22,\"^1\",3,536870913],[23,\"^1\",3,536870913],[24,\"^1\",3,536870913],[25,\"^1\",3,536870913],[26,\"^1\",3,536870913],[27,\"^1\",3,536870913],[28,\"^1\",3,536870913],[29,\"^1\",3,536870913],[30,\"^1\",3,536870913]]]", + "addresses": null + }, + { + "addr": 1000325, + "content": "[\"^ \",\"~:keys\",[[31,\"~:block/tags\",3,536870913],[32,\"^1\",3,536870913],[33,\"^1\",3,536870913],[34,\"^1\",3,536870913],[35,\"^1\",3,536870913],[36,\"^1\",3,536870913],[37,\"^1\",3,536870913],[38,\"^1\",3,536870913],[39,\"^1\",3,536870913],[41,\"^1\",3,536870913],[42,\"^1\",3,536870913],[43,\"^1\",3,536870913],[45,\"^1\",3,536870913],[47,\"^1\",3,536870913],[48,\"^1\",3,536870913],[49,\"^1\",3,536870913]]]", + "addresses": null + }, + { + "addr": 1000326, + "content": "[\"^ \",\"~:keys\",[[51,\"~:block/tags\",3,536870913],[52,\"^1\",3,536870913],[53,\"^1\",3,536870913],[54,\"^1\",3,536870913],[55,\"^1\",3,536870913],[56,\"^1\",3,536870913],[62,\"^1\",3,536870913],[63,\"^1\",3,536870913],[64,\"^1\",3,536870913],[65,\"^1\",3,536870913],[66,\"^1\",3,536870913],[67,\"^1\",3,536870913],[68,\"^1\",3,536870913],[69,\"^1\",3,536870913],[70,\"^1\",3,536870913],[71,\"^1\",3,536870913]]]", + "addresses": null + }, + { + "addr": 1000327, + "content": "[\"^ \",\"~:keys\",[[72,\"~:block/tags\",3,536870913],[73,\"^1\",3,536870913],[74,\"^1\",3,536870913],[81,\"^1\",3,536870913],[86,\"^1\",3,536870913],[88,\"^1\",3,536870913],[90,\"^1\",3,536870913],[92,\"^1\",3,536870913],[99,\"^1\",3,536870913],[100,\"^1\",3,536870913],[104,\"^1\",3,536870913],[105,\"^1\",3,536870913],[106,\"^1\",3,536870913],[107,\"^1\",3,536870913],[108,\"^1\",3,536870913],[109,\"^1\",3,536870913]]]", + "addresses": null + }, + { + "addr": 1000328, + "content": "[\"^ \",\"~:keys\",[[110,\"~:block/tags\",3,536870913],[111,\"^1\",3,536870913],[115,\"^1\",3,536870913],[116,\"^1\",3,536870913],[117,\"^1\",3,536870913],[118,\"^1\",3,536870913],[119,\"^1\",3,536870913],[120,\"^1\",3,536870913],[121,\"^1\",3,536870913],[122,\"^1\",3,536870913],[123,\"^1\",3,536870913],[124,\"^1\",3,536870913],[125,\"^1\",3,536870913],[126,\"^1\",3,536870913],[127,\"^1\",3,536870913],[128,\"^1\",3,536870913]]]", + "addresses": null + }, + { + "addr": 1000329, + "content": "[\"^ \",\"~:keys\",[[129,\"~:block/tags\",3,536870913],[130,\"^1\",3,536870913],[131,\"^1\",3,536870913],[132,\"^1\",3,536870913],[133,\"^1\",3,536870913],[134,\"^1\",3,536870913],[135,\"^1\",3,536870913],[136,\"^1\",3,536870913],[137,\"^1\",3,536870913],[138,\"^1\",3,536870913],[139,\"^1\",3,536870913],[141,\"^1\",3,536870913],[142,\"^1\",3,536870913],[143,\"^1\",3,536870913],[144,\"^1\",3,536870913],[145,\"^1\",3,536870913]]]", + "addresses": null + }, + { + "addr": 1000330, + "content": "[\"^ \",\"~:keys\",[[146,\"~:block/tags\",3,536870913],[147,\"^1\",3,536870913],[148,\"^1\",3,536870913],[150,\"^1\",3,536870913],[151,\"^1\",3,536870913],[152,\"^1\",3,536870913],[153,\"^1\",3,536870913],[154,\"^1\",3,536870913],[155,\"^1\",3,536870913],[156,\"^1\",3,536870913],[157,\"^1\",3,536870913],[158,\"^1\",3,536870913],[159,\"^1\",3,536870913],[160,\"^1\",3,536870913],[161,\"^1\",3,536870913],[162,\"^1\",3,536870913],[164,\"^1\",3,536870913],[165,\"^1\",3,536870913],[166,\"^1\",3,536870913],[185,\"^1\",4,536870913],[186,\"^1\",4,536870913],[187,\"^1\",4,536870913],[188,\"^1\",4,536870913],[189,\"^1\",4,536870913],[190,\"^1\",4,536870913],[189,\"~:block/title\",\"$$$favorites\",536870913],[188,\"^2\",\"$$$views\",536870913],[101,\"^2\",\"Advance from completion\",536870913]]]", + "addresses": null + }, + { + "addr": 1000331, + "content": "[\"^ \",\"~:keys\",[[102,\"~:block/title\",\"Advance from scheduled\",536870913],[103,\"^1\",\"Advance from scheduled, skip to future\",536870913],[162,\"^1\",\"Agent Session ID\",536870913],[21,\"^1\",\"Alias\",536870913],[56,\"^1\",\"Annotation color\",536870913],[64,\"^1\",\"Annotation data\",536870913],[63,\"^1\",\"Annotation image\",536870913],[62,\"^1\",\"Annotation page\",536870913],[55,\"^1\",\"Annotation type\",536870913],[165,\"^1\",\"Apply template to tags\",536870913],[53,\"^1\",\"Asset\",536870913],[175,\"^1\",\"Asset\",536870913],[142,\"^1\",\"Asset alignment\",536870913],[141,\"^1\",\"Asset resize metadata\",536870913],[106,\"^1\",\"Assignee\",536870913],[51,\"^1\",\"Background color\",536870913],[75,\"^1\",\"Backlog\",536870913],[42,\"^1\",\"Bidirectional property title\",536870913],[13,\"^1\",\"Built in?\",536870913]]]", + "addresses": null + }, + { + "addr": 1000332, + "content": "[\"^ \",\"~:keys\",[[80,\"~:block/title\",\"Canceled\",536870913],[173,\"^1\",\"Card\",536870913],[174,\"^1\",\"Cards\",536870913],[70,\"^1\",\"Choice checkbox state\",536870913],[71,\"^1\",\"Choice classes\",536870913],[72,\"^1\",\"Choice exclusions\",536870913],[30,\"^1\",\"Closed value property\",536870913],[176,\"^1\",\"Code\",536870913],[36,\"^1\",\"Code Mode\",536870913],[171,\"^1\",\"Comment\",536870913],[68,\"^1\",\"Commented blocks\",536870913],[170,\"^1\",\"Comments\",536870913],[186,\"^1\",\"Contents\",536870913],[15,\"^1\",\"Created from property\",536870913],[95,\"^1\",\"Day\",536870913],[86,\"^1\",\"Deadline\",536870913],[37,\"^1\",\"Default value\",536870913],[155,\"^1\",\"Deleted at\",536870913],[156,\"^1\",\"Deleted by\",536870913],[35,\"^1\",\"Description\",536870913],[77,\"^1\",\"Doing\",536870913],[79,\"^1\",\"Done\",536870913],[143,\"^1\",\"Due\",536870913],[43,\"^1\",\"Enable bidirectional properties\",536870913],[148,\"^1\",\"Enable property history\",536870913]]]", + "addresses": null + }, + { + "addr": 1000333, + "content": "[\"^ \",\"~:keys\",[[110,\"~:block/title\",\"Excluded from Graph view?\",536870913],[67,\"^1\",\"Excluded references\",536870913],[39,\"^1\",\"Extends\",536870913],[132,\"^1\",\"External URL\",536870913],[133,\"^1\",\"External file name\",536870913],[134,\"^1\",\"File Size\",536870913],[131,\"^1\",\"File Type\",536870913],[137,\"^1\",\"File checksum\",536870913],[139,\"^1\",\"File remote metadata\",536870913],[114,\"^1\",\"Gallery View\",536870913],[117,\"^1\",\"Gallery asset property\",536870913],[121,\"^1\",\"Gallery card height\",536870913],[119,\"^1\",\"Gallery card size\",536870913],[120,\"^1\",\"Gallery card width\",536870913],[118,\"^1\",\"Gallery display properties\",536870913],[52,\"^1\",\"Heading\",536870913]]]", + "addresses": null + }, + { + "addr": 1000334, + "content": "[\"^ \",\"~:keys\",[[45,\"~:block/title\",\"Hide empty value\",536870913],[47,\"^1\",\"Hide from Node\",536870913],[14,\"^1\",\"Hide this property or page\",536870913],[46,\"^1\",\"Hides a property's value on any node when empty e.g. when a property appears on a node through a tag.\",536870913],[84,\"^1\",\"High\",536870913],[150,\"^1\",\"History block\",536870913],[151,\"^1\",\"History property\",536870913],[153,\"^1\",\"History scalar value\",536870913],[152,\"^1\",\"History value\",536870913],[94,\"^1\",\"Hour\",536870913],[107,\"^1\",\"Icon\",536870913],[136,\"^1\",\"Image height\",536870913],[135,\"^1\",\"Image width\",536870913],[78,\"^1\",\"In Review\",536870913],[66,\"^1\",\"Included references\",536870913],[167,\"^1\",\"Journal\",536870913],[31,\"^1\",\"Journal date\",536870913]]]", + "addresses": null + }, + { + "addr": 1000335, + "content": "[\"^ \",\"~:keys\",[[138,\"~:block/title\",\"Last visit page\",536870913],[185,\"^1\",\"Library\",536870913],[113,\"^1\",\"List View\",536870913],[65,\"^1\",\"List type\",536870913],[82,\"^1\",\"Low\",536870913],[178,\"^1\",\"Math\",536870913],[83,\"^1\",\"Medium\",536870913],[140,\"^1\",\"Metadata of asset in remote storage\",536870913],[93,\"^1\",\"Minute\",536870913],[97,\"^1\",\"Month\",536870913],[34,\"^1\",\"Node Display Type\",536870913],[99,\"^1\",\"Node Repeats?\",536870913],[25,\"^1\",\"Node collapsed?\",536870913],[32,\"^1\",\"Node created at\",536870913],[154,\"^1\",\"Node created by\",536870913],[28,\"^1\",\"Node links to\",536870913],[24,\"^1\",\"Node order\",536870913],[26,\"^1\",\"Node page\",536870913],[23,\"^1\",\"Node parent\",536870913]]]", + "addresses": null + }, + { + "addr": 1000336, + "content": "[\"^ \",\"~:keys\",[[27,\"~:block/title\",\"Node references\",536870913],[29,\"^1\",\"Node title\",536870913],[33,\"^1\",\"Node updated at\",536870913],[38,\"^1\",\"Non ref type default value\",536870913],[179,\"^1\",\"PDF Annotation\",536870913],[4,\"^1\",\"Page\",536870913],[49,\"^1\",\"Page Tags\",536870913],[81,\"^1\",\"Priority\",536870913],[73,\"^1\",\"Properties displayed as checkbox\",536870913],[3,\"^1\",\"Property\",536870913],[19,\"^1\",\"Property classes\",536870913],[18,\"^1\",\"Property position\",536870913],[16,\"^1\",\"Property public?\",536870913],[12,\"^1\",\"Property type\",536870913],[20,\"^1\",\"Property value\",536870913],[17,\"^1\",\"Property view context\",536870913],[50,\"^1\",\"Provides a way for a page to associate to another page i.e. backward compatible tagging.\",536870913],[109,\"^1\",\"Published URL\",536870913],[108,\"^1\",\"Publishing Public?\",536870913]]]", + "addresses": null + }, + { + "addr": 1000337, + "content": "[\"^ \",\"~:keys\",[[48,\"~:block/title\",\"Query\",536870913],[172,\"^1\",\"Query\",536870913],[187,\"^1\",\"Quick add\",536870913],[177,\"^1\",\"Quote\",536870913],[160,\"^1\",\"Reaction emoji\",536870913],[161,\"^1\",\"Reaction target\",536870913],[149,\"^1\",\"Records history anytime a property's value changes on a node.\",536870913],[190,\"^1\",\"Recycle\",536870913],[159,\"^1\",\"Recycle original order\",536870913],[158,\"^1\",\"Recycle original page\",536870913],[157,\"^1\",\"Recycle original parent\",536870913],[166,\"^1\",\"Reference to large block title stored in remote object storage\",536870913],[105,\"^1\",\"Repeating Checked Property\",536870913],[104,\"^1\",\"Repeating Temporal Property\",536870913],[90,\"^1\",\"Repeating recur frequency\",536870913],[92,\"^1\",\"Repeating recur unit\",536870913],[100,\"^1\",\"Repeating type\",536870913],[1,\"^1\",\"Root Tag\",536870913],[88,\"^1\",\"Scheduled\",536870913],[144,\"^1\",\"State\",536870913],[74,\"^1\",\"Status\",536870913]]]", + "addresses": null + }, + { + "addr": 1000338, + "content": "[\"^ \",\"~:keys\",[[163,\"~:block/title\",\"Stores the AgentBridge session ID for a routed task.\",536870913],[112,\"^1\",\"Table View\",536870913],[129,\"^1\",\"Table view pinned columns\",536870913],[2,\"^1\",\"Tag\",536870913],[41,\"^1\",\"Tag Properties\",536870913],[22,\"^1\",\"Tags\",536870913],[169,\"^1\",\"Task\",536870913],[5,\"^1\",\"Template\",536870913],[40,\"^1\",\"This enables tags to inherit properties from other tags\",536870913],[130,\"^1\",\"This view belongs to\",536870913],[69,\"^1\",\"Title Format\",536870913],[76,\"^1\",\"Todo\",536870913],[85,\"^1\",\"Urgent\",536870913],[87,\"^1\",\"Use it to finish something at a specific date(time).\",536870913],[89,\"^1\",\"Use it to plan something to start at a specific date(time).\",536870913],[164,\"^1\",\"Used template\",536870913],[147,\"^1\",\"User Avatar\",536870913],[146,\"^1\",\"User Email\",536870913],[145,\"^1\",\"User Name\",536870913],[115,\"^1\",\"View Feature Type\",536870913],[111,\"^1\",\"View Type\",536870913],[128,\"^1\",\"View columns settings\",536870913],[125,\"^1\",\"View filters\",536870913],[116,\"^1\",\"View group by property\",536870913]]]", + "addresses": null + }, + { + "addr": 1000339, + "content": "[\"^ \",\"~:keys\",[[126,\"~:block/title\",\"View hidden columns\",536870913],[127,\"^1\",\"View ordered columns\",536870913],[123,\"^1\",\"View sort groups DESC\",536870913],[122,\"^1\",\"View sort groups by\",536870913],[124,\"^1\",\"View sorting\",536870913],[96,\"^1\",\"Week\",536870913],[44,\"^1\",\"When enabled, this tag will show reverse nodes that link to the current node via properties.\",536870913],[168,\"^1\",\"Whiteboard\",536870913],[98,\"^1\",\"Year\",536870913],[60,\"^1\",\"blue\",536870913],[59,\"^1\",\"green\",536870913],[54,\"^1\",\"ls-type\",536870913],[61,\"^1\",\"purple\",536870913],[58,\"^1\",\"red\",536870913],[57,\"^1\",\"yellow\",536870913],[12,\"~:block/updated-at\",1786627375714,536870913],[13,\"^2\",1786627375714,536870913]]]", + "addresses": null + }, + { + "addr": 1000340, + "content": "[\"^ \",\"~:keys\",[[87,\"~:block/order\",\"b0D\",536870913],[103,\"^1\",\"b0T\",536870913],[119,\"^1\",\"b0j\",536870913],[135,\"^1\",\"b0z\",536870913],[151,\"^1\",\"b1F\",536870913],[57,\"~:block/page\",56,536870913],[89,\"^2\",88,536870913],[50,\"~:block/parent\",49,536870913],[87,\"^3\",86,536870913],[149,\"^3\",148,536870913],[14,\"~:block/tags\",3,536870913],[30,\"^4\",3,536870913],[49,\"^4\",3,536870913],[71,\"^4\",3,536870913],[109,\"^4\",3,536870913],[128,\"^4\",3,536870913],[145,\"^4\",3,536870913],[101,\"~:block/title\",\"Advance from completion\",536870913],[13,\"^5\",\"Built in?\",536870913],[148,\"^5\",\"Enable property history\",536870913],[52,\"^5\",\"Heading\",536870913],[31,\"^5\",\"Journal date\",536870913],[23,\"^5\",\"Node parent\",536870913],[108,\"^5\",\"Publishing Public?\",536870913],[74,\"^5\",\"Status\",536870913],[116,\"^5\",\"View group by property\",536870913],[13,\"~:block/updated-at\",1786627375714,536870913]]]", + "addresses": "[1000313,1000314,1000315,1000316,1000317,1000318,1000319,1000320,1000321,1000322,1000323,1000324,1000325,1000326,1000327,1000328,1000329,1000330,1000331,1000332,1000333,1000334,1000335,1000336,1000337,1000338,1000339]" + }, + { + "addr": 1000341, + "content": "[\"^ \",\"~:keys\",[[14,\"~:block/updated-at\",1786627375714,536870913],[15,\"^1\",1786627375714,536870913],[16,\"^1\",1786627375715,536870913],[17,\"^1\",1786627375715,536870913],[18,\"^1\",1786627375715,536870913],[19,\"^1\",1786627375715,536870913],[20,\"^1\",1786627375716,536870913],[21,\"^1\",1786627375716,536870913],[22,\"^1\",1786627375716,536870913],[23,\"^1\",1786627375716,536870913],[24,\"^1\",1786627375716,536870913],[25,\"^1\",1786627375716,536870913],[26,\"^1\",1786627375716,536870913],[27,\"^1\",1786627375716,536870913],[28,\"^1\",1786627375716,536870913],[29,\"^1\",1786627375717,536870913],[30,\"^1\",1786627375717,536870913]]]", + "addresses": null + }, + { + "addr": 1000342, + "content": "[\"^ \",\"~:keys\",[[31,\"~:block/updated-at\",1786627375717,536870913],[32,\"^1\",1786627375717,536870913],[33,\"^1\",1786627375717,536870913],[34,\"^1\",1786627375717,536870913],[35,\"^1\",1786627375717,536870913],[36,\"^1\",1786627375717,536870913],[37,\"^1\",1786627375717,536870913],[38,\"^1\",1786627375717,536870913],[39,\"^1\",1786627375717,536870913],[40,\"^1\",1786627375717,536870913],[41,\"^1\",1786627375717,536870913],[42,\"^1\",1786627375717,536870913],[43,\"^1\",1786627375717,536870913],[44,\"^1\",1786627375717,536870913],[45,\"^1\",1786627375717,536870913],[46,\"^1\",1786627375718,536870913]]]", + "addresses": null + }, + { + "addr": 1000343, + "content": "[\"^ \",\"~:keys\",[[47,\"~:block/updated-at\",1786627375718,536870913],[48,\"^1\",1786627375718,536870913],[49,\"^1\",1786627375718,536870913],[50,\"^1\",1786627375718,536870913],[51,\"^1\",1786627375718,536870913],[52,\"^1\",1786627375718,536870913],[53,\"^1\",1786627375718,536870913],[54,\"^1\",1786627375718,536870913],[55,\"^1\",1786627375718,536870913],[56,\"^1\",1786627375718,536870913],[57,\"^1\",1786627375718,536870913],[58,\"^1\",1786627375718,536870913],[59,\"^1\",1786627375718,536870913],[60,\"^1\",1786627375718,536870913],[61,\"^1\",1786627375718,536870913],[62,\"^1\",1786627375719,536870913]]]", + "addresses": null + }, + { + "addr": 1000344, + "content": "[\"^ \",\"~:keys\",[[63,\"~:block/updated-at\",1786627375719,536870913],[64,\"^1\",1786627375719,536870913],[65,\"^1\",1786627375719,536870913],[66,\"^1\",1786627375719,536870913],[67,\"^1\",1786627375719,536870913],[68,\"^1\",1786627375719,536870913],[69,\"^1\",1786627375719,536870913],[70,\"^1\",1786627375719,536870913],[71,\"^1\",1786627375719,536870913],[72,\"^1\",1786627375719,536870913],[73,\"^1\",1786627375719,536870913],[74,\"^1\",1786627375719,536870913],[75,\"^1\",1786627375719,536870913],[76,\"^1\",1786627375719,536870913],[77,\"^1\",1786627375719,536870913],[78,\"^1\",1786627375719,536870913]]]", + "addresses": null + }, + { + "addr": 1000345, + "content": "[\"^ \",\"~:keys\",[[79,\"~:block/updated-at\",1786627375719,536870913],[80,\"^1\",1786627375720,536870913],[81,\"^1\",1786627375720,536870913],[82,\"^1\",1786627375720,536870913],[83,\"^1\",1786627375720,536870913],[84,\"^1\",1786627375720,536870913],[85,\"^1\",1786627375720,536870913],[86,\"^1\",1786627375720,536870913],[87,\"^1\",1786627375720,536870913],[88,\"^1\",1786627375720,536870913],[89,\"^1\",1786627375720,536870913],[90,\"^1\",1786627375720,536870913],[91,\"^1\",1786627375720,536870913],[92,\"^1\",1786627375720,536870913],[93,\"^1\",1786627375720,536870913],[94,\"^1\",1786627375720,536870913]]]", + "addresses": null + }, + { + "addr": 1000346, + "content": "[\"^ \",\"~:keys\",[[95,\"~:block/updated-at\",1786627375721,536870913],[96,\"^1\",1786627375721,536870913],[97,\"^1\",1786627375721,536870913],[98,\"^1\",1786627375721,536870913],[99,\"^1\",1786627375721,536870913],[100,\"^1\",1786627375721,536870913],[101,\"^1\",1786627375721,536870913],[102,\"^1\",1786627375721,536870913],[103,\"^1\",1786627375721,536870913],[104,\"^1\",1786627375721,536870913],[105,\"^1\",1786627375721,536870913],[106,\"^1\",1786627375721,536870913],[107,\"^1\",1786627375721,536870913],[108,\"^1\",1786627375721,536870913],[109,\"^1\",1786627375721,536870913],[110,\"^1\",1786627375721,536870913]]]", + "addresses": null + }, + { + "addr": 1000347, + "content": "[\"^ \",\"~:keys\",[[111,\"~:block/updated-at\",1786627375721,536870913],[112,\"^1\",1786627375721,536870913],[113,\"^1\",1786627375721,536870913],[114,\"^1\",1786627375721,536870913],[115,\"^1\",1786627375721,536870913],[116,\"^1\",1786627375721,536870913],[117,\"^1\",1786627375721,536870913],[118,\"^1\",1786627375722,536870913],[119,\"^1\",1786627375722,536870913],[120,\"^1\",1786627375722,536870913],[121,\"^1\",1786627375722,536870913],[122,\"^1\",1786627375722,536870913],[123,\"^1\",1786627375722,536870913],[124,\"^1\",1786627375722,536870913],[125,\"^1\",1786627375722,536870913],[126,\"^1\",1786627375722,536870913]]]", + "addresses": null + }, + { + "addr": 1000348, + "content": "[\"^ \",\"~:keys\",[[127,\"~:block/updated-at\",1786627375722,536870913],[128,\"^1\",1786627375722,536870913],[129,\"^1\",1786627375722,536870913],[130,\"^1\",1786627375722,536870913],[131,\"^1\",1786627375722,536870913],[132,\"^1\",1786627375722,536870913],[133,\"^1\",1786627375722,536870913],[134,\"^1\",1786627375722,536870913],[135,\"^1\",1786627375722,536870913],[136,\"^1\",1786627375722,536870913],[137,\"^1\",1786627375722,536870913],[138,\"^1\",1786627375722,536870913],[139,\"^1\",1786627375722,536870913],[140,\"^1\",1786627375722,536870913],[141,\"^1\",1786627375722,536870913],[142,\"^1\",1786627375722,536870913]]]", + "addresses": null + }, + { + "addr": 1000349, + "content": "[\"^ \",\"~:keys\",[[143,\"~:block/updated-at\",1786627375723,536870913],[144,\"^1\",1786627375723,536870913],[145,\"^1\",1786627375723,536870913],[146,\"^1\",1786627375723,536870913],[147,\"^1\",1786627375723,536870913],[148,\"^1\",1786627375723,536870913],[149,\"^1\",1786627375723,536870913],[150,\"^1\",1786627375723,536870913],[151,\"^1\",1786627375723,536870913],[152,\"^1\",1786627375723,536870913],[153,\"^1\",1786627375723,536870913],[154,\"^1\",1786627375723,536870913],[155,\"^1\",1786627375723,536870913],[156,\"^1\",1786627375723,536870913],[157,\"^1\",1786627375723,536870913],[158,\"^1\",1786627375723,536870913]]]", + "addresses": null + }, + { + "addr": 1000350, + "content": "[\"^ \",\"~:keys\",[[159,\"~:block/updated-at\",1786627375723,536870913],[160,\"^1\",1786627375723,536870913],[161,\"^1\",1786627375724,536870913],[162,\"^1\",1786627375724,536870913],[163,\"^1\",1786627375724,536870913],[164,\"^1\",1786627375724,536870913],[165,\"^1\",1786627375724,536870913],[166,\"^1\",1786627375724,536870913],[1,\"^1\",1786627375725,536870913],[2,\"^1\",1786627375725,536870913],[3,\"^1\",1786627375725,536870913],[4,\"^1\",1786627375725,536870913],[5,\"^1\",1786627375725,536870913],[167,\"^1\",1786627375725,536870913],[168,\"^1\",1786627375725,536870913],[169,\"^1\",1786627375725,536870913]]]", + "addresses": null + }, + { + "addr": 1000351, + "content": "[\"^ \",\"~:keys\",[[170,\"~:block/updated-at\",1786627375725,536870913],[171,\"^1\",1786627375725,536870913],[172,\"^1\",1786627375725,536870913],[173,\"^1\",1786627375725,536870913],[174,\"^1\",1786627375725,536870913],[175,\"^1\",1786627375725,536870913],[176,\"^1\",1786627375725,536870913],[177,\"^1\",1786627375725,536870913],[178,\"^1\",1786627375725,536870913],[179,\"^1\",1786627375725,536870913],[185,\"^1\",1786627375725,536870913],[186,\"^1\",1786627375725,536870913],[187,\"^1\",1786627375725,536870913],[188,\"^1\",1786627375725,536870913],[189,\"^1\",1786627375725,536870913],[190,\"^1\",1786627375725,536870913],[137,\"~:block/uuid\",\"~u00000002-1011-4169-7900-000000000000\",536870913]]]", + "addresses": null + }, + { + "addr": 1000352, + "content": "[\"^ \",\"~:keys\",[[168,\"~:block/uuid\",\"~u00000002-1013-6984-5200-000000000000\",536870913],[3,\"^1\",\"~u00000002-1038-7670-4800-000000000000\",536870913],[166,\"^1\",\"~u00000002-1044-8271-6500-000000000000\",536870913],[123,\"^1\",\"~u00000002-1081-4073-8700-000000000000\",536870913],[143,\"^1\",\"~u00000002-1089-0805-4900-000000000000\",536870913],[108,\"^1\",\"~u00000002-1094-6579-3900-000000000000\",536870913],[61,\"^1\",\"~u00000002-1104-3848-5600-000000000000\",536870913],[13,\"^1\",\"~u00000002-1125-9581-6000-000000000000\",536870913],[161,\"^1\",\"~u00000002-1127-5718-5000-000000000000\",536870913],[131,\"^1\",\"~u00000002-1142-0830-9800-000000000000\",536870913],[150,\"^1\",\"~u00000002-1142-5541-6000-000000000000\",536870913],[30,\"^1\",\"~u00000002-1157-7928-1300-000000000000\",536870913],[113,\"^1\",\"~u00000002-1164-8285-0200-000000000000\",536870913],[144,\"^1\",\"~u00000002-1165-1650-8700-000000000000\",536870913],[134,\"^1\",\"~u00000002-1167-8621-9000-000000000000\",536870913],[177,\"^1\",\"~u00000002-1176-1666-1700-000000000000\",536870913],[27,\"^1\",\"~u00000002-1214-4953-4900-000000000000\",536870913],[72,\"^1\",\"~u00000002-1233-5229-4600-000000000000\",536870913],[70,\"^1\",\"~u00000002-1272-4228-6300-000000000000\",536870913],[169,\"^1\",\"~u00000002-1282-1814-5700-000000000000\",536870913],[174,\"^1\",\"~u00000002-1284-2651-6700-000000000000\",536870913],[141,\"^1\",\"~u00000002-1297-5230-5500-000000000000\",536870913],[171,\"^1\",\"~u00000002-1323-0695-4000-000000000000\",536870913],[173,\"^1\",\"~u00000002-1358-2811-0900-000000000000\",536870913],[145,\"^1\",\"~u00000002-1360-0260-1600-000000000000\",536870913],[132,\"^1\",\"~u00000002-1364-7751-6300-000000000000\",536870913],[63,\"^1\",\"~u00000002-1377-6700-9000-000000000000\",536870913],[20,\"^1\",\"~u00000002-1396-5248-6500-000000000000\",536870913]]]", + "addresses": null + }, + { + "addr": 1000353, + "content": "[\"^ \",\"~:keys\",[[120,\"~:block/uuid\",\"~u00000002-1403-2963-9500-000000000000\",536870913],[101,\"^1\",\"~u00000002-1406-5331-0600-000000000000\",536870913],[24,\"^1\",\"~u00000002-1429-2824-3700-000000000000\",536870913],[94,\"^1\",\"~u00000002-1438-8849-5400-000000000000\",536870913],[32,\"^1\",\"~u00000002-1440-0150-0000-000000000000\",536870913],[176,\"^1\",\"~u00000002-1454-9866-4100-000000000000\",536870913],[31,\"^1\",\"~u00000002-1457-4836-6000-000000000000\",536870913],[4,\"^1\",\"~u00000002-1484-3403-2900-000000000000\",536870913],[127,\"^1\",\"~u00000002-1485-5871-0000-000000000000\",536870913],[114,\"^1\",\"~u00000002-1506-0511-2000-000000000000\",536870913],[93,\"^1\",\"~u00000002-1513-6550-8500-000000000000\",536870913],[33,\"^1\",\"~u00000002-1516-5505-5100-000000000000\",536870913],[98,\"^1\",\"~u00000002-1520-4385-2400-000000000000\",536870913],[69,\"^1\",\"~u00000002-1536-4979-5400-000000000000\",536870913],[17,\"^1\",\"~u00000002-1547-3958-2800-000000000000\",536870913],[38,\"^1\",\"~u00000002-1595-7230-1400-000000000000\",536870913],[151,\"^1\",\"~u00000002-1600-4090-8200-000000000000\",536870913],[76,\"^1\",\"~u00000002-1615-5853-7700-000000000000\",536870913]]]", + "addresses": null + }, + { + "addr": 1000354, + "content": "[\"^ \",\"~:keys\",[[88,\"~:block/uuid\",\"~u00000002-1644-5209-4300-000000000000\",536870913],[102,\"^1\",\"~u00000002-1646-4063-5900-000000000000\",536870913],[146,\"^1\",\"~u00000002-1655-2060-6300-000000000000\",536870913],[158,\"^1\",\"~u00000002-1658-7897-9900-000000000000\",536870913],[117,\"^1\",\"~u00000002-1668-3468-9000-000000000000\",536870913],[128,\"^1\",\"~u00000002-1675-5105-5500-000000000000\",536870913],[66,\"^1\",\"~u00000002-1680-5777-0300-000000000000\",536870913],[86,\"^1\",\"~u00000002-1685-9016-0400-000000000000\",536870913],[125,\"^1\",\"~u00000002-1702-3936-3300-000000000000\",536870913],[5,\"^1\",\"~u00000002-1720-8548-4600-000000000000\",536870913],[57,\"^1\",\"~u00000002-1752-1030-0200-000000000000\",536870913],[162,\"^1\",\"~u00000002-1783-0312-6200-000000000000\",536870913],[118,\"^1\",\"~u00000002-1806-7066-8800-000000000000\",536870913],[22,\"^1\",\"~u00000002-1814-9483-4000-000000000000\",536870913],[79,\"^1\",\"~u00000002-1827-5820-8200-000000000000\",536870913],[83,\"^1\",\"~u00000002-1829-3222-7800-000000000000\",536870913],[60,\"^1\",\"~u00000002-1836-0512-4100-000000000000\",536870913],[77,\"^1\",\"~u00000002-1840-1229-0800-000000000000\",536870913]]]", + "addresses": null + }, + { + "addr": 1000355, + "content": "[\"^ \",\"~:keys\",[[16,\"~:block/uuid\",\"~u00000002-1843-0851-4900-000000000000\",536870913],[135,\"^1\",\"~u00000002-1857-8658-3900-000000000000\",536870913],[52,\"^1\",\"~u00000002-1858-7494-1500-000000000000\",536870913],[105,\"^1\",\"~u00000002-1866-3655-5300-000000000000\",536870913],[18,\"^1\",\"~u00000002-1869-2008-6400-000000000000\",536870913],[78,\"^1\",\"~u00000002-1870-0014-4300-000000000000\",536870913],[28,\"^1\",\"~u00000002-1872-3999-9300-000000000000\",536870913],[99,\"^1\",\"~u00000002-1908-1217-8900-000000000000\",536870913],[112,\"^1\",\"~u00000002-1942-5424-0000-000000000000\",536870913],[167,\"^1\",\"~u00000002-1979-7410-8100-000000000000\",536870913],[100,\"^1\",\"~u00000002-1984-2972-8100-000000000000\",536870913],[59,\"^1\",\"~u00000002-1992-2016-2600-000000000000\",536870913],[85,\"^1\",\"~u00000002-1996-4346-6700-000000000000\",536870913],[156,\"^1\",\"~u00000002-1998-0431-1200-000000000000\",536870913],[119,\"^1\",\"~u00000002-2012-8168-3300-000000000000\",536870913],[178,\"^1\",\"~u00000002-2038-9631-2100-000000000000\",536870913],[109,\"^1\",\"~u00000002-2045-1825-0100-000000000000\",536870913],[45,\"^1\",\"~u00000002-2062-3258-9900-000000000000\",536870913],[97,\"^1\",\"~u00000002-2073-3937-9700-000000000000\",536870913]]]", + "addresses": null + }, + { + "addr": 1000356, + "content": "[\"^ \",\"~:keys\",[[124,\"~:block/uuid\",\"~u00000002-2081-0259-4000-000000000000\",536870913],[82,\"^1\",\"~u00000002-2107-4537-4800-000000000000\",536870913],[138,\"^1\",\"~u00000002-2107-8035-3500-000000000000\",536870913],[21,\"^1\",\"~u00000002-2112-6446-9900-000000000000\",536870913],[41,\"^1\",\"~u00000002-2123-7120-5000-000000000000\",536870913],[96,\"^1\",\"~u00000002-2130-9244-4900-000000000000\",536870913],[49,\"^1\",\"~u00000002-2133-5311-8500-000000000000\",536870913],[25,\"^1\",\"~u00000002-2140-2109-9100-000000000000\",536870913],[68,\"^1\",\"~u00000002-2143-9561-0900-000000000000\",536870913],[111,\"^1\",\"~u00000002-2182-3760-7000-000000000000\",536870913],[106,\"^1\",\"~u00000002-2190-1503-4000-000000000000\",536870913],[172,\"^1\",\"~u00000002-2324-8016-6000-000000000000\",536870913],[81,\"^1\",\"~u00000002-2392-2841-1000-000000000000\",536870913],[153,\"^1\",\"~u00000002-2393-3777-5000-000000000000\",536870913],[67,\"^1\",\"~u00000002-2426-7588-9000-000000000000\",536870913],[170,\"^1\",\"~u00000002-2556-9161-5000-000000000000\",536870913],[47,\"^1\",\"~u00000002-2610-3727-0000-000000000000\",536870913],[129,\"^1\",\"~u00000002-2673-7513-8000-000000000000\",536870913],[1,\"^1\",\"~u00000002-2737-8382-7000-000000000000\",536870913]]]", + "addresses": null + }, + { + "addr": 1000357, + "content": "[\"^ \",\"~:keys\",[[43,\"~:block/uuid\",\"~u00000002-2917-8613-8000-000000000000\",536870913],[73,\"^1\",\"~u00000002-3215-3256-9000-000000000000\",536870913],[54,\"^1\",\"~u00000002-3269-7934-5000-000000000000\",536870913],[136,\"^1\",\"~u00000002-3306-1650-4000-000000000000\",536870913],[35,\"^1\",\"~u00000002-3362-3620-0000-000000000000\",536870913],[130,\"^1\",\"~u00000002-3627-4319-0000-000000000000\",536870913],[95,\"^1\",\"~u00000002-3924-1785-8000-000000000000\",536870913],[133,\"^1\",\"~u00000002-3995-2972-0000-000000000000\",536870913],[58,\"^1\",\"~u00000002-4136-2216-2000-000000000000\",536870913],[147,\"^1\",\"~u00000002-4165-4885-8000-000000000000\",536870913],[165,\"^1\",\"~u00000002-4291-2432-2000-000000000000\",536870913],[121,\"^1\",\"~u00000002-4342-2083-2000-000000000000\",536870913],[34,\"^1\",\"~u00000002-4424-4618-9000-000000000000\",536870913],[110,\"^1\",\"~u00000002-4524-3306-5000-000000000000\",536870913],[157,\"^1\",\"~u00000002-4970-0399-4000-000000000000\",536870913],[179,\"^1\",\"~u00000002-5049-5962-0000-000000000000\",536870913],[152,\"^1\",\"~u00000002-5131-3603-7000-000000000000\",536870913],[51,\"^1\",\"~u00000002-5191-2660-6000-000000000000\",536870913],[2,\"^1\",\"~u00000002-5389-0208-3000-000000000000\",536870913],[64,\"^1\",\"~u00000002-5458-2940-2000-000000000000\",536870913],[84,\"^1\",\"~u00000002-5672-2766-8000-000000000000\",536870913],[107,\"^1\",\"~u00000002-5891-2328-5000-000000000000\",536870913],[42,\"^1\",\"~u00000002-6050-5418-7000-000000000000\",536870913],[65,\"^1\",\"~u00000002-6078-1711-1000-000000000000\",536870913],[122,\"^1\",\"~u00000002-6253-4002-1000-000000000000\",536870913],[56,\"^1\",\"~u00000002-6747-9388-7000-000000000000\",536870913],[14,\"^1\",\"~u00000002-6836-5746-0000-000000000000\",536870913]]]", + "addresses": null + }, + { + "addr": 1000358, + "content": "[\"^ \",\"~:keys\",[[155,\"~:block/uuid\",\"~u00000002-6852-1375-1000-000000000000\",536870913],[92,\"^1\",\"~u00000002-6903-0624-7000-000000000000\",536870913],[29,\"^1\",\"~u00000002-7104-4568-4000-000000000000\",536870913],[142,\"^1\",\"~u00000002-7135-0412-8000-000000000000\",536870913],[75,\"^1\",\"~u00000002-7233-3491-0000-000000000000\",536870913],[103,\"^1\",\"~u00000002-7441-9615-3000-000000000000\",536870913],[39,\"^1\",\"~u00000002-7475-9380-3000-000000000000\",536870913],[80,\"^1\",\"~u00000002-7526-4326-2000-000000000000\",536870913],[62,\"^1\",\"~u00000002-7532-8459-6000-000000000000\",536870913],[71,\"^1\",\"~u00000002-7629-6621-9000-000000000000\",536870913],[175,\"^1\",\"~u00000002-7975-0297-0000-000000000000\",536870913],[148,\"^1\",\"~u00000002-8058-5960-2000-000000000000\",536870913],[116,\"^1\",\"~u00000002-8092-1623-6000-000000000000\",536870913],[26,\"^1\",\"~u00000002-8223-1410-8000-000000000000\",536870913],[104,\"^1\",\"~u00000002-8346-1078-4000-000000000000\",536870913],[12,\"^1\",\"~u00000002-8384-2404-0000-000000000000\",536870913],[154,\"^1\",\"~u00000002-8544-3390-8000-000000000000\",536870913],[36,\"^1\",\"~u00000002-8578-9716-5000-000000000000\",536870913],[15,\"^1\",\"~u00000002-8618-9226-7000-000000000000\",536870913],[159,\"^1\",\"~u00000002-8699-2537-0000-000000000000\",536870913],[90,\"^1\",\"~u00000002-8716-1592-2000-000000000000\",536870913],[53,\"^1\",\"~u00000002-8768-5679-0000-000000000000\",536870913],[37,\"^1\",\"~u00000002-8920-7966-2000-000000000000\",536870913]]]", + "addresses": null + }, + { + "addr": 1000359, + "content": "[\"^ \",\"~:keys\",[[74,\"~:block/uuid\",\"~u00000002-9072-1685-3000-000000000000\",536870913],[19,\"^1\",\"~u00000002-9137-5048-6000-000000000000\",536870913],[23,\"^1\",\"~u00000002-9183-0906-4000-000000000000\",536870913],[115,\"^1\",\"~u00000002-9391-4187-1000-000000000000\",536870913],[48,\"^1\",\"~u00000002-9741-4126-0000-000000000000\",536870913],[126,\"^1\",\"~u00000002-9750-5719-2000-000000000000\",536870913],[164,\"^1\",\"~u00000002-9803-6990-6000-000000000000\",536870913],[160,\"^1\",\"~u00000002-9877-5864-5000-000000000000\",536870913],[139,\"^1\",\"~u00000002-9907-5046-9000-000000000000\",536870913],[55,\"^1\",\"~u00000002-9984-3783-2000-000000000000\",536870913],[189,\"^1\",\"~u00000004-1018-5888-4100-000000000000\",536870913],[46,\"^1\",\"~u00000004-1118-8585-8000-000000000000\",536870913],[163,\"^1\",\"~u00000004-1209-4784-1500-000000000000\",536870913],[50,\"^1\",\"~u00000004-1223-8671-8000-000000000000\",536870913],[185,\"^1\",\"~u00000004-1294-7765-6000-000000000000\",536870913],[182,\"^1\",\"~u00000004-1335-6485-2300-000000000000\",536870913],[87,\"^1\",\"~u00000004-1356-3664-2200-000000000000\",536870913],[149,\"^1\",\"~u00000004-1438-1481-0000-000000000000\",536870913]]]", + "addresses": null + }, + { + "addr": 1000360, + "content": "[\"^ \",\"~:keys\",[[10,\"~:block/uuid\",\"~u00000004-1595-0218-3700-000000000000\",536870913],[91,\"^1\",\"~u00000004-1654-1034-2600-000000000000\",536870913],[186,\"^1\",\"~u00000004-1690-2597-3200-000000000000\",536870913],[181,\"^1\",\"~u00000004-1713-4660-3800-000000000000\",536870913],[44,\"^1\",\"~u00000004-1830-0481-2500-000000000000\",536870913],[188,\"^1\",\"~u00000004-1906-3437-5800-000000000000\",536870913],[184,\"^1\",\"~u00000004-2116-2824-5200-000000000000\",536870913],[180,\"^1\",\"~u00000004-3919-3813-3000-000000000000\",536870913],[183,\"^1\",\"~u00000004-4049-4381-0000-000000000000\",536870913],[40,\"^1\",\"~u00000004-4485-6574-6000-000000000000\",536870913],[140,\"^1\",\"~u00000004-5450-2102-9000-000000000000\",536870913],[190,\"^1\",\"~u00000004-7238-1304-3000-000000000000\",536870913],[187,\"^1\",\"~u00000004-7336-8251-3000-000000000000\",536870913],[89,\"^1\",\"~u00000004-9817-6380-0000-000000000000\",536870913],[21,\"~:db/ident\",\"~:block/alias\",536870913],[30,\"^2\",\"~:block/closed-value-property\",536870913],[25,\"^2\",\"~:block/collapsed?\",536870913],[32,\"^2\",\"~:block/created-at\",536870913],[31,\"^2\",\"~:block/journal-day\",536870913],[28,\"^2\",\"~:block/link\",536870913],[24,\"^2\",\"~:block/order\",536870913]]]", + "addresses": null + }, + { + "addr": 1000361, + "content": "[\"^ \",\"~:keys\",[[26,\"~:db/ident\",\"~:block/page\",536870913],[23,\"^1\",\"~:block/parent\",536870913],[27,\"^1\",\"~:block/refs\",536870913],[22,\"^1\",\"~:block/tags\",536870913],[29,\"^1\",\"~:block/title\",536870913],[33,\"^1\",\"~:block/updated-at\",536870913],[175,\"^1\",\"~:logseq.class/Asset\",536870913],[173,\"^1\",\"~:logseq.class/Card\",536870913],[174,\"^1\",\"~:logseq.class/Cards\",536870913],[176,\"^1\",\"~:logseq.class/Code-block\",536870913],[171,\"^1\",\"~:logseq.class/Comment\",536870913],[170,\"^1\",\"~:logseq.class/Comments\",536870913],[167,\"^1\",\"~:logseq.class/Journal\",536870913],[178,\"^1\",\"~:logseq.class/Math-block\",536870913],[4,\"^1\",\"~:logseq.class/Page\",536870913],[179,\"^1\",\"~:logseq.class/Pdf-annotation\",536870913],[3,\"^1\",\"~:logseq.class/Property\",536870913]]]", + "addresses": null + }, + { + "addr": 1000362, + "content": "[\"^ \",\"~:keys\",[[172,\"~:db/ident\",\"~:logseq.class/Query\",536870913],[177,\"^1\",\"~:logseq.class/Quote-block\",536870913],[1,\"^1\",\"~:logseq.class/Root\",536870913],[2,\"^1\",\"~:logseq.class/Tag\",536870913],[169,\"^1\",\"~:logseq.class/Task\",536870913],[5,\"^1\",\"~:logseq.class/Template\",536870913],[168,\"^1\",\"~:logseq.class/Whiteboard\",536870913],[6,\"^1\",\"~:logseq.kv/db-type\",536870913],[9,\"^1\",\"~:logseq.kv/graph-created-at\",536870913],[8,\"^1\",\"~:logseq.kv/graph-initial-schema-version\",536870913],[11,\"^1\",\"~:logseq.kv/local-graph-uuid\",536870913],[7,\"^1\",\"~:logseq.kv/schema-version\",536870913],[53,\"^1\",\"~:logseq.property/asset\",536870913],[106,\"^1\",\"~:logseq.property/assignee\",536870913],[51,\"^1\",\"~:logseq.property/background-color\",536870913],[13,\"^1\",\"~:logseq.property/built-in?\",536870913]]]", + "addresses": null + }, + { + "addr": 1000363, + "content": "[\"^ \",\"~:keys\",[[73,\"~:db/ident\",\"~:logseq.property/checkbox-display-properties\",536870913],[70,\"^1\",\"~:logseq.property/choice-checkbox-state\",536870913],[71,\"^1\",\"~:logseq.property/choice-classes\",536870913],[72,\"^1\",\"~:logseq.property/choice-exclusions\",536870913],[19,\"^1\",\"~:logseq.property/classes\",536870913],[60,\"^1\",\"~:logseq.property/color.blue\",536870913],[59,\"^1\",\"~:logseq.property/color.green\",536870913],[61,\"^1\",\"~:logseq.property/color.purple\",536870913],[58,\"^1\",\"~:logseq.property/color.red\",536870913],[57,\"^1\",\"~:logseq.property/color.yellow\",536870913],[154,\"^1\",\"~:logseq.property/created-by-ref\",536870913],[15,\"^1\",\"~:logseq.property/created-from-property\",536870913],[86,\"^1\",\"~:logseq.property/deadline\",536870913],[37,\"^1\",\"~:logseq.property/default-value\",536870913],[155,\"^1\",\"~:logseq.property/deleted-at\",536870913],[156,\"^1\",\"~:logseq.property/deleted-by-ref\",536870913],[35,\"^1\",\"~:logseq.property/description\",536870913],[10,\"^1\",\"~:logseq.property/empty-placeholder\",536870913],[148,\"^1\",\"~:logseq.property/enable-history?\",536870913],[110,\"^1\",\"~:logseq.property/exclude-from-graph-view\",536870913],[52,\"^1\",\"~:logseq.property/heading\",536870913],[45,\"^1\",\"~:logseq.property/hide-empty-value\",536870913]]]", + "addresses": null + }, + { + "addr": 1000364, + "content": "[\"^ \",\"~:keys\",[[14,\"~:db/ident\",\"~:logseq.property/hide?\",536870913],[107,\"^1\",\"~:logseq.property/icon\",536870913],[54,\"^1\",\"~:logseq.property/ls-type\",536870913],[65,\"^1\",\"~:logseq.property/order-list-type\",536870913],[49,\"^1\",\"~:logseq.property/page-tags\",536870913],[81,\"^1\",\"~:logseq.property/priority\",536870913],[84,\"^1\",\"~:logseq.property/priority.high\",536870913],[82,\"^1\",\"~:logseq.property/priority.low\",536870913],[83,\"^1\",\"~:logseq.property/priority.medium\",536870913],[85,\"^1\",\"~:logseq.property/priority.urgent\",536870913],[16,\"^1\",\"~:logseq.property/public?\",536870913],[108,\"^1\",\"~:logseq.property/publishing-public?\",536870913],[48,\"^1\",\"~:logseq.property/query\",536870913],[38,\"^1\",\"~:logseq.property/scalar-default-value\",536870913],[88,\"^1\",\"~:logseq.property/scheduled\",536870913],[74,\"^1\",\"~:logseq.property/status\",536870913]]]", + "addresses": null + }, + { + "addr": 1000365, + "content": "[\"^ \",\"~:keys\",[[75,\"~:db/ident\",\"~:logseq.property/status.backlog\",536870913],[80,\"^1\",\"~:logseq.property/status.canceled\",536870913],[77,\"^1\",\"~:logseq.property/status.doing\",536870913],[79,\"^1\",\"~:logseq.property/status.done\",536870913],[78,\"^1\",\"~:logseq.property/status.in-review\",536870913],[76,\"^1\",\"~:logseq.property/status.todo\",536870913],[165,\"^1\",\"~:logseq.property/template-applied-to\",536870913],[12,\"^1\",\"~:logseq.property/type\",536870913],[18,\"^1\",\"~:logseq.property/ui-position\",536870913],[164,\"^1\",\"~:logseq.property/used-template\",536870913],[20,\"^1\",\"~:logseq.property/value\",536870913],[17,\"^1\",\"~:logseq.property/view-context\",536870913],[130,\"^1\",\"~:logseq.property/view-for\",536870913],[162,\"^1\",\"~:logseq.property.agent/session-id\",536870913],[142,\"^1\",\"~:logseq.property.asset/align\",536870913],[137,\"^1\",\"~:logseq.property.asset/checksum\",536870913],[133,\"^1\",\"~:logseq.property.asset/external-file-name\",536870913],[132,\"^1\",\"~:logseq.property.asset/external-url\",536870913],[136,\"^1\",\"~:logseq.property.asset/height\",536870913],[138,\"^1\",\"~:logseq.property.asset/last-visit-page\",536870913],[139,\"^1\",\"~:logseq.property.asset/remote-metadata\",536870913],[141,\"^1\",\"~:logseq.property.asset/resize-metadata\",536870913],[134,\"^1\",\"~:logseq.property.asset/size\",536870913],[131,\"^1\",\"~:logseq.property.asset/type\",536870913],[135,\"^1\",\"~:logseq.property.asset/width\",536870913],[42,\"^1\",\"~:logseq.property.class/bidirectional-property-title\",536870913],[43,\"^1\",\"~:logseq.property.class/enable-bidirectional?\",536870913],[39,\"^1\",\"~:logseq.property.class/extends\",536870913],[47,\"^1\",\"~:logseq.property.class/hide-from-node\",536870913]]]", + "addresses": null + }, + { + "addr": 1000366, + "content": "[\"^ \",\"~:keys\",[[30,\"~:block/updated-at\",1786627375717,536870913],[46,\"^1\",1786627375718,536870913],[62,\"^1\",1786627375719,536870913],[78,\"^1\",1786627375719,536870913],[94,\"^1\",1786627375720,536870913],[110,\"^1\",1786627375721,536870913],[126,\"^1\",1786627375722,536870913],[142,\"^1\",1786627375722,536870913],[158,\"^1\",1786627375723,536870913],[169,\"^1\",1786627375725,536870913],[137,\"~:block/uuid\",\"~u00000002-1011-4169-7900-000000000000\",536870913],[20,\"^2\",\"~u00000002-1396-5248-6500-000000000000\",536870913],[76,\"^2\",\"~u00000002-1615-5853-7700-000000000000\",536870913],[77,\"^2\",\"~u00000002-1840-1229-0800-000000000000\",536870913],[97,\"^2\",\"~u00000002-2073-3937-9700-000000000000\",536870913],[1,\"^2\",\"~u00000002-2737-8382-7000-000000000000\",536870913],[14,\"^2\",\"~u00000002-6836-5746-0000-000000000000\",536870913],[37,\"^2\",\"~u00000002-8920-7966-2000-000000000000\",536870913],[149,\"^2\",\"~u00000004-1438-1481-0000-000000000000\",536870913],[24,\"~:db/ident\",\"~:block/order\",536870913],[3,\"^3\",\"~:logseq.class/Property\",536870913],[13,\"^3\",\"~:logseq.property/built-in?\",536870913],[45,\"^3\",\"~:logseq.property/hide-empty-value\",536870913],[74,\"^3\",\"~:logseq.property/status\",536870913],[47,\"^3\",\"~:logseq.property.class/hide-from-node\",536870913]]]", + "addresses": "[1000341,1000342,1000343,1000344,1000345,1000346,1000347,1000348,1000349,1000350,1000351,1000352,1000353,1000354,1000355,1000356,1000357,1000358,1000359,1000360,1000361,1000362,1000363,1000364,1000365]" + }, + { + "addr": 1000367, + "content": "[\"^ \",\"~:keys\",[[41,\"~:db/ident\",\"~:logseq.property.class/properties\",536870913],[36,\"^1\",\"~:logseq.property.code/lang\",536870913],[68,\"^1\",\"~:logseq.property.comments/blocks\",536870913],[143,\"^1\",\"~:logseq.property.fsrs/due\",536870913],[144,\"^1\",\"~:logseq.property.fsrs/state\",536870913],[150,\"^1\",\"~:logseq.property.history/block\",536870913],[151,\"^1\",\"~:logseq.property.history/property\",536870913],[152,\"^1\",\"~:logseq.property.history/ref-value\",536870913],[153,\"^1\",\"~:logseq.property.history/scalar-value\",536870913],[69,\"^1\",\"~:logseq.property.journal/title-format\",536870913],[67,\"^1\",\"~:logseq.property.linked-references/excludes\",536870913],[66,\"^1\",\"~:logseq.property.linked-references/includes\",536870913],[34,\"^1\",\"~:logseq.property.node/display-type\",536870913],[56,\"^1\",\"~:logseq.property.pdf/hl-color\",536870913],[63,\"^1\",\"~:logseq.property.pdf/hl-image\",536870913],[62,\"^1\",\"~:logseq.property.pdf/hl-page\",536870913],[55,\"^1\",\"~:logseq.property.pdf/hl-type\",536870913],[64,\"^1\",\"~:logseq.property.pdf/hl-value\",536870913],[109,\"^1\",\"~:logseq.property.publish/published-url\",536870913],[160,\"^1\",\"~:logseq.property.reaction/emoji-id\",536870913],[161,\"^1\",\"~:logseq.property.reaction/target\",536870913],[159,\"^1\",\"~:logseq.property.recycle/original-order\",536870913],[158,\"^1\",\"~:logseq.property.recycle/original-page\",536870913],[157,\"^1\",\"~:logseq.property.recycle/original-parent\",536870913],[105,\"^1\",\"~:logseq.property.repeat/checked-property\",536870913],[90,\"^1\",\"~:logseq.property.repeat/recur-frequency\",536870913],[92,\"^1\",\"~:logseq.property.repeat/recur-unit\",536870913],[95,\"^1\",\"~:logseq.property.repeat/recur-unit.day\",536870913],[94,\"^1\",\"~:logseq.property.repeat/recur-unit.hour\",536870913]]]", + "addresses": null + }, + { + "addr": 1000368, + "content": "[\"^ \",\"~:keys\",[[93,\"~:db/ident\",\"~:logseq.property.repeat/recur-unit.minute\",536870913],[97,\"^1\",\"~:logseq.property.repeat/recur-unit.month\",536870913],[96,\"^1\",\"~:logseq.property.repeat/recur-unit.week\",536870913],[98,\"^1\",\"~:logseq.property.repeat/recur-unit.year\",536870913],[100,\"^1\",\"~:logseq.property.repeat/repeat-type\",536870913],[101,\"^1\",\"~:logseq.property.repeat/repeat-type.dotted-plus\",536870913],[103,\"^1\",\"~:logseq.property.repeat/repeat-type.double-plus\",536870913],[102,\"^1\",\"~:logseq.property.repeat/repeat-type.plus\",536870913],[99,\"^1\",\"~:logseq.property.repeat/repeated?\",536870913],[104,\"^1\",\"~:logseq.property.repeat/temporal-property\",536870913],[166,\"^1\",\"~:logseq.property.sync/large-title-object\",536870913],[125,\"^1\",\"~:logseq.property.table/filters\",536870913],[126,\"^1\",\"~:logseq.property.table/hidden-columns\",536870913],[127,\"^1\",\"~:logseq.property.table/ordered-columns\",536870913],[129,\"^1\",\"~:logseq.property.table/pinned-columns\",536870913],[128,\"^1\",\"~:logseq.property.table/sized-columns\",536870913],[124,\"^1\",\"~:logseq.property.table/sorting\",536870913],[147,\"^1\",\"~:logseq.property.user/avatar\",536870913],[146,\"^1\",\"~:logseq.property.user/email\",536870913],[145,\"^1\",\"~:logseq.property.user/name\",536870913],[115,\"^1\",\"~:logseq.property.view/feature-type\",536870913],[117,\"^1\",\"~:logseq.property.view/gallery-asset-property\",536870913],[121,\"^1\",\"~:logseq.property.view/gallery-card-height\",536870913],[119,\"^1\",\"~:logseq.property.view/gallery-card-size\",536870913]]]", + "addresses": null + }, + { + "addr": 1000369, + "content": "[\"^ \",\"~:keys\",[[120,\"~:db/ident\",\"~:logseq.property.view/gallery-card-width\",536870913],[118,\"^1\",\"~:logseq.property.view/gallery-display-properties\",536870913],[116,\"^1\",\"~:logseq.property.view/group-by-property\",536870913],[122,\"^1\",\"~:logseq.property.view/sort-groups-by-property\",536870913],[123,\"^1\",\"~:logseq.property.view/sort-groups-desc?\",536870913],[111,\"^1\",\"~:logseq.property.view/type\",536870913],[114,\"^1\",\"~:logseq.property.view/type.gallery\",536870913],[113,\"^1\",\"~:logseq.property.view/type.list\",536870913],[112,\"^1\",\"~:logseq.property.view/type.table\",536870913],[180,\"~:file/path\",\"logseq/config.edn\",536870913],[181,\"^;\",\"logseq/custom.css\",536870913],[182,\"^;\",\"logseq/custom.js\",536870913],[183,\"^;\",\"logseq/publish.css\",536870913],[184,\"^;\",\"logseq/publish.js\",536870913],[1,\"~:logseq.property/built-in?\",true,536870913],[2,\"^<\",true,536870913],[3,\"^<\",true,536870913],[4,\"^<\",true,536870913],[5,\"^<\",true,536870913],[12,\"^<\",true,536870913],[13,\"^<\",true,536870913],[14,\"^<\",true,536870913],[15,\"^<\",true,536870913],[16,\"^<\",true,536870913],[17,\"^<\",true,536870913],[18,\"^<\",true,536870913]]]", + "addresses": null + }, + { + "addr": 1000370, + "content": "[\"^ \",\"~:keys\",[[19,\"~:logseq.property/built-in?\",true,536870913],[20,\"^1\",true,536870913],[21,\"^1\",true,536870913],[22,\"^1\",true,536870913],[23,\"^1\",true,536870913],[24,\"^1\",true,536870913],[25,\"^1\",true,536870913],[26,\"^1\",true,536870913],[27,\"^1\",true,536870913],[28,\"^1\",true,536870913],[29,\"^1\",true,536870913],[30,\"^1\",true,536870913],[31,\"^1\",true,536870913],[32,\"^1\",true,536870913],[33,\"^1\",true,536870913],[34,\"^1\",true,536870913]]]", + "addresses": null + }, + { + "addr": 1000371, + "content": "[\"^ \",\"~:keys\",[[35,\"~:logseq.property/built-in?\",true,536870913],[36,\"^1\",true,536870913],[37,\"^1\",true,536870913],[38,\"^1\",true,536870913],[39,\"^1\",true,536870913],[40,\"^1\",true,536870913],[41,\"^1\",true,536870913],[42,\"^1\",true,536870913],[43,\"^1\",true,536870913],[44,\"^1\",true,536870913],[45,\"^1\",true,536870913],[46,\"^1\",true,536870913],[47,\"^1\",true,536870913],[48,\"^1\",true,536870913],[49,\"^1\",true,536870913],[50,\"^1\",true,536870913]]]", + "addresses": null + }, + { + "addr": 1000372, + "content": "[\"^ \",\"~:keys\",[[51,\"~:logseq.property/built-in?\",true,536870913],[52,\"^1\",true,536870913],[53,\"^1\",true,536870913],[54,\"^1\",true,536870913],[55,\"^1\",true,536870913],[56,\"^1\",true,536870913],[57,\"^1\",true,536870913],[58,\"^1\",true,536870913],[59,\"^1\",true,536870913],[60,\"^1\",true,536870913],[61,\"^1\",true,536870913],[62,\"^1\",true,536870913],[63,\"^1\",true,536870913],[64,\"^1\",true,536870913],[65,\"^1\",true,536870913],[66,\"^1\",true,536870913]]]", + "addresses": null + }, + { + "addr": 1000373, + "content": "[\"^ \",\"~:keys\",[[67,\"~:logseq.property/built-in?\",true,536870913],[68,\"^1\",true,536870913],[69,\"^1\",true,536870913],[70,\"^1\",true,536870913],[71,\"^1\",true,536870913],[72,\"^1\",true,536870913],[73,\"^1\",true,536870913],[74,\"^1\",true,536870913],[75,\"^1\",true,536870913],[76,\"^1\",true,536870913],[77,\"^1\",true,536870913],[78,\"^1\",true,536870913],[79,\"^1\",true,536870913],[80,\"^1\",true,536870913],[81,\"^1\",true,536870913],[82,\"^1\",true,536870913]]]", + "addresses": null + }, + { + "addr": 1000374, + "content": "[\"^ \",\"~:keys\",[[83,\"~:logseq.property/built-in?\",true,536870913],[84,\"^1\",true,536870913],[85,\"^1\",true,536870913],[86,\"^1\",true,536870913],[87,\"^1\",true,536870913],[88,\"^1\",true,536870913],[89,\"^1\",true,536870913],[90,\"^1\",true,536870913],[91,\"^1\",true,536870913],[92,\"^1\",true,536870913],[93,\"^1\",true,536870913],[94,\"^1\",true,536870913],[95,\"^1\",true,536870913],[96,\"^1\",true,536870913],[97,\"^1\",true,536870913],[98,\"^1\",true,536870913]]]", + "addresses": null + }, + { + "addr": 1000375, + "content": "[\"^ \",\"~:keys\",[[99,\"~:logseq.property/built-in?\",true,536870913],[100,\"^1\",true,536870913],[101,\"^1\",true,536870913],[102,\"^1\",true,536870913],[103,\"^1\",true,536870913],[104,\"^1\",true,536870913],[105,\"^1\",true,536870913],[106,\"^1\",true,536870913],[107,\"^1\",true,536870913],[108,\"^1\",true,536870913],[109,\"^1\",true,536870913],[110,\"^1\",true,536870913],[111,\"^1\",true,536870913],[112,\"^1\",true,536870913],[113,\"^1\",true,536870913],[114,\"^1\",true,536870913]]]", + "addresses": null + }, + { + "addr": 1000376, + "content": "[\"^ \",\"~:keys\",[[115,\"~:logseq.property/built-in?\",true,536870913],[116,\"^1\",true,536870913],[117,\"^1\",true,536870913],[118,\"^1\",true,536870913],[119,\"^1\",true,536870913],[120,\"^1\",true,536870913],[121,\"^1\",true,536870913],[122,\"^1\",true,536870913],[123,\"^1\",true,536870913],[124,\"^1\",true,536870913],[125,\"^1\",true,536870913],[126,\"^1\",true,536870913],[127,\"^1\",true,536870913],[128,\"^1\",true,536870913],[129,\"^1\",true,536870913],[130,\"^1\",true,536870913]]]", + "addresses": null + }, + { + "addr": 1000377, + "content": "[\"^ \",\"~:keys\",[[131,\"~:logseq.property/built-in?\",true,536870913],[132,\"^1\",true,536870913],[133,\"^1\",true,536870913],[134,\"^1\",true,536870913],[135,\"^1\",true,536870913],[136,\"^1\",true,536870913],[137,\"^1\",true,536870913],[138,\"^1\",true,536870913],[139,\"^1\",true,536870913],[140,\"^1\",true,536870913],[141,\"^1\",true,536870913],[142,\"^1\",true,536870913],[143,\"^1\",true,536870913],[144,\"^1\",true,536870913],[145,\"^1\",true,536870913],[146,\"^1\",true,536870913]]]", + "addresses": null + }, + { + "addr": 1000378, + "content": "[\"^ \",\"~:keys\",[[147,\"~:logseq.property/built-in?\",true,536870913],[148,\"^1\",true,536870913],[149,\"^1\",true,536870913],[150,\"^1\",true,536870913],[151,\"^1\",true,536870913],[152,\"^1\",true,536870913],[153,\"^1\",true,536870913],[154,\"^1\",true,536870913],[155,\"^1\",true,536870913],[156,\"^1\",true,536870913],[157,\"^1\",true,536870913],[158,\"^1\",true,536870913],[159,\"^1\",true,536870913],[160,\"^1\",true,536870913],[161,\"^1\",true,536870913],[162,\"^1\",true,536870913]]]", + "addresses": null + }, + { + "addr": 1000379, + "content": "[\"^ \",\"~:keys\",[[163,\"~:logseq.property/built-in?\",true,536870913],[164,\"^1\",true,536870913],[165,\"^1\",true,536870913],[166,\"^1\",true,536870913],[167,\"^1\",true,536870913],[168,\"^1\",true,536870913],[169,\"^1\",true,536870913],[170,\"^1\",true,536870913],[171,\"^1\",true,536870913],[172,\"^1\",true,536870913],[173,\"^1\",true,536870913],[174,\"^1\",true,536870913],[175,\"^1\",true,536870913],[176,\"^1\",true,536870913],[177,\"^1\",true,536870913],[178,\"^1\",true,536870913],[179,\"^1\",true,536870913],[185,\"^1\",true,536870913],[186,\"^1\",true,536870913],[187,\"^1\",true,536870913],[188,\"^1\",true,536870913],[189,\"^1\",true,536870913],[190,\"^1\",true,536870913],[76,\"~:logseq.property/choice-checkbox-state\",false,536870913],[79,\"^2\",true,536870913],[106,\"~:logseq.property/classes\",4,536870913],[164,\"^3\",5,536870913],[40,\"~:logseq.property/created-from-property\",35,536870913],[44,\"^4\",35,536870913],[46,\"^4\",35,536870913],[50,\"^4\",35,536870913]]]", + "addresses": null + }, + { + "addr": 1000380, + "content": "[\"^ \",\"~:keys\",[[87,\"~:logseq.property/created-from-property\",35,536870913],[89,\"^1\",35,536870913],[140,\"^1\",35,536870913],[149,\"^1\",35,536870913],[163,\"^1\",35,536870913],[57,\"^1\",56,536870913],[58,\"^1\",56,536870913],[59,\"^1\",56,536870913],[60,\"^1\",56,536870913],[61,\"^1\",56,536870913],[75,\"^1\",74,536870913],[76,\"^1\",74,536870913],[77,\"^1\",74,536870913],[78,\"^1\",74,536870913],[79,\"^1\",74,536870913],[80,\"^1\",74,536870913],[82,\"^1\",81,536870913],[83,\"^1\",81,536870913],[84,\"^1\",81,536870913],[85,\"^1\",81,536870913]]]", + "addresses": null + }, + { + "addr": 1000381, + "content": "[\"^ \",\"~:keys\",[[91,\"~:logseq.property/created-from-property\",90,536870913],[93,\"^1\",92,536870913],[94,\"^1\",92,536870913],[95,\"^1\",92,536870913],[96,\"^1\",92,536870913],[97,\"^1\",92,536870913],[98,\"^1\",92,536870913],[101,\"^1\",100,536870913],[102,\"^1\",100,536870913],[103,\"^1\",100,536870913],[112,\"^1\",111,536870913],[113,\"^1\",111,536870913],[114,\"^1\",111,536870913],[74,\"~:logseq.property/default-value\",76,536870913],[90,\"^2\",91,536870913],[92,\"^2\",95,536870913],[100,\"^2\",103,536870913],[111,\"^2\",112,536870913],[39,\"~:logseq.property/description\",40,536870913],[43,\"^3\",44,536870913],[45,\"^3\",46,536870913]]]", + "addresses": null + }, + { + "addr": 1000382, + "content": "[\"^ \",\"~:keys\",[[49,\"~:logseq.property/description\",50,536870913],[86,\"^1\",87,536870913],[88,\"^1\",89,536870913],[139,\"^1\",140,536870913],[148,\"^1\",149,536870913],[162,\"^1\",163,536870913],[74,\"~:logseq.property/hide-empty-value\",true,536870913],[81,\"^2\",true,536870913],[86,\"^2\",true,536870913],[88,\"^2\",true,536870913],[90,\"^2\",true,536870913],[92,\"^2\",true,536870913],[100,\"^2\",true,536870913],[106,\"^2\",true,536870913],[132,\"~:logseq.property/hide?\",false,536870913],[143,\"^3\",false,536870913],[144,\"^3\",false,536870913],[145,\"^3\",false,536870913],[146,\"^3\",false,536870913],[147,\"^3\",false,536870913],[12,\"^3\",true,536870913],[13,\"^3\",true,536870913],[14,\"^3\",true,536870913],[15,\"^3\",true,536870913],[16,\"^3\",true,536870913],[17,\"^3\",true,536870913],[18,\"^3\",true,536870913]]]", + "addresses": null + }, + { + "addr": 1000383, + "content": "[\"^ \",\"~:keys\",[[94,\"~:db/ident\",\"~:logseq.property.repeat/recur-unit.hour\",536870913],[119,\"^1\",\"~:logseq.property.view/gallery-card-size\",536870913],[18,\"~:logseq.property/built-in?\",true,536870913],[34,\"^4\",true,536870913],[50,\"^4\",true,536870913],[66,\"^4\",true,536870913],[82,\"^4\",true,536870913],[98,\"^4\",true,536870913],[114,\"^4\",true,536870913],[130,\"^4\",true,536870913],[146,\"^4\",true,536870913],[162,\"^4\",true,536870913],[50,\"~:logseq.property/created-from-property\",35,536870913],[85,\"^5\",81,536870913],[45,\"~:logseq.property/description\",46,536870913],[18,\"~:logseq.property/hide?\",true,536870913]]]", + "addresses": "[1000367,1000368,1000369,1000370,1000371,1000372,1000373,1000374,1000375,1000376,1000377,1000378,1000379,1000380,1000381,1000382]" + }, + { + "addr": 1000384, + "content": "[\"^ \",\"~:keys\",[[19,\"~:logseq.property/hide?\",true,536870913],[20,\"^1\",true,536870913],[23,\"^1\",true,536870913],[24,\"^1\",true,536870913],[25,\"^1\",true,536870913],[26,\"^1\",true,536870913],[27,\"^1\",true,536870913],[28,\"^1\",true,536870913],[29,\"^1\",true,536870913],[30,\"^1\",true,536870913],[31,\"^1\",true,536870913],[32,\"^1\",true,536870913],[33,\"^1\",true,536870913],[34,\"^1\",true,536870913],[36,\"^1\",true,536870913],[37,\"^1\",true,536870913]]]", + "addresses": null + }, + { + "addr": 1000385, + "content": "[\"^ \",\"~:keys\",[[38,\"~:logseq.property/hide?\",true,536870913],[48,\"^1\",true,536870913],[51,\"^1\",true,536870913],[52,\"^1\",true,536870913],[53,\"^1\",true,536870913],[54,\"^1\",true,536870913],[55,\"^1\",true,536870913],[56,\"^1\",true,536870913],[62,\"^1\",true,536870913],[63,\"^1\",true,536870913],[64,\"^1\",true,536870913],[65,\"^1\",true,536870913],[66,\"^1\",true,536870913],[67,\"^1\",true,536870913],[68,\"^1\",true,536870913],[70,\"^1\",true,536870913]]]", + "addresses": null + }, + { + "addr": 1000386, + "content": "[\"^ \",\"~:keys\",[[71,\"~:logseq.property/hide?\",true,536870913],[72,\"^1\",true,536870913],[73,\"^1\",true,536870913],[99,\"^1\",true,536870913],[104,\"^1\",true,536870913],[105,\"^1\",true,536870913],[108,\"^1\",true,536870913],[110,\"^1\",true,536870913],[111,\"^1\",true,536870913],[115,\"^1\",true,536870913],[116,\"^1\",true,536870913],[117,\"^1\",true,536870913],[118,\"^1\",true,536870913],[119,\"^1\",true,536870913],[120,\"^1\",true,536870913],[121,\"^1\",true,536870913]]]", + "addresses": null + }, + { + "addr": 1000387, + "content": "[\"^ \",\"~:keys\",[[122,\"~:logseq.property/hide?\",true,536870913],[123,\"^1\",true,536870913],[124,\"^1\",true,536870913],[125,\"^1\",true,536870913],[126,\"^1\",true,536870913],[127,\"^1\",true,536870913],[128,\"^1\",true,536870913],[129,\"^1\",true,536870913],[130,\"^1\",true,536870913],[131,\"^1\",true,536870913],[133,\"^1\",true,536870913],[134,\"^1\",true,536870913],[135,\"^1\",true,536870913],[136,\"^1\",true,536870913],[137,\"^1\",true,536870913],[138,\"^1\",true,536870913]]]", + "addresses": null + }, + { + "addr": 1000388, + "content": "[\"^ \",\"~:keys\",[[139,\"~:logseq.property/hide?\",true,536870913],[141,\"^1\",true,536870913],[142,\"^1\",true,536870913],[150,\"^1\",true,536870913],[151,\"^1\",true,536870913],[152,\"^1\",true,536870913],[153,\"^1\",true,536870913],[154,\"^1\",true,536870913],[155,\"^1\",true,536870913],[156,\"^1\",true,536870913],[157,\"^1\",true,536870913],[158,\"^1\",true,536870913],[159,\"^1\",true,536870913],[160,\"^1\",true,536870913],[161,\"^1\",true,536870913],[162,\"^1\",true,536870913],[164,\"^1\",true,536870913],[166,\"^1\",true,536870913],[187,\"^1\",true,536870913],[188,\"^1\",true,536870913],[189,\"^1\",true,536870913],[190,\"^1\",true,536870913],[114,\"~:logseq.property/icon\",[\"^ \",\"~:type\",\"~:tabler-icon\",\"~:id\",\"layout-grid\"],536870913],[172,\"^2\",[\"^ \",\"^3\",\"^4\",\"^5\",\"search\"],536870913],[174,\"^2\",[\"^ \",\"^3\",\"^4\",\"^5\",\"search\"],536870913],[112,\"^2\",[\"^ \",\"^3\",\"^4\",\"^5\",\"table\"],536870913],[113,\"^2\",[\"^ \",\"^3\",\"^4\",\"^5\",\"list\"],536870913],[170,\"^2\",[\"^ \",\"^3\",\"^4\",\"^5\",\"message-circle\"],536870913],[19,\"~:logseq.property/public?\",false,536870913],[20,\"^6\",false,536870913]]]", + "addresses": null + }, + { + "addr": 1000389, + "content": "[\"^ \",\"~:keys\",[[23,\"~:logseq.property/public?\",false,536870913],[24,\"^1\",false,536870913],[25,\"^1\",false,536870913],[26,\"^1\",false,536870913],[27,\"^1\",false,536870913],[28,\"^1\",false,536870913],[29,\"^1\",false,536870913],[30,\"^1\",false,536870913],[31,\"^1\",false,536870913],[32,\"^1\",false,536870913],[33,\"^1\",false,536870913],[34,\"^1\",false,536870913],[36,\"^1\",false,536870913],[37,\"^1\",false,536870913],[38,\"^1\",false,536870913],[68,\"^1\",false,536870913]]]", + "addresses": null + }, + { + "addr": 1000390, + "content": "[\"^ \",\"~:keys\",[[69,\"~:logseq.property/public?\",false,536870913],[71,\"^1\",false,536870913],[72,\"^1\",false,536870913],[90,\"^1\",false,536870913],[92,\"^1\",false,536870913],[100,\"^1\",false,536870913],[111,\"^1\",false,536870913],[115,\"^1\",false,536870913],[116,\"^1\",false,536870913],[117,\"^1\",false,536870913],[118,\"^1\",false,536870913],[119,\"^1\",false,536870913],[120,\"^1\",false,536870913],[121,\"^1\",false,536870913],[122,\"^1\",false,536870913],[123,\"^1\",false,536870913]]]", + "addresses": null + }, + { + "addr": 1000391, + "content": "[\"^ \",\"~:keys\",[[124,\"~:logseq.property/public?\",false,536870913],[125,\"^1\",false,536870913],[126,\"^1\",false,536870913],[127,\"^1\",false,536870913],[128,\"^1\",false,536870913],[129,\"^1\",false,536870913],[130,\"^1\",false,536870913],[131,\"^1\",false,536870913],[133,\"^1\",false,536870913],[134,\"^1\",false,536870913],[135,\"^1\",false,536870913],[136,\"^1\",false,536870913],[137,\"^1\",false,536870913],[138,\"^1\",false,536870913],[139,\"^1\",false,536870913],[141,\"^1\",false,536870913],[142,\"^1\",false,536870913],[143,\"^1\",false,536870913],[144,\"^1\",false,536870913],[155,\"^1\",false,536870913],[156,\"^1\",false,536870913],[157,\"^1\",false,536870913],[158,\"^1\",false,536870913],[159,\"^1\",false,536870913],[160,\"^1\",false,536870913],[161,\"^1\",false,536870913],[164,\"^1\",false,536870913],[166,\"^1\",false,536870913],[21,\"^1\",true,536870913],[22,\"^1\",true,536870913],[35,\"^1\",true,536870913],[39,\"^1\",true,536870913]]]", + "addresses": null + }, + { + "addr": 1000392, + "content": "[\"^ \",\"~:keys\",[[41,\"~:logseq.property/public?\",true,536870913],[42,\"^1\",true,536870913],[43,\"^1\",true,536870913],[45,\"^1\",true,536870913],[47,\"^1\",true,536870913],[48,\"^1\",true,536870913],[49,\"^1\",true,536870913],[74,\"^1\",true,536870913],[81,\"^1\",true,536870913],[86,\"^1\",true,536870913],[88,\"^1\",true,536870913],[106,\"^1\",true,536870913],[108,\"^1\",true,536870913],[109,\"^1\",true,536870913],[110,\"^1\",true,536870913],[132,\"^1\",true,536870913],[145,\"^1\",true,536870913],[146,\"^1\",true,536870913],[147,\"^1\",true,536870913],[148,\"^1\",true,536870913],[162,\"^1\",true,536870913],[165,\"^1\",true,536870913],[119,\"~:logseq.property/scalar-default-value\",\"~:default\",536870913],[123,\"^2\",true,536870913],[20,\"~:logseq.property/type\",\"~:any\",536870913]]]", + "addresses": null + }, + { + "addr": 1000393, + "content": "[\"^ \",\"~:keys\",[[38,\"~:logseq.property/type\",\"~:any\",536870913],[52,\"^1\",\"^2\",536870913],[153,\"^1\",\"^2\",536870913],[13,\"^1\",\"~:checkbox\",536870913],[14,\"^1\",\"^3\",536870913],[16,\"^1\",\"^3\",536870913],[25,\"^1\",\"^3\",536870913],[43,\"^1\",\"^3\",536870913],[45,\"^1\",\"^3\",536870913],[47,\"^1\",\"^3\",536870913],[70,\"^1\",\"^3\",536870913],[99,\"^1\",\"^3\",536870913],[108,\"^1\",\"^3\",536870913],[110,\"^1\",\"^3\",536870913],[123,\"^1\",\"^3\",536870913],[148,\"^1\",\"^3\",536870913],[22,\"^1\",\"~:class\",536870913],[39,\"^1\",\"^4\",536870913],[71,\"^1\",\"^4\",536870913],[165,\"^1\",\"^4\",536870913],[124,\"^1\",\"~:coll\",536870913],[127,\"^1\",\"^5\",536870913],[32,\"^1\",\"~:datetime\",536870913],[33,\"^1\",\"^6\",536870913]]]", + "addresses": null + }, + { + "addr": 1000394, + "content": "[\"^ \",\"~:keys\",[[86,\"~:logseq.property/type\",\"~:datetime\",536870913],[88,\"^1\",\"^2\",536870913],[143,\"^1\",\"^2\",536870913],[155,\"^1\",\"^2\",536870913],[35,\"^1\",\"~:default\",536870913],[48,\"^1\",\"^3\",536870913],[51,\"^1\",\"^3\",536870913],[56,\"^1\",\"^3\",536870913],[65,\"^1\",\"^3\",536870913],[74,\"^1\",\"^3\",536870913],[81,\"^1\",\"^3\",536870913],[92,\"^1\",\"^3\",536870913],[100,\"^1\",\"^3\",536870913],[111,\"^1\",\"^3\",536870913],[15,\"^1\",\"~:entity\",536870913],[19,\"^1\",\"^4\",536870913],[23,\"^1\",\"^4\",536870913],[26,\"^1\",\"^4\",536870913],[27,\"^1\",\"^4\",536870913],[28,\"^1\",\"^4\",536870913]]]", + "addresses": null + }, + { + "addr": 1000395, + "content": "[\"^ \",\"~:keys\",[[30,\"~:logseq.property/type\",\"~:entity\",536870913],[37,\"^1\",\"^2\",536870913],[53,\"^1\",\"^2\",536870913],[63,\"^1\",\"^2\",536870913],[150,\"^1\",\"^2\",536870913],[152,\"^1\",\"^2\",536870913],[154,\"^1\",\"^2\",536870913],[156,\"^1\",\"^2\",536870913],[12,\"^1\",\"~:keyword\",536870913],[17,\"^1\",\"^3\",536870913],[18,\"^1\",\"^3\",536870913],[34,\"^1\",\"^3\",536870913],[54,\"^1\",\"^3\",536870913],[55,\"^1\",\"^3\",536870913],[115,\"^1\",\"^3\",536870913],[119,\"^1\",\"^3\",536870913],[126,\"^1\",\"^3\",536870913],[142,\"^1\",\"^3\",536870913],[64,\"^1\",\"~:map\",536870913],[107,\"^1\",\"^4\",536870913],[125,\"^1\",\"^4\",536870913],[128,\"^1\",\"^4\",536870913],[139,\"^1\",\"^4\",536870913],[141,\"^1\",\"^4\",536870913],[144,\"^1\",\"^4\",536870913],[166,\"^1\",\"^4\",536870913],[66,\"^1\",\"~:node\",536870913],[67,\"^1\",\"^5\",536870913]]]", + "addresses": null + }, + { + "addr": 1000396, + "content": "[\"^ \",\"~:keys\",[[68,\"~:logseq.property/type\",\"~:node\",536870913],[72,\"^1\",\"^2\",536870913],[106,\"^1\",\"^2\",536870913],[130,\"^1\",\"^2\",536870913],[157,\"^1\",\"^2\",536870913],[158,\"^1\",\"^2\",536870913],[161,\"^1\",\"^2\",536870913],[164,\"^1\",\"^2\",536870913],[90,\"^1\",\"~:number\",536870913],[21,\"^1\",\"~:page\",536870913],[49,\"^1\",\"^4\",536870913],[41,\"^1\",\"~:property\",536870913],[73,\"^1\",\"^5\",536870913],[104,\"^1\",\"^5\",536870913],[105,\"^1\",\"^5\",536870913],[116,\"^1\",\"^5\",536870913],[117,\"^1\",\"^5\",536870913],[118,\"^1\",\"^5\",536870913],[122,\"^1\",\"^5\",536870913],[129,\"^1\",\"^5\",536870913],[151,\"^1\",\"^5\",536870913],[31,\"^1\",\"~:raw-number\",536870913],[62,\"^1\",\"^6\",536870913],[120,\"^1\",\"^6\",536870913],[121,\"^1\",\"^6\",536870913],[134,\"^1\",\"^6\",536870913],[135,\"^1\",\"^6\",536870913],[136,\"^1\",\"^6\",536870913],[138,\"^1\",\"^6\",536870913],[24,\"^1\",\"~:string\",536870913]]]", + "addresses": null + }, + { + "addr": 1000397, + "content": "[\"^ \",\"~:keys\",[[29,\"~:logseq.property/type\",\"~:string\",536870913],[36,\"^1\",\"^2\",536870913],[42,\"^1\",\"^2\",536870913],[69,\"^1\",\"^2\",536870913],[131,\"^1\",\"^2\",536870913],[132,\"^1\",\"^2\",536870913],[133,\"^1\",\"^2\",536870913],[137,\"^1\",\"^2\",536870913],[145,\"^1\",\"^2\",536870913],[146,\"^1\",\"^2\",536870913],[147,\"^1\",\"^2\",536870913],[159,\"^1\",\"^2\",536870913],[160,\"^1\",\"^2\",536870913],[162,\"^1\",\"^2\",536870913],[109,\"^1\",\"~:url\",536870913],[86,\"~:logseq.property/ui-position\",\"~:block-below\",536870913],[88,\"^4\",\"^5\",536870913],[106,\"^4\",\"^5\",536870913],[74,\"^4\",\"~:block-left\",536870913],[81,\"^4\",\"^6\",536870913],[91,\"~:logseq.property/value\",1,536870913],[34,\"~:logseq.property/view-context\",\"~:block\",536870913],[36,\"^8\",\"^9\",536870913]]]", + "addresses": null + }, + { + "addr": 1000398, + "content": "[\"^ \",\"~:keys\",[[48,\"~:logseq.property/view-context\",\"~:block\",536870913],[39,\"^1\",\"~:class\",536870913],[42,\"^1\",\"^3\",536870913],[43,\"^1\",\"^3\",536870913],[47,\"^1\",\"^3\",536870913],[41,\"^1\",\"~:never\",536870913],[71,\"^1\",\"^4\",536870913],[72,\"^1\",\"^4\",536870913],[21,\"^1\",\"~:page\",536870913],[49,\"^1\",\"^5\",536870913],[108,\"^1\",\"^5\",536870913],[109,\"^1\",\"^5\",536870913],[110,\"^1\",\"^5\",536870913],[37,\"^1\",\"~:property\",536870913],[38,\"^1\",\"^6\",536870913],[45,\"^1\",\"^6\",536870913]]]", + "addresses": null + }, + { + "addr": 1000399, + "content": "[\"^ \",\"~:keys\",[[148,\"~:logseq.property/view-context\",\"~:property\",536870913],[2,\"~:logseq.property.class/extends\",1,536870913],[3,\"^3\",1,536870913],[4,\"^3\",1,536870913],[5,\"^3\",1,536870913],[169,\"^3\",1,536870913],[170,\"^3\",1,536870913],[171,\"^3\",1,536870913],[172,\"^3\",1,536870913],[173,\"^3\",1,536870913],[175,\"^3\",1,536870913],[176,\"^3\",1,536870913],[177,\"^3\",1,536870913],[178,\"^3\",1,536870913],[179,\"^3\",1,536870913],[167,\"^3\",4,536870913],[168,\"^3\",4,536870913],[174,\"^3\",172,536870913],[170,\"~:logseq.property.class/hide-from-node\",true,536870913]]]", + "addresses": null + }, + { + "addr": 1000400, + "content": "[\"^ \",\"~:keys\",[[171,\"~:logseq.property.class/hide-from-node\",true,536870913],[175,\"^1\",true,536870913],[176,\"^1\",true,536870913],[177,\"^1\",true,536870913],[178,\"^1\",true,536870913],[179,\"^1\",true,536870913],[176,\"~:logseq.property.class/properties\",34,536870913],[177,\"^2\",34,536870913],[178,\"^2\",34,536870913],[176,\"^2\",36,536870913],[172,\"^2\",48,536870913],[179,\"^2\",53,536870913],[179,\"^2\",54,536870913],[179,\"^2\",55,536870913],[179,\"^2\",56,536870913],[179,\"^2\",62,536870913],[179,\"^2\",63,536870913],[179,\"^2\",64,536870913],[170,\"^2\",68,536870913],[169,\"^2\",74,536870913],[169,\"^2\",81,536870913],[169,\"^2\",86,536870913],[169,\"^2\",88,536870913],[175,\"^2\",131,536870913],[175,\"^2\",134,536870913],[175,\"^2\",137,536870913],[173,\"^2\",143,536870913],[173,\"^2\",144,536870913],[5,\"^2\",165,536870913],[167,\"~:logseq.property.journal/title-format\",\"MMM do, yyyy\",536870913],[175,\"~:logseq.property.view/type\",114,536870913]]]", + "addresses": null + }, + { + "addr": 1000401, + "content": "[\"^ \",\"~:keys\",[[37,\"~:logseq.property/hide?\",true,536870913],[70,\"^1\",true,536870913],[121,\"^1\",true,536870913],[138,\"^1\",true,536870913],[20,\"~:logseq.property/public?\",false,536870913],[68,\"^2\",false,536870913],[123,\"^2\",false,536870913],[39,\"^2\",true,536870913],[20,\"~:logseq.property/type\",\"~:any\",536870913],[33,\"^3\",\"~:datetime\",536870913],[28,\"^3\",\"~:entity\",536870913],[67,\"^3\",\"~:node\",536870913],[24,\"^3\",\"~:string\",536870913],[36,\"~:logseq.property/view-context\",\"~:block\",536870913],[45,\"^9\",\"~:property\",536870913],[170,\"~:logseq.property.class/hide-from-node\",true,536870913],[175,\"~:logseq.property.view/type\",114,536870913]]]", + "addresses": "[1000384,1000385,1000386,1000387,1000388,1000389,1000390,1000391,1000392,1000393,1000394,1000395,1000396,1000397,1000398,1000399,1000400]" + }, + { + "addr": 1000402, + "content": "[\"^ \",\"~:keys\",[[71,\"~:block/order\",\"ax\",536870913],[13,\"~:block/updated-at\",1786627375714,536870913],[47,\"~:db/ident\",\"~:logseq.property.class/hide-from-node\",536870913],[18,\"~:logseq.property/hide?\",true,536870913],[175,\"~:logseq.property.view/type\",114,536870913]]]", + "addresses": "[1000312,1000340,1000366,1000383,1000401]" + } + ] +} diff --git a/logseq_sync/test/production_transport/tls_peer.ml b/logseq_sync/test/production_transport/tls_peer.ml new file mode 100644 index 00000000..0fa24c4f --- /dev/null +++ b/logseq_sync/test/production_transport/tls_peer.ml @@ -0,0 +1,230 @@ +let rec remove path = + match Unix.lstat path with + | { st_kind = Unix.S_DIR; _ } -> + Array.iter (fun name -> remove (Filename.concat path name)) (Sys.readdir path); + Unix.rmdir path + | _ -> Unix.unlink path + | exception Unix.Unix_error (Unix.ENOENT, _, _) -> () +;; + +let with_directory callback = + let path = Filename.temp_file "native-http-peer-" "" in + Sys.remove path; + Unix.mkdir path 0o700; + Fun.protect ~finally:(fun () -> remove path) (fun () -> callback path) +;; + +let peer = + {python| +import base64, gzip, hashlib, json, os, select, socket, ssl, struct, subprocess, sys, time, threading +root = sys.argv[1] +with open(root+'/cases.json') as f: cases = json.load(f) +key, cert = root + '/key.pem', root + '/cert.pem' +subprocess.run(['openssl','req','-x509','-newkey','ec','-pkeyopt','ec_paramgen_curve:P-256', + '-nodes','-keyout',key,'-out',cert,'-days','1','-subj','/CN=localhost'], + check=True, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL) +ctx = ssl.SSLContext(ssl.PROTOCOL_TLS_SERVER) +ctx.load_cert_chain(cert,key) +for attempt in range(16): + listener = socket.socket(socket.AF_INET6) + listener.setsockopt(socket.IPPROTO_IPV6, socket.IPV6_V6ONLY, 1) + listener4 = socket.socket() + try: + listener.bind(('::1',0)) + listener4.bind(('127.0.0.1', listener.getsockname()[1])) + break + except OSError: + listener.close(); listener4.close() + if attempt == 15: raise +listener.listen(8); listener.settimeout(5); listener4.listen(8) +with open(root+'/ready.tmp','w') as f: f.write(str(listener.getsockname()[1])) +os.replace(root+'/ready.tmp', root+'/ready') +report=[]; stalled=[] +def observe_stalled_tls(raw, item, index): + try: + hello=b'' + while True: + part=raw.recv(16384) + if not part: + item['closed']=True; break + if not item['client_hello']: + hello+=part + if len(hello)>5: + record_length=int.from_bytes(hello[3:5],'big') + if len(hello)>=5+record_length and hello[0]==22 and hello[5]==1: + item['client_hello']=True + path=root+'/client-hello-'+str(index) + with open(path+'.tmp','w') as f: f.write('ClientHello observed') + os.replace(path+'.tmp',path) + except (ConnectionResetError, BrokenPipeError): item['closed']=True + except socket.timeout: pass + finally: raw.close() +def serve_case(index,case,raw): + raw.settimeout(2) + if case.get('stall_tls'): + item={'closed':False,'client_hello':False}; report.append(item) + worker=threading.Thread(target=observe_stalled_tls,args=(raw,item,index),daemon=True) + worker.start(); stalled.append(worker); return + time.sleep(case.get('tls_delay',0)) + try: conn=ctx.wrap_socket(raw,server_side=True) + except (ssl.SSLError,OSError): + raw.close(); report.append({'closed':True,'frames':[]}); return + request=b'' + while b'\r\n\r\n' not in request: + part=conn.recv(16384) + if not part: raise RuntimeError('no request') + request+=part + header_end=request.index(b'\r\n\r\n')+4 + request_headers=dict(line.split(b':',1) for line in request[:header_end].split(b'\r\n')[1:] if b':' in line) + length=next((int(v.strip()) for k,v in request_headers.items() if k.lower()==b'content-length'),0) + while len(request)=2: + length=pending[1]&127; offset=2 + if length==126: + if len(pending)<4: break + length=int.from_bytes(pending[2:4],'big'); offset=4 + elif length==127: + if len(pending)<10: break + length=int.from_bytes(pending[2:10],'big'); offset=10 + masked=bool(pending[1]&128) + if len(pending) Seq.map (fun c -> Printf.sprintf "%02x" (Char.code c)) + |> List.of_seq + |> String.concat "" +;; + +let case ?(ws = false) ?(extra = "") ?(segment = 16384) ?(eof = false) wire = + `Assoc + [ "wire", `String (hex wire) + ; "ws", `Bool ws + ; "extra", `String extra + ; "segment", `Int segment + ; "eof", `Bool eof + ] +;; + +let wait clock predicate = + Eio.Time.with_timeout_exn clock 5. (fun () -> + while not (predicate ()) do + Eio.Time.sleep clock 0.005 + done) +;; + +let read_file path = + let input = open_in_bin path in + Fun.protect + ~finally:(fun () -> close_in_noerr input) + (fun () -> really_input_string input (in_channel_length input)) +;; + +let scaled_clock clock scale = + let module Clock = struct + type t = unit + type time = float + + let now () = Eio.Time.now clock *. scale + let sleep_until () deadline = Eio.Time.sleep_until clock (deadline /. scale) + end + in + Eio.Resource.T ((), Eio.Time.Pi.clock (module Clock)) +;; + +let with_peer cases f = + with_directory (fun support -> + Eio_main.run (fun environment -> + Eio.Switch.run (fun sw -> + let clock = Eio.Stdenv.clock environment in + Yojson.Basic.to_file (Filename.concat support "cases.json") (`List cases); + let process = + Eio.Process.spawn + ~sw + (Eio.Stdenv.process_mgr environment) + [ "python3"; "-c"; peer; support ] + in + wait clock (fun () -> Sys.file_exists (Filename.concat support "ready")); + let port = int_of_string (read_file (Filename.concat support "ready")) in + let origin = Uri.of_string (Printf.sprintf "https://localhost:%d" port) in + f ~environment ~sw ~support ~origin; + Eio.Time.with_timeout_exn clock 5. (fun () -> Eio.Process.await_exn process); + Yojson.Basic.from_file (Filename.concat support "report")))) +;; diff --git a/logseq_sync/test/production_transport/transport_owner_contract.ml b/logseq_sync/test/production_transport/transport_owner_contract.ml new file mode 100644 index 00000000..e76fa81e --- /dev/null +++ b/logseq_sync/test/production_transport/transport_owner_contract.ml @@ -0,0 +1,574 @@ +module D = Sync_dsl.Core +module Transport = Logseq_sync_transport.Core +module P = Logseq_sync_protocol.Sync_protocol +module O = Logseq_overlay_db.Types + +let get = function + | Ok value -> value + | Error error -> Alcotest.fail (D.error_message error) +;; + +let cursor n = Result.get_ok (O.Server_cursor.of_int64 n) + +let identity = + get + (D.identity + ~origin:"https://example.com" + ~account:"a" + ~graph:"g" + ~account_generation:0L + ~graph_generation:0L) +;; + +let limits = + get + (D.limits + ~maximum_items:8 + ~maximum_bytes:1024 + ~maximum_pending:8 + ~maximum_resources:8 + ~steps_per_turn:8 + ~maximum_tasks:8 + ~maximum_messages:4 + ~maximum_message_bytes:1024 + ~maximum_demands:8 + ~maximum_active_assets:2 + ~foreground_reserve:1 + ~foreground_demand_reserve:1 + ~foreground_descriptor_reserve_bytes:128 + ~asset_cache_budget_bytes:4096L + ~asset_maximum_file_bytes:1024 + ~asset_pending_budget_bytes:4096L + ~snapshot_maximum_artifact_bytes:4096L + ~snapshot_staging_budget_bytes:4096L + ~snapshot_scratch_budget_bytes:4096L) +;; + +let expect_error predicate = function + | Error error when predicate error -> () + | Error error -> Alcotest.fail (D.error_message error) + | Ok _ -> Alcotest.fail "Expected error" +;; + +let protocol = function + | D.Protocol_failure _ -> true + | _ -> false +;; + +type runner = { run : 'a. 'a D.transport -> ('a, D.error) result } + +let with_owner + ?(on_send = fun _ _ -> ()) + ?(connect = fun () -> Ok ()) + ?(token = fun ~epoch:_ ~refresh:_ -> Ok "synthetic") + ?(fetch_snapshot = + fun ~scope:_ _ -> Error (D.Storage_failure "synthetic snapshot unavailable")) + callback + = + Eio_main.run (fun env -> + Eio.Switch.run (fun sw -> + let stopped = ref 0 + and joined = ref 0 + and sent = ref [] in + let message = ref (fun (_ : string) -> ()) + and closed = ref (fun (_ : D.connection_end) -> ()) in + let fail_handoff = ref false + and ledger = ref [] in + let backend : Transport.backend = + { connect = + (fun ~token:_ ~on_message ~on_close -> + message := on_message; + closed := on_close; + Result.map + (fun () -> + { Transport.activate = (fun () -> Ok ()) + ; send = + (fun wire -> + sent := wire :: !sent; + on_send on_message wire; + Ok ()) + ; stop = (fun () -> incr stopped) + ; await_stopped = (fun () -> incr joined) + }) + (connect ())) + } + in + let owner = + get + (Transport.create + ~sw + ~clock:(Eio.Stdenv.mono_clock env) + ~identity + ~limits + ~deadlines: + { setup_seconds = 0.03 + ; tls_handshake_seconds = 0.03 + ; http_response_seconds = 0.03 + ; download_seconds = 0.1 + } + ~cleanup_seconds:0.1 + ~backend + ~token + ~fetch_snapshot) + in + get (Transport.authorize owner ~identity ~epoch:1L); + let scope : Transport.scope = + { check_current = (fun () -> Ok ()) + ; admit_network = (fun () -> Ok ()) + ; handoff = + (fun references -> + if !fail_handoff + then Error D.Stale_context + else ( + ledger := Rrbvec.to_list references @ !ledger; + Ok ())) + } + in + let run = + { run = (fun operation -> Transport.run owner ~scope ~identity operation) } + in + callback owner run message closed stopped joined sent ledger fail_handoff)) +;; + +let ownership () = + with_owner (fun owner run _ _ stopped joined _ ledger _ -> + let connection = get (run.run D.Connect) in + Alcotest.(check int) "handoff" 1 (List.length !ledger); + get (Transport.validate_resource owner (D.Connection_reference connection)); + let forged = get (D.connection ~owner:identity ~token:"forged" ~generation:1L) in + expect_error + (function + | D.Stale_context -> true + | _ -> false) + (run.run (D.Receive forged)); + get (run.run (D.Retire_connection connection)); + get (run.run (D.Retire_connection connection)); + Alcotest.(check int) "one stop" 1 !stopped; + Alcotest.(check int) "physical join" 1 !joined) +;; + +let failed_handoff () = + with_owner (fun _ run _ _ stopped joined _ _ fail -> + fail := true; + expect_error + (function + | D.Stale_context -> true + | _ -> false) + (run.run D.Connect); + Alcotest.(check int) "untransferred socket stopped" 1 !stopped; + Alcotest.(check int) "untransferred socket joined" 1 !joined) +;; + +let notifications () = + with_owner (fun _ run message _ _ _ _ _ _ -> + let connection = get (run.run D.Connect) in + let emit n = + !message + (Result.get_ok (P.encode_server_message (P.Server.Changed { t = cursor n }))) + in + emit 2L; + emit 1L; + emit 3L; + match (get (run.run (D.Receive connection))).message with + | D.Remote_changed head -> + Alcotest.(check int64) + "greatest adjacent target" + 3L + (O.Server_cursor.to_int64 head.through) + | _ -> Alcotest.fail "Expected changed hint") +;; + +let timeout () = + with_owner (fun _ run _ _ stopped joined _ _ _ -> + let connection = get (run.run D.Connect) in + expect_error + (function + | D.Network_failure _ -> true + | _ -> false) + (run.run (D.On_connection (connection, D.Head))); + Alcotest.(check int) "timeout stops" 1 !stopped; + Alcotest.(check int) "timeout joins" 1 !joined) +;; + +let scoped_matching () = + with_owner (fun _ run message _ _ _ _ _ _ -> + let connection = get (run.run D.Connect) in + !message + (Result.get_ok + (P.encode_server_message + (P.Server.Tx_batch_ok { t = cursor 1L; checksum = Some "0123456789abcdef" }))); + expect_error protocol (run.run (D.On_connection (connection, D.Head)))) +;; + +let auth_epoch () = + with_owner (fun owner run _ _ stopped joined _ _ _ -> + let connection = get (run.run D.Connect) in + get (Transport.fence owner ~identity); + expect_error + (function + | D.Network_closed -> true + | _ -> false) + (run.run (D.On_connection (connection, D.Head))); + get (Transport.close_network owner ~identity); + Alcotest.(check int) "epoch close" 1 !stopped; + Alcotest.(check int) "epoch join" 1 !joined; + expect_error + (function + | D.Stale_context -> true + | _ -> false) + (Transport.authorize owner ~identity ~epoch:1L); + get (Transport.authorize owner ~identity ~epoch:2L)) +;; + +let invalid_nesting () = + with_owner (fun _ run _ _ _ _ _ _ _ -> + let connection = get (run.run D.Connect) in + expect_error + (function + | D.Invalid_input _ -> true + | _ -> false) + (run.run (D.On_connection (connection, D.Receive connection))); + expect_error + (function + | D.Invalid_input _ -> true + | _ -> false) + (run.run (D.On_connection (connection, D.On_connection (connection, D.Head))))) +;; + +let emit callback message = callback (Result.get_ok (P.encode_server_message message)) + +let fresh_head () = + let revision = ref 0L in + with_owner + ~on_send:(fun callback wire -> + match Result.get_ok (P.decode_client_message wire) with + | P.Client.Hello _ -> + revision := Int64.succ !revision; + emit callback (P.Server.Hello { t = cursor !revision; checksum = None }) + | _ -> ()) + (fun _ run _ _ _ _ _ _ _ -> + let connection = get (run.run D.Connect) in + let first = get (run.run (D.On_connection (connection, D.Head))) in + let second = get (run.run D.Head) in + Alcotest.(check int64) "first head" 1L (O.Server_cursor.to_int64 first.through); + Alcotest.(check int64) + "fresh second head" + 2L + (O.Server_cursor.to_int64 second.through)) +;; + +let uuid index = + Result.get_ok + (O.Graph.Uuid.of_string (Printf.sprintf "00000000-0000-4000-8000-%012d" index)) +;; + +let batch = + let wires = + List.init 3 (fun index -> + Result.get_ok + (O.submission_wire + ~maximum_bytes:1024 + ~mutation_id:(uuid (index + 1)) + ~operation:O.Save_block_operation + ~protected_transaction:("frozen-wire-" ^ string_of_int index))) + in + Result.get_ok + (O.submission_batch + ~maximum_wires:8 + ~maximum_bytes:1024 + ~batch_id: + (Result.get_ok (O.Submission_batch_id.of_string "submission-batch:v1:synthetic")) + ~t_before:(cursor 1L) + ~wires) +;; + +let refresh_once () = + let attempts = ref 0 + and credentials = ref [] in + with_owner + ~connect:(fun () -> + incr attempts; + if !attempts = 1 then Error Transport.Unauthorized else Ok ()) + ~token:(fun ~epoch ~refresh -> + credentials := (epoch, refresh) :: !credentials; + Ok "synthetic") + (fun _ run _ _ _ _ _ _ _ -> + ignore (get (run.run D.Connect)); + Alcotest.(check int) "one refresh" 2 !attempts; + Alcotest.(check (list (pair int64 bool))) + "same checked epoch" + [ 1L, false; 1L, true ] + (List.rev !credentials)) +;; + +let authentication_is_not_refreshable () = + let attempts = ref 0 in + with_owner + ~connect:(fun () -> + incr attempts; + Error (Transport.Failed (D.Authentication_failure "synthetic certificate failure"))) + (fun _ run _ _ _ _ _ _ _ -> + expect_error + (function + | D.Authentication_failure _ -> true + | _ -> false) + (run.run D.Connect); + Alcotest.(check int) "certificate failure never refreshes" 1 !attempts) +;; + +let refresh_bound () = + let attempts = ref 0 in + with_owner + ~connect:(fun () -> + incr attempts; + Error Transport.Unauthorized) + (fun _ run _ _ _ _ _ _ _ -> + expect_error + (function + | D.Authentication_failure _ -> true + | _ -> false) + (run.run D.Connect); + Alcotest.(check int) "second rejection is terminal" 2 !attempts) +;; + +let pull_validation () = + let response = ref (P.Server.Pull_ok { t = cursor 3L; checksum = None; txs = [] }) in + with_owner + ~on_send:(fun callback wire -> + match Result.get_ok (P.decode_client_message wire) with + | P.Client.Pull _ -> emit callback !response + | _ -> ()) + (fun _ run _ _ _ _ _ _ _ -> + let connection = get (run.run D.Connect) in + (match get (run.run (D.On_connection (connection, D.Pull (cursor 1L)))) with + | D.No_progress { since; through; expected_checksum } -> + Alcotest.(check bool) + "missing checksum remains optional" + true + (Option.is_none expected_checksum); + Alcotest.(check int64) + "no progress baseline" + 1L + (O.Server_cursor.to_int64 since); + Alcotest.(check int64) + "no progress target" + 3L + (O.Server_cursor.to_int64 through) + | _ -> Alcotest.fail "Expected No_progress"); + response + := P.Server.Pull_ok { t = cursor 3L; checksum = Some "0123456789abcdef"; txs = [] }; + (match get (run.run (D.Pull (cursor 1L))) with + | D.No_progress { expected_checksum = Some expected; _ } -> + Alcotest.(check string) + "empty Pull retains checksum" + "checksum:v1:0123456789abcdef" + (O.Checksum.to_string expected) + | _ -> Alcotest.fail "Expected No_progress checksum"); + response + := P.Server.Pull_ok + { t = cursor 3L + ; checksum = None + ; txs = + [ { t = cursor 2L; tx = "wire-2"; outliner_op = None } + ; { t = cursor 3L; tx = "wire-3"; outliner_op = None } + ] + }; + (match get (run.run (D.Pull (cursor 1L))) with + | D.Pulled batch -> + Alcotest.(check int) + "ordered transactions" + 2 + (List.length (O.authoritative_batch_transactions batch)) + | _ -> Alcotest.fail "Expected Pulled"); + response + := P.Server.Pull_ok + { t = cursor 3L + ; checksum = None + ; txs = + [ { t = cursor 1L; tx = "replay"; outliner_op = None } + ; { t = cursor 0L; tx = "invalid replay"; outliner_op = None } + ] + }; + expect_error protocol (run.run (D.Pull (cursor 2L)))) +;; + +let push_matching () = + let observed = ref None in + with_owner + ~on_send:(fun callback wire -> + match Result.get_ok (P.decode_client_message wire) with + | P.Client.Tx_batch value -> + observed + := Some + ( value.client_revision + , value.t_before + , List.map (fun (tx : P.Client.transaction) -> tx.tx) value.txs ); + emit + callback + (P.Server.Tx_batch_ok { t = cursor 4L; checksum = Some "0123456789abcdef" }) + | _ -> ()) + (fun _ run _ _ _ _ _ _ _ -> + let connection = get (run.run D.Connect) in + (match get (run.run (D.On_connection (connection, D.Push batch))) with + | D.Accepted barrier -> + Alcotest.(check int64) + "advancing barrier" + 4L + (O.Server_cursor.to_int64 barrier.through) + | _ -> Alcotest.fail "Expected acceptance"); + match !observed with + | Some (revision, before, wires) -> + Alcotest.(check (option string)) + "durable batch ID" + (Some (O.Submission_batch_id.to_string (O.submission_batch_id batch))) + revision; + Alcotest.(check int64) "frozen baseline" 1L (O.Server_cursor.to_int64 before); + Alcotest.(check (list string)) + "exact durable wires" + [ "frozen-wire-0"; "frozen-wire-1"; "frozen-wire-2" ] + wires + | None -> Alcotest.fail "Missing submission") +;; + +let push_unknown () = + with_owner (fun _ run _ _ stopped joined _ _ _ -> + let connection = get (run.run D.Connect) in + expect_error + (function + | D.Outcome_unknown _ -> true + | _ -> false) + (run.run (D.On_connection (connection, D.Push batch))); + Alcotest.(check int) "unknown send stops" 1 !stopped; + Alcotest.(check int) "unknown send joins" 1 !joined) +;; + +let rejection_partition () = + let rejection : P.rejection = + { reason = P.Invalid_tx + ; t = Some (cursor 2L) + ; checksum = Some "0123456789abcdef" + ; success_tx_ids = [ uuid 1 ] + ; failed_tx_id = Some (uuid 2) + ; missing_block_uuids = [] + ; error_detail = Some "secret rejected wire" + ; data = None + } + in + let response = ref rejection in + with_owner + ~on_send:(fun callback wire -> + match Result.get_ok (P.decode_client_message wire) with + | P.Client.Tx_batch _ -> emit callback (P.Server.Tx_reject !response) + | _ -> ()) + (fun _ run _ _ _ _ _ _ _ -> + ignore (get (run.run D.Connect)); + (match get (run.run (D.Push batch)) with + | D.Rejected (O.Definitive { partition; _ }) -> + Alcotest.(check int) "accepted prefix" 1 (List.length partition.accepted_prefix); + Alcotest.(check bool) + "failed member" + true + (partition.failed_member = Some (uuid 2)); + Alcotest.(check bool) + "unexecuted suffix" + true + (partition.unexecuted_suffix = [ uuid 3 ]); + Alcotest.(check (list string)) + "no raw server diagnostics" + [] + partition.diagnostics + | _ -> Alcotest.fail "Expected definitive rejection"); + response := { rejection with success_tx_ids = [ uuid 3 ] }; + expect_error protocol (run.run (D.Push batch))) +;; + +let snapshot_handoff () = + let fetch_snapshot ~scope through = + let resource = get (D.resource ~owner:identity ~token:"synthetic-snapshot") in + let snapshot = + get + (D.snapshot ~resource ~through ~checksum:"checksum:v1:synthetic" ~encrypted:false) + in + Result.map + (fun () -> snapshot) + (scope.Transport.handoff (Rrbvec.singleton (D.Snapshot_reference snapshot))) + in + with_owner ~fetch_snapshot (fun _ run _ _ _ _ _ ledger _ -> + let connection = get (run.run D.Connect) in + let snapshot = + get (run.run (D.On_connection (connection, D.Fetch_snapshot (cursor 3L)))) + in + Alcotest.(check int64) + "snapshot exact baseline" + 3L + (O.Server_cursor.to_int64 snapshot.through); + Alcotest.(check int) "original ledgers transferred" 2 (List.length !ledger)) +;; + +let retire_joins_waiters () = + with_owner (fun _ run _ _ stopped joined _ _ _ -> + let connection = get (run.run D.Connect) in + let receiver_finished = ref false in + Eio.Fiber.both + (fun () -> + let result = run.run (D.Receive connection) in + receiver_finished := true; + expect_error + (function + | D.Network_closed -> true + | _ -> false) + result) + (fun () -> + Eio.Fiber.yield (); + get (run.run (D.Retire_connection connection))); + Alcotest.(check bool) "receiver physically completed" true !receiver_finished; + Alcotest.(check int) "shared stop" 1 !stopped; + Alcotest.(check int) "shared join" 1 !joined) +;; + +let one_response_family () = + with_owner (fun _ run _ _ _ _ _ _ _ -> + let connection = get (run.run D.Connect) in + Eio.Fiber.both + (fun () -> + expect_error + (function + | D.Network_failure _ -> true + | _ -> false) + (run.run (D.On_connection (connection, D.Head)))) + (fun () -> + Eio.Fiber.yield (); + expect_error + (function + | D.Capacity_exceeded _ -> true + | _ -> false) + (run.run (D.On_connection (connection, D.Pull (cursor 1L)))))) +;; + +let () = + Alcotest.run + "production transport" + [ ( "owner" + , List.map + (fun (name, test) -> Alcotest.test_case name `Quick test) + [ "refresh once", refresh_once + ; "authentication does not refresh", authentication_is_not_refreshable + ; "second unauthorized is terminal", refresh_bound + ; "pull validates before replay filtering", pull_validation + ; "push retains exact durable wires", push_matching + ; "unknown send joins", push_unknown + ; "rejection ordered partition", rejection_partition + ; "snapshot original handoff", snapshot_handoff + ; "retirement joins waiters", retire_joins_waiters + ; "one response family", one_response_family + ; "Head performs a fresh request", fresh_head + ; "original ownership and physical retirement", ownership + ; "failed handoff joins", failed_handoff + ; "bounded hint coalescing", notifications + ; "response deadline joins", timeout + ; "unsolicited response fails", scoped_matching + ; "authorization epoch fence", auth_epoch + ; "invalid nested constructors", invalid_nesting + ] ) + ] +;; diff --git a/logseq_sync/test/pure_reducer_bad_case_01.ml b/logseq_sync/test/pure_reducer_bad_case_01.ml deleted file mode 100644 index 6005c3cb..00000000 --- a/logseq_sync/test/pure_reducer_bad_case_01.ml +++ /dev/null @@ -1,25 +0,0 @@ -open Pure_reducer_bad_case_support - -(* Scenario: Snapshot progress arrives when no snapshot download owns the event. - The reducer must ignore it without effects, then still accept authentication - and start catalog discovery. *) -let run () = - let origin = initial () in - let event = - Core.Snapshot_download_progress - { graph_id; received_bytes = 1L; total_bytes = Some 2L } - in - let rejected = check_step "BC01" origin event (observe origin) [] in - let startup_event = Core.Account_authenticated { user_id = Some "user" } in - let startup = - Core.step rejected.next startup_event - |> check_replay "BC01 follow-up" rejected.next startup_event - in - match startup.effects with - | [ Core.Publish (Core.State_changed _) - ; Core.Run (Core.Request (_, Core.Fetch_catalog _)) - ] -> () - | _ -> Alcotest.fail "BC01 valid startup event was not accepted" -;; - -let scenario = Alcotest.test_case "BC01 unowned snapshot progress" `Quick run diff --git a/logseq_sync/test/pure_reducer_bad_case_02.ml b/logseq_sync/test/pure_reducer_bad_case_02.ml deleted file mode 100644 index eb45ba37..00000000 --- a/logseq_sync/test/pure_reducer_bad_case_02.ml +++ /dev/null @@ -1,28 +0,0 @@ -open Pure_reducer_bad_case_support - -(* Scenario: A completed mirror inspection is replayed after graph attachment. - The reducer must not attach the graph again and must retain the current - WebSocket scope for its legitimate completion. *) -let run () = - let fixture = connected_fixture () in - let event = Core.Mirror_inspected (Core.Mirror_available fixture.mirror_request) in - let rejected = - check_step "BC02" fixture.attached.next event (observe fixture.attached.next) [] - in - let follow_up = Core.Websocket_opened fixture.connection in - let accepted = - Core.step rejected.next follow_up - |> check_replay "BC02 follow-up" rejected.next follow_up - in - match accepted.effects with - | [ Core.Publish (Core.State_changed _) - ; Core.Run (Core.Send_websocket send) - ; Core.Run (Core.Schedule_timer timer) - ] - when timer.delay_seconds = 30. - && timer.scope.connection_generation = Some send.scope.connection_generation -> - () - | _ -> Alcotest.fail "BC02 current WebSocket scope was not retained" -;; - -let scenario = Alcotest.test_case "BC02 duplicate mirror inspection" `Quick run diff --git a/logseq_sync/test/pure_reducer_bad_case_03.ml b/logseq_sync/test/pure_reducer_bad_case_03.ml deleted file mode 100644 index 23cbc806..00000000 --- a/logseq_sync/test/pure_reducer_bad_case_03.ml +++ /dev/null @@ -1,30 +0,0 @@ -let server_cursor_of_int value = - Logseq_db_types.Server_cursor.of_int64 (Int64.of_int value) |> Result.get_ok -;; - -open Pure_reducer_bad_case_support - -(* Scenario: Graph attachment is replayed after the WebSocket is already live. - The reducer must not request another token and the current connection must - still accept its next server message. *) -let run () = - let fixture = connected_fixture () in - let event = Core.Graph_attached fixture.attachment in - let rejected = - check_step "BC03" fixture.opened.next event (observe fixture.opened.next) [] - in - let message = - Protocol.Server.Pull_ok { t = server_cursor_of_int 0; checksum = None; txs = [] } - in - let follow_up = Core.Websocket_message (fixture.connection, message) in - let accepted = - Core.step rejected.next follow_up - |> check_replay "BC03 follow-up" rejected.next follow_up - in - match accepted.effects with - | [ Core.Publish (Core.State_changed state) ] - when state.snapshot.sync_phase = Core.Current -> () - | _ -> Alcotest.fail "BC03 current connection was not retained" -;; - -let scenario = Alcotest.test_case "BC03 duplicate graph attachment" `Quick run diff --git a/logseq_sync/test/pure_reducer_bad_case_04.ml b/logseq_sync/test/pure_reducer_bad_case_04.ml deleted file mode 100644 index c5b7297a..00000000 --- a/logseq_sync/test/pure_reducer_bad_case_04.ml +++ /dev/null @@ -1,30 +0,0 @@ -let server_cursor_of_int value = - Logseq_db_types.Server_cursor.of_int64 (Int64.of_int value) |> Result.get_ok -;; - -open Pure_reducer_bad_case_support - -(* Scenario: WebSocket_opened is replayed for an already live connection. The - reducer must not send a duplicate pull and must preserve the connection's - ownership of the next server message. *) -let run () = - let fixture = connected_fixture () in - let event = Core.Websocket_opened fixture.connection in - let rejected = - check_step "BC04" fixture.opened.next event (observe fixture.opened.next) [] - in - let message = - Protocol.Server.Pull_ok { t = server_cursor_of_int 0; checksum = None; txs = [] } - in - let follow_up = Core.Websocket_message (fixture.connection, message) in - let accepted = - Core.step rejected.next follow_up - |> check_replay "BC04 follow-up" rejected.next follow_up - in - match accepted.effects with - | [ Core.Publish (Core.State_changed state) ] - when state.snapshot.sync_phase = Core.Current -> () - | _ -> Alcotest.fail "BC04 live connection did not accept its next message" -;; - -let scenario = Alcotest.test_case "BC04 duplicate WebSocket open" `Quick run diff --git a/logseq_sync/test/pure_reducer_bad_case_05.ml b/logseq_sync/test/pure_reducer_bad_case_05.ml deleted file mode 100644 index 256fd64e..00000000 --- a/logseq_sync/test/pure_reducer_bad_case_05.ml +++ /dev/null @@ -1,33 +0,0 @@ -let server_cursor_of_int value = - Logseq_db_types.Server_cursor.of_int64 (Int64.of_int value) |> Result.get_ok -;; - -open Pure_reducer_bad_case_support - -(* Scenario: A server message arrives after its WebSocket connection closed. - The reducer must ignore the late message, then recover with a distinct - connection generation. *) -let run () = - let fixture = connected_fixture () in - let closed_event = - Core.Websocket_closed (fixture.connection, Core.Recoverable_close { message = Some "closed"; attention = Requires_attention }) - in - let closed = Core.step fixture.opened.next closed_event in - let late_message = - Core.Websocket_message - ( fixture.connection - , Protocol.Server.Pull_ok { t = server_cursor_of_int 0; checksum = None; txs = [] } - ) - in - let rejected = check_step "BC05" closed.next late_message (observe closed.next) [] in - let recovery = retry_after_retirement rejected.next fixture.connection in - match recovery.effects with - | [ Core.Publish (Core.State_changed _); Core.Run (Core.Start_websocket request) ] -> - Alcotest.(check bool) - "BC05 replacement connection generation" - true - (request.scope.connection_generation <> fixture.connection.connection_generation) - | _ -> Alcotest.fail "BC05 did not start a replacement WebSocket" -;; - -let scenario = Alcotest.test_case "BC05 message after WebSocket close" `Quick run diff --git a/logseq_sync/test/pure_reducer_bad_case_06.ml b/logseq_sync/test/pure_reducer_bad_case_06.ml deleted file mode 100644 index c4b3be4f..00000000 --- a/logseq_sync/test/pure_reducer_bad_case_06.ml +++ /dev/null @@ -1,57 +0,0 @@ -let server_cursor_of_int value = - Logseq_db_types.Server_cursor.of_int64 (Int64.of_int value) |> Result.get_ok -;; - -open Pure_reducer_bad_case_support - -(* Scenario: An authoritative apply completion arrives without a pending apply. - The reducer must ignore it, then accept the same completion after a pull has - created the matching authoritative owner. *) -let run () = - let fixture = connected_fixture () in - let scope = fixture.connection.graph in - let cursor1 = token Overlay.Server_cursor.of_int64 1L in - let sync_token = token Overlay.sync_token_of_string "sync-token:v1:bc06" in - let generation = token Overlay.Generation.of_string "generation:v1:6" in - let projection = token Overlay.Projection_revision.of_string "projection:v1:6" in - let sync = Overlay.sync_view ~token:sync_token ~checkpoint:cursor1 ~submissions:[] in - let commit : Overlay.authoritative_commit = - { generation - ; before_projection_revision = projection - ; after_projection_revision = projection - ; checkpoint = cursor1 - ; sync_token - ; terminal_receipts = [] - ; replanned_queued_ids = [] - ; blocked_ids = [] - ; logical_change_summary = Overlay.No_logical_change - } - in - let completion = Core.Authoritative_batch_applied { scope; commit; sync } in - let rejected = - check_step "BC06" fixture.opened.next completion (observe fixture.opened.next) [] - in - let remote_tx : Protocol.Server.pull_transaction = - { t = server_cursor_of_int 1; tx = "[]"; outliner_op = None } - in - let request_event = - Core.Websocket_message - ( fixture.connection - , Protocol.Server.Pull_ok - { t = server_cursor_of_int 1; checksum = None; txs = [ remote_tx ] } ) - in - let requested = Core.step rejected.next request_event in - (match requested.effects with - | [ Core.Delegate (Core.Apply_authoritative_batch _) - ; Core.Publish (Core.State_changed state) ] - when List.exists (fun (group : Core.diagnostic_group) -> - List.assoc_opt "Remote tx" group.entries = Some "1") state.diagnostics.groups -> () - | _ -> Alcotest.fail "BC06 did not create a valid authoritative owner"); - let accepted = Core.step requested.next completion in - match accepted.effects with - | [ Core.Publish (Core.State_changed state) ] - when state.snapshot.applied_server_t = Some (server_cursor_of_int 1) -> () - | _ -> Alcotest.fail "BC06 exact authoritative completion was not accepted" -;; - -let scenario = Alcotest.test_case "BC06 unsolicited authoritative apply" `Quick run diff --git a/logseq_sync/test/pure_reducer_bad_case_07.ml b/logseq_sync/test/pure_reducer_bad_case_07.ml deleted file mode 100644 index c7cd13a9..00000000 --- a/logseq_sync/test/pure_reducer_bad_case_07.ml +++ /dev/null @@ -1,50 +0,0 @@ -open Pure_reducer_bad_case_support - -(* Scenario: An old account's catalog fetch completes after local restoration - starts a new account generation. The stale completion must be ignored while - the new account's exact catalog load completion remains valid. *) -let run () = - let old_authenticated = - Core.step (initial ()) (Core.Account_authenticated { user_id = Some "old-user" }) - in - let old_completion, old_ticket = - match old_authenticated.effects with - | [ Core.Publish (Core.State_changed _) - ; Core.Run (Core.Request (ticket, Core.Fetch_catalog _)) - ] -> - ( Core.Runner_completed (Core.Completion (ticket, Ok [ graph ])) - , Core.effect_ticket_id ticket ) - | _ -> Alcotest.fail "BC07 old account did not request catalog" - in - let restoring = - Core.step old_authenticated.next (Core.Restore_local_account { user_id = "new-user" }) - in - let load_completion = - match restoring.effects with - | [ Core.Publish (Core.State_changed _) - ; Core.Run (Core.Request (ticket, Core.Load_catalog _)) - ] -> - Alcotest.(check bool) - "BC07 account requests have distinct effect IDs" - true - (Core.effect_ticket_id ticket <> old_ticket); - Core.Runner_completed - (Core.Completion - ( ticket - , Ok - (Some - (Core.catalog_cache ~user_id:"new-user" ~graphs:[] ~selected_graph:None)) - )) - | _ -> Alcotest.fail "BC07 replacement account did not request local catalog" - in - let rejected = - check_step "BC07" restoring.next old_completion (observe restoring.next) [] - in - let accepted = Core.step rejected.next load_completion in - match accepted.effects with - | [ Core.Publish (Core.State_changed state) ] - when state.snapshot.startup.awaiting_selection -> () - | _ -> Alcotest.fail "BC07 new account load completion was not accepted" -;; - -let scenario = Alcotest.test_case "BC07 stale catalog completion" `Quick run diff --git a/logseq_sync/test/pure_reducer_bad_case_08.ml b/logseq_sync/test/pure_reducer_bad_case_08.ml deleted file mode 100644 index 4b13c90f..00000000 --- a/logseq_sync/test/pure_reducer_bad_case_08.ml +++ /dev/null @@ -1,56 +0,0 @@ -open Pure_reducer_bad_case_support - -(* Scenario: An old WebSocket opens after recovery started a replacement. - The stale connection must be ignored without consuming the fresh scope. *) -let run () = - let fixture = connected_fixture () in - let background = - Core.step - fixture.attached.next - (Core.Foreground_changed { foreground = false; lifecycle_generation = 1L }) - in - let resumed = - Core.step - background.next - (Core.Foreground_changed { foreground = true; lifecycle_generation = 2L }) - in - Alcotest.(check bool) - "BC08 replacement waits for physical retirement" - false - (List.exists - (function - | Core.Run (Core.Start_websocket _) -> true - | _ -> false) - resumed.effects); - let recovery = Core.step resumed.next (Core.Websocket_retired fixture.connection) in - let fresh_connection = - List.find_map - (function - | Core.Run (Core.Start_websocket request) -> Some request.scope - | _ -> None) - recovery.effects - |> Option.get - in - Alcotest.(check bool) - "BC08 connection generations differ" - true - (fresh_connection.connection_generation <> fixture.connection.connection_generation); - let stale_event = Core.Websocket_opened fixture.connection in - let rejected = check_step "BC08" recovery.next stale_event (observe recovery.next) [] in - let fresh_event = Core.Websocket_opened fresh_connection in - let accepted = - Core.step rejected.next fresh_event - |> check_replay "BC08 follow-up" rejected.next fresh_event - in - match accepted.effects with - | [ Core.Publish (Core.State_changed _) - ; Core.Run (Core.Send_websocket send) - ; Core.Run (Core.Schedule_timer timer) - ] - when timer.delay_seconds = 30. - && timer.scope.connection_generation = Some send.scope.connection_generation -> - () - | _ -> Alcotest.fail "BC08 fresh connection was not accepted" -;; - -let scenario = Alcotest.test_case "BC08 replaced WebSocket connection" `Quick run diff --git a/logseq_sync/test/pure_reducer_bad_case_09.ml b/logseq_sync/test/pure_reducer_bad_case_09.ml deleted file mode 100644 index e9b7fa68..00000000 --- a/logseq_sync/test/pure_reducer_bad_case_09.ml +++ /dev/null @@ -1,29 +0,0 @@ -open Pure_reducer_bad_case_support - -(* Scenario: Sync inspection data arrives without a pending Inspect_sync. The - reducer must ignore it, then accept the same data after a local outbox change - creates the exact inspection owner and reserve the eligible submission. *) -let run () = - let fixture, current = current_connected_fixture () in - let scope = fixture.connection.graph in - let checkpoint = token Overlay.Server_cursor.of_int64 0L in - let sync_token = token Overlay.sync_token_of_string "sync-token:v1:bc09" in - let sync = queued_sync sync_token checkpoint (List.hd mutation_ids) in - let event = Core.Sync_inspected { scope; sync } in - let rejected = check_step "BC09" current.next event (observe current.next) [] in - let changed = Core.step rejected.next Core.Local_outbox_changed in - (match changed.effects with - | [ Core.Delegate (Core.Inspect_sync requested_scope) ] when requested_scope = scope -> - () - | _ -> Alcotest.fail "BC09 valid sync inspection was not requested"); - let accepted = Core.step changed.next event in - match accepted.effects with - | [ Core.Publish (Core.State_changed state) - ; Core.Delegate (Core.Apply_outbox_transition request) - ] - when state.snapshot.sync_phase = Core.Submitting - && request.transition = Overlay.Submit_group [ List.hd mutation_ids ] -> () - | _ -> Alcotest.fail "BC09 exact requested sync result was not accepted" -;; - -let scenario = Alcotest.test_case "BC09 unsolicited sync inspection" `Quick run diff --git a/logseq_sync/test/pure_reducer_bad_case_10.ml b/logseq_sync/test/pure_reducer_bad_case_10.ml deleted file mode 100644 index b2bfa61d..00000000 --- a/logseq_sync/test/pure_reducer_bad_case_10.ml +++ /dev/null @@ -1,37 +0,0 @@ -open Pure_reducer_bad_case_support - -(* Scenario: An outbox completion commits a transition that differs from the - pending request. The mismatch must be ignored without consuming ownership, - then the exact completion must send the reserved transaction batch. *) -let run () = - let fixture = pending_submission_fixture () in - let mismatched_commit = { fixture.commit with transition = Overlay.Submit_group [] } in - let mismatched_event = - Core.Outbox_transition_applied - { scope = fixture.request.scope - ; commit = mismatched_commit - ; sync = fixture.result_sync - } - in - let rejected = - check_step "BC10" fixture.origin mismatched_event (observe fixture.origin) [] - in - let exact_event = - Core.Outbox_transition_applied - { scope = fixture.request.scope - ; commit = fixture.commit - ; sync = fixture.result_sync - } - in - let accepted = Core.step rejected.next exact_event in - match accepted.effects with - | [ Core.Run (Core.Send_websocket { scope; message = Protocol.Client.Tx_batch _ }) - ; Core.Run (Core.Schedule_timer timer) - ] - when scope = fixture.connected.connection - && timer.delay_seconds = 30. - && timer.scope.connection_generation = Some scope.connection_generation -> () - | _ -> Alcotest.fail "BC10 exact transition completion was not accepted" -;; - -let scenario = Alcotest.test_case "BC10 mismatched outbox transition" `Quick run diff --git a/logseq_sync/test/pure_reducer_bad_case_support.ml b/logseq_sync/test/pure_reducer_bad_case_support.ml deleted file mode 100644 index 00dc0e92..00000000 --- a/logseq_sync/test/pure_reducer_bad_case_support.ml +++ /dev/null @@ -1,286 +0,0 @@ -let server_cursor_of_int value = - Logseq_db_types.Server_cursor.of_int64 (Int64.of_int value) |> Result.get_ok -;; - -module Core = Logseq_sync_pure_reducer.Core -module Overlay = Logseq_overlay_db.Types -module Protocol = Logseq_sync_pure_reducer.Sync_protocol - -type observed = - { state : Core.state - ; admitted_graph_scope : Core.graph_scope option - } - -let observe core = - { state = Core.state core; admitted_graph_scope = Core.admitted_graph_scope core } -;; - -let check_observed test_id message expected actual = - Alcotest.(check bool) (test_id ^ " " ^ message) true (expected = actual) -;; - -let check_instructions test_id expected actual = - Alcotest.(check bool) - (test_id ^ " ordered instructions") - true - (Core.equal_instructions expected actual) -;; - -let check_replay test_id origin event (first : Core.transition) = - let origin_observed = observe origin in - check_observed test_id "origin remains immutable" origin_observed (observe origin); - let replay = Core.step origin event in - check_observed test_id "replay observation" (observe first.next) (observe replay.next); - check_instructions test_id first.effects replay.effects; - first -;; - -let check_step test_id origin event expected effects = - let origin_observed = observe origin in - let first = Core.step origin event in - check_observed test_id "public observation" expected (observe first.next); - check_instructions test_id effects first.effects; - check_observed test_id "origin remains immutable" origin_observed (observe origin); - check_replay test_id origin event first -;; - -let limits () = - Core.limits - ~maximum_response_bytes:Logseq_db_types.Limits.maximum_response_bytes - ~maximum_artifact_bytes:(1024 * 1024) - ~submission_batch_size:32 - |> Result.get_ok -;; - -let initial () = - Core.config - ~managed_sync_origin:(Uri.of_string "https://api.logseq.io") - ~limits:(limits ()) - |> Result.get_ok - |> Core.initial - |> Result.get_ok - |> fun core -> (Core.step core (Core.Sign_in_requested {account_generation=0;presentation_generation=0})).next -;; - -let token module_of_string value = module_of_string value |> Result.get_ok - -let graph_id = - Logseq_db_types.Graph_types.Uuid.of_string "11111111-1111-4111-8111-111111111111" - |> Result.get_ok -;; - -let graph : Core.graph = - { graph_id - ; name = "Journal" - ; schema = { major = 1; minor = 0; exact = true } - ; encrypted = false - } -;; - -let mutation_ids = - [ "22222222-2222-4222-8222-222222222221" - ; "22222222-2222-4222-8222-222222222222" - ; "22222222-2222-4222-8222-222222222223" - ] - |> List.map (fun value -> - Logseq_db_types.Graph_types.Uuid.of_string value |> Result.get_ok) -;; - -let selected_graph () = - let authenticated = - Core.step (initial ()) (Core.Account_authenticated { user_id = Some "user" }) - in - let catalog = - List.find_map - (function - | Core.Run (Core.Request (ticket, Core.Fetch_catalog _)) -> - Some - (Core.step - authenticated.next - (Core.Runner_completed (Core.Completion (ticket, Ok [ graph ])))) - | Run _ | Delegate _ | Publish _ -> None) - authenticated.effects - |> Option.get - in - let selected = Core.step catalog.next (Core.Graph_selected graph_id) in - selected, Core.admitted_graph_scope selected.next |> Option.get -;; - -type connected_fixture = - { mirror_request : Core.mirror_request - ; attachment : Core.graph_attachment - ; attached : Core.transition - ; connection : Core.connection_scope - ; opened : Core.transition - } - -let connected_fixture () = - let selected, scope = selected_graph () in - let mirror_request = - match selected.effects with - | Core.Delegate (Core.Inspect_mirror request) :: _ when request.scope = scope -> - request - | _ -> Alcotest.fail "connected fixture did not emit Inspect_mirror" - in - let inspected = - Core.step selected.next (Core.Mirror_inspected (Core.Mirror_available mirror_request)) - in - let attach_request = - match inspected.effects with - | [ Core.Delegate (Core.Attach_graph request) ] when request = mirror_request -> - request - | _ -> Alcotest.fail "connected fixture did not emit exact Attach_graph" - in - let checkpoint = token Overlay.Server_cursor.of_int64 0L in - let sync_token = token Overlay.sync_token_of_string "sync-token:v1:fixture-0" in - let sync = Overlay.sync_view ~token:sync_token ~checkpoint ~submissions:[] in - let attachment : Core.graph_attachment = { scope = attach_request.scope; sync } in - let attached = Core.step inspected.next (Core.Graph_attached attachment) in - let connection = - match attached.effects with - | [ Core.Publish (Core.State_changed _); Core.Run (Core.Start_websocket request) ] -> - request.scope - | _ -> Alcotest.fail "connected fixture did not start WebSocket" - in - let opened = Core.step attached.next (Core.Websocket_opened connection) in - (match opened.effects with - | [ Core.Publish (Core.State_changed _) - ; Core.Run - (Core.Send_websocket - { scope; message = Protocol.Client.Pull { since = Some since } }) - ; Core.Run (Core.Schedule_timer timer) - ] - when scope = connection - && Logseq_db_types.Server_cursor.equal since Logseq_db_types.Server_cursor.zero - && timer.delay_seconds = 30. - && timer.scope.connection_generation = Some connection.connection_generation -> - () - | _ -> Alcotest.fail "connected fixture did not issue opening Pull"); - { mirror_request; attachment; attached; connection; opened } -;; - -let current_connected_fixture () = - let fixture = connected_fixture () in - let current = - Core.step - fixture.opened.next - (Core.Websocket_message - ( fixture.connection - , Protocol.Server.Pull_ok - { t = server_cursor_of_int 0; checksum = None; txs = [] } )) - in - (match current.effects with - | [ Core.Publish (Core.State_changed state) ] - when state.snapshot.sync_phase = Core.Current -> () - | _ -> Alcotest.fail "current fixture did not complete opening Pull"); - fixture, current -;; - -let queued_sync sync_token checkpoint mutation_id = - let fingerprint = - token Overlay.Mutation_fingerprint.of_string "mutation-fingerprint:v1:bad-case" - in - let descriptor : Overlay.submission_descriptor = - { mutation_id - ; fingerprint - ; state = Overlay.Queued - ; operation = Overlay.Save_block_operation - ; dependency_eligible = true - ; attempt_count = 0 - ; plaintext_bytes = 16 - ; protected_bytes = Some 16 - } - in - Overlay.sync_view ~token:sync_token ~checkpoint ~submissions:[ descriptor ] -;; - -type pending_submission_fixture = - { connected : connected_fixture - ; origin : Core.t - ; request : Core.outbox_transition_request - ; commit : Overlay.outbox_commit - ; result_sync : Overlay.sync_view - } - -let pending_submission_fixture () = - let connected, current = current_connected_fixture () in - let scope = connected.connection.graph in - let changed = Core.step current.next Core.Local_outbox_changed in - (match changed.effects with - | [ Core.Delegate (Core.Inspect_sync requested_scope) ] when requested_scope = scope -> - () - | _ -> Alcotest.fail "pending submission fixture did not inspect sync state"); - let mutation_id = List.hd mutation_ids in - let checkpoint = token Overlay.Server_cursor.of_int64 0L in - let expected = token Overlay.sync_token_of_string "sync-token:v1:bad-case-before" in - let sync = queued_sync expected checkpoint mutation_id in - let planned = Core.step changed.next (Core.Sync_inspected { scope; sync }) in - let request = - match planned.effects with - | [ Core.Publish (Core.State_changed _) - ; Core.Delegate (Core.Apply_outbox_transition request) - ] - when request.scope = scope - && request.transition = Overlay.Submit_group [ mutation_id ] -> request - | _ -> Alcotest.fail "pending submission fixture did not reserve Submit_group" - in - let batch_id = - token Overlay.Submission_batch_id.of_string "submission-batch:v1:bad-case" - in - let wire = - Overlay.submission_wire - ~maximum_bytes:1024 - ~mutation_id - ~operation:Overlay.Save_block_operation - ~protected_transaction:"protected-bad-case" - |> Result.get_ok - in - let batch = - Overlay.submission_batch - ~maximum_wires:1 - ~maximum_bytes:4096 - ~batch_id - ~t_before:checkpoint - ~wires:[ wire ] - |> Result.get_ok - in - let generation = token Overlay.Generation.of_string "generation:v1:10" in - let projection = token Overlay.Projection_revision.of_string "projection:v1:10" in - let result_token = token Overlay.sync_token_of_string "sync-token:v1:bad-case-after" in - let commit : Overlay.outbox_commit = - { generation - ; before_projection_revision = projection - ; after_projection_revision = projection - ; sync_token = result_token - ; transition = request.transition - ; activity = Overlay.Logically_active - ; logical_change_summary = Overlay.No_logical_change - ; submission_batch = Some batch - } - in - let submitted_descriptor = - match Overlay.sync_view_submissions sync with - | [ descriptor ] -> { descriptor with state = Overlay.Submitted batch_id } - | _ -> assert false - in - let result_sync = - Overlay.sync_view - ~token:result_token - ~checkpoint - ~submissions:[ submitted_descriptor ] - in - { connected; origin = planned.next; request; commit; result_sync } -;; - -let retry_after_retirement core connection = - let retired = Core.step core (Core.Websocket_retired connection) in - let timer = - List.find_map - (function - | Core.Run (Core.Schedule_timer timer) -> Some timer - | _ -> None) - retired.effects - |> Option.get - in - Core.step retired.next (Core.Timer_elapsed timer.id) -;; diff --git a/logseq_sync/test/runner_contract.ml b/logseq_sync/test/runner_contract.ml deleted file mode 100644 index 28c762f3..00000000 --- a/logseq_sync/test/runner_contract.ml +++ /dev/null @@ -1,2606 +0,0 @@ -module Core = Logseq_sync_pure_reducer.Core - -let retire_cache request = - let retirement = Core.retirement_id_of_string "96000000-0000-4000-8000-000000000001" |> Result.get_ok in - match request (Core.Prepare_graph_asset_deletion retirement) with - | Ok (Core.Asset_deletion_prepared deletion) -> request (Core.Commit_graph_asset_deletion deletion) - | Error _ as error -> error - | _ -> Error (Core.Asset_invalid_content "Expected prepared retirement") -;; -module Runner = Logseq_sync_effect_runner.Effect_runner - -let fail format = Printf.ksprintf (fun message -> Alcotest.fail message) format - -let remove_tree path = - let rec loop path = - match Unix.lstat path with - | { st_kind = Unix.S_DIR; _ } -> - Sys.readdir path |> Array.iter (fun name -> loop (Filename.concat path name)); - Unix.rmdir path - | _ -> Unix.unlink path - | exception Unix.Unix_error (Unix.ENOENT, _, _) -> () - in - loop path -;; - -let with_support f = - let path = Filename.temp_file "logseq-sync-runner-" "" in - Sys.remove path; - Unix.mkdir path 0o700; - Fun.protect ~finally:(fun () -> remove_tree path) (fun () -> f path) -;; - -let secrets () = - Runner.secrets - ~unlock_private_key: - (fun - ~managed_sync_origin:_ ~user_id:_ ~password:_ ~private_key_package:_ -> - Error "private key unavailable") - ~unlock_graph_key:(fun ~managed_sync_origin:_ ~user_id:_ ~encrypted_graph_key:_ -> - Error "graph key unavailable") - ~load_wrapped_graph_key:(fun ~managed_sync_origin:_ ~user_id:_ ~graph_id:_ -> - Error (Runner.Wrapped_graph_key_unavailable "wrapped graph key unavailable")) - ~verify_and_save_wrapped_graph_key: - (fun - ~managed_sync_origin:_ ~user_id:_ ~graph_id:_ ~encrypted_graph_key:_ -> - Error "wrapped graph key store unavailable") - ~delete_account_secrets:(fun ~managed_sync_origin:_ ~user_id:_ -> Ok ()) - |> Result.get_ok -;; - -let crypto () = - Runner.crypto - ~encrypt_aes_gcm:(fun ~key:_ ~plaintext:_ -> Error "encryption unavailable") - ~decrypt_aes_gcm:(fun ~key:_ ~iv:_ ~ciphertext:_ -> Error "decryption unavailable") - |> Result.get_ok -;; - -let core ?(origin = Uri.of_string "https://api.logseq.io") () = - let limits = - Core.limits - ~maximum_response_bytes:Logseq_db_types.Limits.maximum_response_bytes - ~maximum_artifact_bytes:(1024 * 1024 * 1024) - ~submission_batch_size:32 - |> Result.get_ok - in - Core.config ~managed_sync_origin:origin ~limits - |> Result.get_ok - |> Core.initial - |> Result.get_ok - |> fun core -> (Core.step core (Core.Sign_in_requested {account_generation=0;presentation_generation=0})).next -;; - -let load_catalog_transition () = - Core.step (core ()) (Restore_local_account { user_id = "user-1" }) -;; - -let cancellation_effect () = - let restoring = load_catalog_transition () in - let cancelled = Core.step restoring.next Core.Shutdown in - List.find_map - (function - | Core.Run (Core.Cancel_effects _ as runnable) -> Some runnable - | _ -> None) - cancelled.effects - |> Option.get -;; - -let load_catalog_effect () = - Core.step (core ()) (Restore_local_account { user_id = "user-1" }) - |> fun transition -> - List.find_map - (function - | Core.Run (Core.Request (_, Core.Load_catalog _) as instruction) -> - Some instruction - | Run _ | Delegate _ | Publish _ -> None) - transition.effects - |> Option.get -;; - -let dependencies - ?secrets_dependency - ?crypto_dependency - ?id_token_dependency - ~environment - ~support - ~fork - () - = - let runtime = Runner.runtime ~fork ~sleep:(fun _ -> ()) |> Result.get_ok in - let transport = - Runner.transport - ~websocket_liveness:Runner.Disabled - ~tls_authenticator:(Runner.system_tls_authenticator () |> Result.get_ok) - ~network:(Eio.Stdenv.net environment) - ~clock:(Eio.Stdenv.clock environment) - |> Result.get_ok - in - let local_store = - Runner.local_store ~application_support_directory:support () |> Result.get_ok - in - let artifact_store = - Runner.artifact_store ~staging_directory:(Filename.concat support "staging") - |> Result.get_ok - in - Runner.dependencies - ~runtime - ~transport - ~local_store - ~artifact_store - ~secrets:(Option.value secrets_dependency ~default:(secrets ())) - ~crypto:(Option.value crypto_dependency ~default:(crypto ())) - ~id_token_provider: - (Option.value - id_token_dependency - ~default: - (Runner.id_token_provider - ~acquire:(fun _ -> Ok "test-id-token") - ~invalidate:(fun _ ~token:_ -> ()))) - |> Result.get_ok -;; - -let graph_id () = - Logseq_db_types.Graph_types.Uuid.of_string "11111111-1111-4111-8111-111111111111" - |> Result.get_ok -;; - -let encrypted_graph () : Core.graph = - { graph_id = graph_id () - ; name = "Encrypted Journal" - ; schema = { major = 1; minor = 0; exact = true } - ; encrypted = true - } -;; - -let test_catalog_load_uses_account_and_origin_scoped_path () = - with_support (fun support -> - let worker_root = Filename.concat support "logseq-db-worker" in - let catalog_root = Filename.concat worker_root "sync-catalogs" in - Unix.mkdir worker_root 0o700; - Unix.mkdir catalog_root 0o700; - let cache = - Core.catalog_cache - ~user_id:"user-1" - ~graphs:[ encrypted_graph () ] - ~selected_graph:None - in - let path = - Filename.concat - catalog_root - "748ac0b0c274b30bc6fc1da756958deab4ebdef06a2d6373a377e2b4d8cec6df.json" - in - let channel = open_out_bin path in - Fun.protect - ~finally:(fun () -> close_out_noerr channel) - (fun () -> output_string channel (Core.encode_catalog_cache cache)); - Eio_main.run (fun environment -> - Eio.Switch.run (fun sw -> - let tasks = ref [] in - let posted = ref [] in - let dependencies = - dependencies - ~environment - ~support - ~fork:(fun ~sw:_ task -> tasks := task :: !tasks) - () - in - let runner = - Runner.create ~sw dependencies ~post:(fun event -> posted := event :: !posted) - |> Result.get_ok - in - let restoring = - Core.step (core ()) (Restore_local_account { user_id = "user-1" }) - in - Runner.submit runner (load_catalog_effect ()); - List.iter (fun task -> task ()) (List.rev !tasks); - let restored = - match !posted with - | [ event ] -> Core.step restoring.next event - | _ -> fail "catalog load did not post exactly one completion" - in - Alcotest.(check int) - "scoped catalog is restored" - 1 - (List.length (Core.state restored.next).snapshot.catalog)))) -;; - -(* The runner owns whether a queued callback starts. A pure Core completion - cannot reproduce a cancelled callback writing to the filesystem. *) -let test_cancelled_queued_catalog_save_does_not_write () = - with_support (fun support -> - Eio_main.run (fun environment -> - Eio.Switch.run (fun sw -> - let tasks = ref [] in - let posted = ref [] in - let deps = - dependencies - ~environment - ~support - ~fork:(fun ~sw:_ task -> tasks := task :: !tasks) - () - in - let runner = - Runner.create ~sw deps ~post:(fun event -> posted := event :: !posted) - |> Result.get_ok - in - let authenticated = - Core.step (core ()) (Account_authenticated { user_id = Some "user-1" }) - in - let catalog = - List.find_map - (function - | Core.Run (Core.Request (ticket, Core.Fetch_catalog _)) -> - Some - (Core.step - authenticated.next - (Runner_completed (Completion (ticket, Ok [ encrypted_graph () ])))) - | _ -> None) - authenticated.effects - |> Option.get - in - let selected = Core.step catalog.next (Graph_selected (graph_id ())) in - let save = - List.find_map - (function - | Core.Run (Core.Request (_, Core.Save_catalog _) as instruction) -> - Some instruction - | _ -> None) - selected.effects - |> Option.get - in - Runner.submit runner save; - Runner.submit runner (cancellation_effect ()); - List.iter (fun task -> task ()) (List.rev !tasks); - Alcotest.(check bool) - "cancelled save has no durable side effects" - false - (Sys.file_exists (Filename.concat support "logseq-db-worker")); - Alcotest.(check int) "cancelled save posts no completion" 0 (List.length !posted)))) -;; - -let cached_key_effect ?origin () = - let authenticated = - Core.step (core ?origin ()) (Account_authenticated { user_id = Some "user-1" }) - in - let graph = encrypted_graph () in - let catalog = - List.find_map - (function - | Core.Run (Core.Request (ticket, Core.Fetch_catalog _)) -> - Some - (Core.step - authenticated.next - (Runner_completed (Completion (ticket, Ok [ graph ])))) - | Run _ | Delegate _ | Publish _ -> None) - authenticated.effects - |> function - | Some transition -> transition - | None -> fail "transition did not fetch the graph catalog" - in - let selected = Core.step catalog.next (Graph_selected graph.graph_id) in - let scope = - List.find_map - (function - | Core.Delegate (Core.Inspect_mirror request) -> Some request.scope - | Run _ | Delegate _ | Publish _ -> None) - selected.effects - |> function - | Some scope -> scope - | None -> fail "encrypted graph selection did not inspect its mirror" - in - let missing = Core.step selected.next (Mirror_inspected (Mirror_absent scope)) in - let instruction = - List.find_map - (function - | Core.Run (Core.Request (_, Core.Load_and_unlock_graph_key _) as instruction) -> - Some instruction - | Run _ | Delegate _ | Publish _ -> None) - missing.effects - |> function - | Some instruction -> instruction - | None -> fail "encrypted graph bootstrap did not request its cached graph key" - in - missing, instruction -;; - -let test_dependency_constructors_validate_owned_resources () = - with_support (fun support -> - let missing = Filename.concat support "missing" in - Alcotest.check - Alcotest.bool - "missing application support directory is rejected" - true - (Result.is_error (Runner.local_store ~application_support_directory:missing ()))) -;; - -let test_submit_is_async_and_posts_a_typed_completion () = - with_support (fun support -> - Eio_main.run (fun environment -> - Eio.Switch.run (fun sw -> - let tasks = ref [] in - let posted = ref [] in - let dependencies = - dependencies - ~environment - ~support - ~fork:(fun ~sw:_ task -> tasks := task :: !tasks) - () - in - let runner = - Runner.create ~sw dependencies ~post:(fun event -> posted := event :: !posted) - in - let runner = Result.get_ok runner in - Runner.submit runner (load_catalog_effect ()); - Alcotest.(check int) "submit does not synchronously post" 0 (List.length !posted); - List.iter (fun task -> task ()) (List.rev !tasks); - Alcotest.(check int) "one completion is posted" 1 (List.length !posted); - match !posted with - | [ Core.Runner_completed _ ] -> () - | _ -> Alcotest.fail "load catalog did not post its typed completion"))) -;; - -let test_cancellation_suppresses_late_completion () = - with_support (fun support -> - Eio_main.run (fun environment -> - Eio.Switch.run (fun sw -> - let tasks = ref [] in - let posted = ref [] in - let dependencies = - dependencies - ~environment - ~support - ~fork:(fun ~sw:_ task -> tasks := task :: !tasks) - () - in - let runner = - Runner.create ~sw dependencies ~post:(fun event -> posted := event :: !posted) - |> Result.get_ok - in - let instruction = load_catalog_effect () in - Runner.submit runner instruction; - Runner.submit runner (cancellation_effect ()); - List.iter (fun task -> task ()) (List.rev !tasks); - Alcotest.(check int) - "cancelled work posts no late completion" - 0 - (List.length !posted); - Runner.shutdown runner; - Runner.shutdown runner; - Runner.submit runner instruction; - Alcotest.(check int) "shutdown rejects new work" 0 (List.length !posted)))) -;; - -let protected_run runner ~tasks ~posted core operation scope action = - tasks := []; - posted := []; - let transition = - Core.step - core - (Core.Protected_requested - { protected_operation = operation - ; protected_scope = scope - ; protected_action = action - }) - in - List.iter - (function - | Core.Run runnable -> Runner.submit runner runnable - | _ -> ()) - transition.effects; - let queued = List.rev !tasks in - tasks := []; - List.iter (fun task -> task ()) queued; - let finished = - List.fold_left (fun core event -> (Core.step core event).next) transition.next !posted - in - let outputs = - List.concat_map - (fun event -> - let completed = Core.step transition.next event in - List.filter_map - (function - | Core.Publish (Core.Protected_finished output) -> - Some output.protected_result - | _ -> None) - completed.effects) - !posted - in - ignore finished; - match outputs with - | [ result ] -> result - | _ -> fail "protected request did not resolve through Core output" -;; - -let test_cached_wrapped_key_is_unlocked_before_protected_value_decryption - ?(malformed_plaintext = false) ?(decryption_failure = false) ?(decryption_exception = false) ?(encryption_failure = false) ?(cancel_online = false) () = - with_support (fun support -> - Eio_main.run (fun environment -> - Eio.Switch.run (fun sw -> - let wrapped_key = "cached-wrapped-graph-key" in - let raw_key = String.make 32 'k' in - let unlock_inputs = ref [] in - let tasks = ref [] in - let posted = ref [] in - let secrets_dependency = - Runner.secrets - ~unlock_private_key: - (fun - ~managed_sync_origin:_ ~user_id:_ ~password:_ ~private_key_package:_ -> - Error "private key unlock was not expected") - ~unlock_graph_key: - (fun - ~managed_sync_origin:_ ~user_id:_ ~encrypted_graph_key -> - unlock_inputs := encrypted_graph_key :: !unlock_inputs; - if String.equal encrypted_graph_key wrapped_key - then Ok raw_key - else Error "unexpected wrapped graph key") - ~load_wrapped_graph_key:(fun ~managed_sync_origin:_ ~user_id:_ ~graph_id:_ -> - Ok wrapped_key) - ~verify_and_save_wrapped_graph_key: - (fun - ~managed_sync_origin:_ ~user_id:_ ~graph_id:_ ~encrypted_graph_key:_ -> - Error "wrapped graph key save was not expected") - ~delete_account_secrets:(fun ~managed_sync_origin:_ ~user_id:_ -> Ok ()) - |> Result.get_ok - in - let crypto_dependency = - Runner.crypto - ~encrypt_aes_gcm:(fun ~key:_ ~plaintext:_ -> - Error "encryption was not expected") - ~decrypt_aes_gcm:(fun ~key ~iv:_ ~ciphertext:_ -> - if String.equal key raw_key - then if decryption_exception then raise (Unix.Unix_error (Unix.EACCES, "PRIVATE_OPERATION", "PRIVATE_PATH")) - else if decryption_failure - then Error "SENSITIVE-synthetic-crypto-error" - else if malformed_plaintext - then Ok {|"~XSENSITIVE-synthetic-body"|} - else - Ok - (Transit_native.Transit.Json.to_string - (Transit_core.Json.String "plaintext")) - else Error "protected value received a wrapped graph key") - |> Result.get_ok - in - let dependencies = - dependencies - ~secrets_dependency - ~crypto_dependency - ~environment - ~support - ~fork:(fun ~sw:_ task -> tasks := task :: !tasks) - () - in - let runner = - Runner.create ~sw dependencies ~post:(fun event -> posted := event :: !posted) - |> Result.get_ok - in - let before, instruction = cached_key_effect () in - let handle = - match instruction with - | Core.Request (ticket, Core.Load_and_unlock_graph_key scope) -> - Core.graph_key_handle - ~id:("graph-key-" ^ Core.effect_id_to_string (Core.effect_ticket_id ticket)) - ~scope - | Await_io _ - | Asset_io _ - | Protected_io _ - | Request _ - | Start_websocket _ - | Send_websocket _ - | Close_websocket _ - | Schedule_timer _ - | Cancel_effects _ | Cancel_network_effects _ -> fail "cached key effect had an unexpected request type" - in - Runner.submit runner instruction; - List.iter (fun task -> task ()) (List.rev !tasks); - Alcotest.(check (list string)) - "cached wrapped key is explicitly unlocked" - [ wrapped_key ] - (List.rev !unlock_inputs); - Alcotest.(check int) "cached key request completes once" 1 (List.length !posted); - let ciphertext = - Transit_native.Transit.Json.to_string - (Transit_core.Json.Array - [ Transit_core.Json.Binary "123456789012" - ; Transit_core.Json.Binary "ciphertext-and-tag" - ]) - in - let unlocked = - match !posted with - | [ event ] -> (Core.step before.next event).next - | _ -> fail "missing key completion" - in - let unlocked = - if not cancel_online then unlocked else ( - tasks := []; - let startup = (Core.state unlocked).snapshot.startup in - let cancelled = Core.step unlocked (Core.Online_cancelled - {account_generation=startup.account_generation; presentation_generation=startup.presentation_generation}) in - List.iter (function Core.Run instruction -> Runner.submit runner instruction | _ -> ()) cancelled.effects; - List.iter (fun task -> task ()) (List.rev !tasks); - cancelled.next) - in - let result = protected_run - runner - ~tasks - ~posted - unlocked - "decrypt-cached" - (Core.graph_key_handle_scope handle) - (if encryption_failure then Core.Encrypt_values (handle, ["plaintext"]) - else Core.Decrypt_value (handle, ciphertext)) in - if encryption_failure then - Alcotest.(check bool) "encryption failure returns a bounded diagnostic" true - (match result with - | Error (Core.Effect_failed source) -> - let module D = Logseq_db_types.Failure_diagnostic in - (match D.decode source with - | Some detail -> D.operation detail = "encryptValue" - && D.reason detail = "cryptoInvalid" - | None -> false) - | _ -> false) - else if decryption_exception then - Alcotest.(check bool) "protected provider exception retains only its safe cause" true - (match result with - | Error (Core.Effect_failed source) -> - let module D = Logseq_db_types.Failure_diagnostic in - (match D.decode source with - | Some detail -> D.operation detail = "decryptValue" - && D.reason detail = "cryptoInvalid" && D.source_class detail = "io-permission" - | None -> false) - | _ -> false) - else if decryption_failure then - Alcotest.(check bool) "crypto error excludes sensitive detail" true - (match result with - | Error (Core.Effect_failed message) -> - let module D = Logseq_db_types.Failure_diagnostic in - (match D.decode message with - | Some detail -> D.operation detail = "decryptValue" - && D.reason detail = "cryptoInvalid" - && not (String.contains (D.message detail) 'S') - | None -> false) - | _ -> false) - else if malformed_plaintext then - Alcotest.(check bool) "decrypted parser error excludes plaintext" true - (match result with - | Error (Core.Effect_failed message) -> - let module D = Logseq_db_types.Failure_diagnostic in - (match D.decode message with - | Some detail -> D.reason detail = "parseFailed" - && String.ends_with ~suffix:"decrypted Transit is invalid." (D.message detail) - | None -> false) - | _ -> false) - else - Alcotest.(check bool) "returned handle decrypts with the raw graph key" true - (result = Ok (Core.Decrypted_value "plaintext"))))) -;; - -let test_cached_wrapped_key_unlock_failure_is_fail_closed () = - with_support (fun support -> - Eio_main.run (fun environment -> - Eio.Switch.run (fun sw -> - let tasks = ref [] in - let posted = ref [] in - let secrets_dependency = - Runner.secrets - ~unlock_private_key: - (fun - ~managed_sync_origin:_ ~user_id:_ ~password:_ ~private_key_package:_ -> - Error "private key unlock was not expected") - ~unlock_graph_key: - (fun - ~managed_sync_origin:_ ~user_id:_ ~encrypted_graph_key:_ -> - Error "cached graph key could not be unlocked") - ~load_wrapped_graph_key:(fun ~managed_sync_origin:_ ~user_id:_ ~graph_id:_ -> - Ok "invalid-wrapped-key") - ~verify_and_save_wrapped_graph_key: - (fun - ~managed_sync_origin:_ ~user_id:_ ~graph_id:_ ~encrypted_graph_key:_ -> - Error "wrapped graph key save was not expected") - ~delete_account_secrets:(fun ~managed_sync_origin:_ ~user_id:_ -> Ok ()) - |> Result.get_ok - in - let dependencies = - dependencies - ~secrets_dependency - ~environment - ~support - ~fork:(fun ~sw:_ task -> tasks := task :: !tasks) - () - in - let runner = - Runner.create ~sw dependencies ~post:(fun event -> posted := event :: !posted) - |> Result.get_ok - in - let before, instruction = cached_key_effect () in - let handle = - match instruction with - | Core.Request (ticket, Core.Load_and_unlock_graph_key scope) -> - Core.graph_key_handle - ~id:("graph-key-" ^ Core.effect_id_to_string (Core.effect_ticket_id ticket)) - ~scope - | Await_io _ - | Asset_io _ - | Protected_io _ - | Request _ - | Start_websocket _ - | Send_websocket _ - | Close_websocket _ - | Schedule_timer _ - | Cancel_effects _ | Cancel_network_effects _ -> fail "cached key effect had an unexpected request type" - in - Runner.submit runner instruction; - List.iter (fun task -> task ()) (List.rev !tasks); - let failed = - match !posted with - | [ event ] -> Core.step before.next event - | _ -> fail "cached key failure did not post exactly one completion" - in - Alcotest.check - Alcotest.bool - "unusable cached key starts bounded online preparation" - true - ((Core.state failed.next).snapshot.startup.failure = None); - let recovery = failed in - Alcotest.check - Alcotest.bool - "automatic recovery requests E2EE authorization" - true - (List.exists - (function - | Core.Run (Core.Request (_, Core.Fetch_e2ee_graph_key _)) -> true - | Run _ | Delegate _ | Publish _ -> false) - recovery.effects); - Alcotest.(check bool) - "failed cached key is not stored as a usable handle" - true - (match - protected_run - runner - ~tasks - ~posted - failed.next - "decrypt-failed" - (Core.graph_key_handle_scope handle) - (Core.Decrypt_value (handle, "plaintext")) - with - | Error (Core.Effect_failed _) -> true - | _ -> false)))) -;; - -let account_deletion_effect () = - let authenticated = - Core.step (core ()) (Account_authenticated { user_id = Some "account-user" }) - in - let signed_out = - Core.step authenticated.next (Account_authenticated { user_id = None }) - in - List.find_map - (function - | Core.Run (Core.Request (_, Core.Delete_account_secrets _) as runnable) -> - Some runnable - | Run _ | Delegate _ | Publish _ -> None) - signed_out.effects - |> Option.get -;; - -let test_account_deletion_callback_receives_exact_identity_once () = - with_support (fun support -> - Eio_main.run (fun environment -> - Eio.Switch.run (fun sw -> - let tasks = ref [] in - let posted = ref [] in - let accounts = ref [] in - let secrets_dependency = - Runner.secrets - ~unlock_private_key: - (fun - ~managed_sync_origin:_ ~user_id:_ ~password:_ ~private_key_package:_ -> - Error "unexpected private-key unlock") - ~unlock_graph_key: - (fun - ~managed_sync_origin:_ ~user_id:_ ~encrypted_graph_key:_ -> - Error "unexpected graph-key unlock") - ~load_wrapped_graph_key:(fun ~managed_sync_origin:_ ~user_id:_ ~graph_id:_ -> - Error (Runner.Wrapped_graph_key_unavailable "unexpected wrapped-key load")) - ~verify_and_save_wrapped_graph_key: - (fun - ~managed_sync_origin:_ ~user_id:_ ~graph_id:_ ~encrypted_graph_key:_ -> - Error "unexpected wrapped-key save") - ~delete_account_secrets:(fun ~managed_sync_origin ~user_id -> - accounts := (Uri.to_string managed_sync_origin, user_id) :: !accounts; - Ok ()) - |> Result.get_ok - in - let dependencies = - dependencies - ~secrets_dependency - ~environment - ~support - ~fork:(fun ~sw:_ task -> tasks := task :: !tasks) - () - in - let runner = - Runner.create ~sw dependencies ~post:(fun event -> posted := event :: !posted) - |> Result.get_ok - in - Runner.submit runner (account_deletion_effect ()); - List.iter (fun task -> task ()) (List.rev !tasks); - Alcotest.(check (list (pair string string))) - "account callback receives the captured identity once" - [ "https://api.logseq.io", "account-user" ] - (List.rev !accounts); - Alcotest.(check int) "account deletion completes" 1 (List.length !posted)))) -;; - -let test_account_deletion_callback_failures_are_typed () = - with_support (fun support -> - Eio_main.run (fun environment -> - Eio.Switch.run (fun sw -> - let tasks = ref [] in - let posted = ref [] in - let secrets_dependency = - Runner.secrets - ~unlock_private_key: - (fun - ~managed_sync_origin:_ ~user_id:_ ~password:_ ~private_key_package:_ -> - Error "unexpected private-key unlock") - ~unlock_graph_key: - (fun - ~managed_sync_origin:_ ~user_id:_ ~encrypted_graph_key:_ -> - Error "unexpected graph-key unlock") - ~load_wrapped_graph_key:(fun ~managed_sync_origin:_ ~user_id:_ ~graph_id:_ -> - Error (Runner.Wrapped_graph_key_unavailable "unexpected wrapped-key load")) - ~verify_and_save_wrapped_graph_key: - (fun - ~managed_sync_origin:_ ~user_id:_ ~graph_id:_ ~encrypted_graph_key:_ -> - Error "unexpected wrapped-key save") - ~delete_account_secrets:(fun ~managed_sync_origin:_ ~user_id:_ -> - Error "account-delete") - |> Result.get_ok - in - let dependencies = - dependencies - ~secrets_dependency - ~environment - ~support - ~fork:(fun ~sw:_ task -> tasks := task :: !tasks) - () - in - let runner = - Runner.create ~sw dependencies ~post:(fun event -> posted := event :: !posted) - |> Result.get_ok - in - Runner.submit runner (account_deletion_effect ()); - List.iter (fun task -> task ()) (List.rev !tasks); - let failures = - List.rev !posted - |> List.map (function - | Core.Runner_completed - (Core.Completion (_, Error (Core.Effect_failed message))) -> message - | _ -> fail "deletion callback did not map its error to Effect_failed") - in - Alcotest.(check (list string)) - "callback errors remain typed" - [ "account-delete" ] - failures))) -;; - -let private_key_unlock_and_sign_out () = - let authenticated = - Core.step (core ()) (Account_authenticated { user_id = Some "serialized-user" }) - in - let catalog = - List.find_map - (function - | Core.Run (Core.Request (ticket, Core.Fetch_catalog _)) -> - Some - (Core.step - authenticated.next - (Runner_completed (Completion (ticket, Ok [ encrypted_graph () ])))) - | Run _ | Delegate _ | Publish _ -> None) - authenticated.effects - |> Option.get - in - let selected = Core.step catalog.next (Graph_selected (graph_id ())) in - let scope = Core.admitted_graph_scope selected.next |> Option.get in - let missing = Core.step selected.next (Mirror_inspected (Mirror_absent scope)) in - let failed = - List.find_map - (function - | Core.Run (Core.Request (ticket, Core.Load_and_unlock_graph_key _)) -> - Some - (Core.step - missing.next - (Runner_completed - (Completion (ticket, Error (Core.Effect_failed "missing key"))))) - | Run _ | Delegate _ | Publish _ -> None) - missing.effects - |> Option.get - in - let recovery = failed in - let user_key_fetch = - List.find_map - (function - | Core.Run (Core.Request (ticket, Core.Fetch_e2ee_graph_key _)) -> - Some - (Core.step - recovery.next - (Runner_completed (Completion (ticket, Ok "encrypted-graph-key")))) - | Run _ | Delegate _ | Publish _ -> None) - recovery.effects - |> Option.get - in - let password_prompt = - List.find_map - (function - | Core.Run (Core.Request (ticket, Core.Fetch_e2ee_user_keys _)) -> - Some - (Core.step - user_key_fetch.next - (Runner_completed (Completion (ticket, Ok "private-key-package")))) - | Run _ | Delegate _ | Publish _ -> None) - user_key_fetch.effects - |> Option.get - in - let unlocking = Core.step password_prompt.next (Core.E2ee_password_submitted - (Core.unlock_challenge (Core.state password_prompt.next).snapshot.unlock |> Option.get, "password")) in - let unlock = - List.find_map - (function - | Core.Run (Core.Request (_, Core.Unlock_private_key _) as runnable) -> - Some runnable - | Run _ | Delegate _ | Publish _ -> None) - unlocking.effects - |> Option.get - in - let signed_out = Core.step unlocking.next (Account_authenticated { user_id = None }) in - unlock, signed_out.effects -;; - -let test_secret_cleanup_is_serialized_after_an_older_write () = - with_support (fun support -> - Eio_main.run (fun environment -> - Eio.Switch.run (fun sw -> - let write_started, resolve_write_started = Eio.Promise.create () in - let release_write, resolve_release_write = Eio.Promise.create () in - let deletion_finished, resolve_deletion_finished = Eio.Promise.create () in - let order = ref [] in - let secrets_dependency = - Runner.secrets - ~unlock_private_key: - (fun - ~managed_sync_origin:_ ~user_id:_ ~password:_ ~private_key_package:_ -> - Eio.Cancel.protect (fun () -> - order := "write-started" :: !order; - Eio.Promise.resolve resolve_write_started (); - Eio.Promise.await release_write; - order := "write-finished" :: !order; - Ok ())) - ~unlock_graph_key: - (fun - ~managed_sync_origin:_ ~user_id:_ ~encrypted_graph_key:_ -> - Error "unexpected graph-key unlock") - ~load_wrapped_graph_key:(fun ~managed_sync_origin:_ ~user_id:_ ~graph_id:_ -> - Error (Runner.Wrapped_graph_key_unavailable "unexpected wrapped-key load")) - ~verify_and_save_wrapped_graph_key: - (fun - ~managed_sync_origin:_ ~user_id:_ ~graph_id:_ ~encrypted_graph_key:_ -> - Error "unexpected wrapped-key save") - ~delete_account_secrets:(fun ~managed_sync_origin:_ ~user_id:_ -> - order := "delete-account" :: !order; - Eio.Promise.resolve resolve_deletion_finished (); - Ok ()) - |> Result.get_ok - in - let dependencies = - dependencies - ~secrets_dependency - ~environment - ~support - ~fork:(fun ~sw task -> Eio.Fiber.fork ~sw task) - () - in - let runner = - Runner.create ~sw dependencies ~post:(fun _ -> ()) |> Result.get_ok - in - let unlock, sign_out_effects = private_key_unlock_and_sign_out () in - Runner.submit runner unlock; - Eio.Promise.await write_started; - List.iter - (function - | Core.Run runner_effect -> Runner.submit runner runner_effect - | Delegate _ | Publish _ -> ()) - sign_out_effects; - Eio.Fiber.yield (); - Alcotest.(check bool) - "delete waits while the older write is open" - false - (List.mem "delete-account" !order); - Eio.Promise.resolve resolve_release_write (); - Eio.Promise.await deletion_finished; - Alcotest.(check (list string)) - "older write finishes before deletion" - [ "write-started"; "write-finished"; "delete-account" ] - (List.rev !order)))) -;; - -let source_contents relative alternatives = - let candidates = relative :: alternatives in - match List.find_opt Sys.file_exists candidates with - | None -> fail "unable to locate source file %s" relative - | Some filename -> - let channel = open_in_bin filename in - Fun.protect - ~finally:(fun () -> close_in_noerr channel) - (fun () -> really_input_string channel (in_channel_length channel)) -;; - -let contains text needle = - let rec loop offset = - if offset + String.length needle > String.length text - then false - else if String.sub text offset (String.length needle) = needle - then true - else loop (offset + 1) - in - String.length needle = 0 || loop 0 -;; - -let test_runner_source_has_no_placeholder_capabilities () = - let source = - source_contents - "logseq_sync/lib/effect_runner/effect_runner.ml" - [ "../lib/effect_runner/effect_runner.ml" ] - in - List.iter - (fun forbidden -> - if contains source forbidden - then fail "runner retains placeholder capability %s" forbidden) - [ "monotonic_ns" - ; "has_private_key" - ; "decrypt_private_key" - ; "decrypt_graph_key" - ; "ignore secrets.delete_account_secrets" - ] -;; - -let scenarios = - [ Alcotest.test_case - "cancelled queued catalog save never writes" - `Quick - test_cancelled_queued_catalog_save_does_not_write - ; Alcotest.test_case - "dependency constructors validate resources" - `Quick - test_dependency_constructors_validate_owned_resources - ; Alcotest.test_case - "catalog load uses account and origin scoped path" - `Quick - test_catalog_load_uses_account_and_origin_scoped_path - ; Alcotest.test_case - "submit posts asynchronously" - `Quick - test_submit_is_async_and_posts_a_typed_completion - ; Alcotest.test_case - "cancellation suppresses late completion" - `Quick - test_cancellation_suppresses_late_completion - ; Alcotest.test_case - "cached wrapped key is unlocked before AES" - `Quick - (test_cached_wrapped_key_is_unlocked_before_protected_value_decryption - ~malformed_plaintext:false) - ; Alcotest.test_case - "network cancellation retains local graph key bytes" - `Quick - (test_cached_wrapped_key_is_unlocked_before_protected_value_decryption ~cancel_online:true) - ; Alcotest.test_case - "decrypted parser error excludes plaintext" - `Quick - (test_cached_wrapped_key_is_unlocked_before_protected_value_decryption - ~malformed_plaintext:true) - ; Alcotest.test_case - "crypto error excludes sensitive detail" - `Quick - (test_cached_wrapped_key_is_unlocked_before_protected_value_decryption - ~decryption_failure:true) - ; Alcotest.test_case - "cached wrapped key unlock failure is fail-closed" - `Quick - test_cached_wrapped_key_unlock_failure_is_fail_closed - ; Alcotest.test_case - "account deletion callback receives exact identity once" - `Quick - test_account_deletion_callback_receives_exact_identity_once - ; Alcotest.test_case - "account deletion callback failures are typed" - `Quick - test_account_deletion_callback_failures_are_typed - ; Alcotest.test_case - "secret cleanup is serialized after older writes" - `Quick - test_secret_cleanup_is_serialized_after_an_older_write - ; Alcotest.test_case - "runner source has no placeholder capabilities" - `Quick - test_runner_source_has_no_placeholder_capabilities - ] -;; - -(* These assertions execute staging/lease filesystem ownership, which pure Core - cannot reproduce; policy acceptance itself is covered by Core_contract. *) -let test_asset_local_lifecycle_uses_completion_outputs () = - with_support (fun support -> - Eio_main.run (fun environment -> - Eio.Switch.run (fun sw -> - let tasks = ref [] - and posted = ref [] in - let deps = - dependencies - ~environment - ~support - ~fork:(fun ~sw:_ task -> tasks := task :: !tasks) - () - in - let runner = - Runner.create ~sw deps ~post:(fun event -> posted := event :: !posted) - |> Result.get_ok - in - let selected, scope = Core_contract.selected_graph Core_contract.graph in - let core = ref selected.next in - let rec pump outputs = - match !tasks, !posted with - | [], [] -> List.rev outputs - | task :: rest, _ -> - tasks := rest; - task (); - pump outputs - | [], event :: rest -> - posted := rest; - let transition = Core.step !core event in - core := transition.next; - let outputs = - List.fold_left - (fun outputs -> function - | Core.Run runnable -> - Runner.submit runner runnable; - outputs - | Core.Publish (Core.Asset_finished output) -> output :: outputs - | _ -> outputs) - outputs - transition.effects - in - pump outputs - in - let request operation action = - let transition = - Core.step !core (Core.Asset_requested { scope; operation; action }) - in - core := transition.next; - List.iter - (function - | Core.Run runnable -> Runner.submit runner runnable - | _ -> ()) - transition.effects; - match pump [] with - | [ output ] -> output.Core.result - | _ -> fail "local asset action did not resolve through Core" - in - let source_file = Filename.concat support "import-source.bin" in - Out_channel.with_open_bin source_file (fun out -> output_string out "local asset"); - let file, checksum = - match - request - "stage" - (Core.Stage_asset_file - { operation = graph_id (); file_type = "bin"; source_file }) - with - | Ok (Core.Asset_staged { file; checksum; size }) -> - Alcotest.(check int64) "staging returns metadata" 11L size; - file, checksum - | _ -> fail "staging did not complete" - in - let lease, path = - match request "retain" (Core.Retain_staged_file file) with - | Ok (Core.Asset_retained (Some (lease, path))) -> lease, path - | _ -> fail "staging lease did not complete" - in - Alcotest.(check bool) - "controlled lease resolves existing path" - true - (Sys.file_exists path); - Alcotest.(check bool) - "release lease completes" - true - (request "release-lease" (Core.Release_asset_file lease) = Ok Core.Asset_unit); - Alcotest.(check bool) - "release staging completes" - true - (request "release-stage" (Core.Release_staged_file file) = Ok Core.Asset_unit); - Alcotest.(check bool) "released staging is removed" false (Sys.file_exists path); - let version = - Logseq_db_types.Asset_descriptor.version ~checksum ~file_type:"bin" - |> Result.get_ok - in - Alcotest.(check bool) - "cache miss passes through Core output" - true - (request "cache" (Core.Check_asset_cache (graph_id (), version)) - = Ok (Core.Asset_cached None)); - Alcotest.(check bool) - "prune passes through Core output" - true - (match request "prune" (Core.Prune_asset_staging []) with - | Ok (Core.Asset_pruned _) -> true - | _ -> false); - Alcotest.(check bool) - "retry timer passes through Core output" - true - (request "timer" (Core.Asset_retry_after 0.25) = Ok Core.Asset_retry_elapsed); - Alcotest.(check bool) - "scope close completes" - true - (request "close" Core.Close_asset_scope = Ok Core.Asset_unit); - Alcotest.(check bool) - "graph deletion completes" - true - (retire_cache (fun action -> request "delete-graph" action) = Ok Core.Asset_unit); - Alcotest.(check bool) - "account deletion completes" - true - (request "delete-account" Core.Delete_account_assets = Ok Core.Asset_unit); - Runner.shutdown runner))) -;; - -let scenarios = - scenarios - @ [ Alcotest.test_case - "asset local IO resolves through Core outputs" - `Quick - test_asset_local_lifecycle_uses_completion_outputs - ] -;; - -(* Core cannot prevent a runner from executing the same submitted capability twice. - Re-execution of Retain would mint an unaccepted lease and strand staging cleanup. *) -let test_asset_submit_replay_does_not_mint_a_second_lease () = - with_support (fun support -> - Eio_main.run (fun environment -> - Eio.Switch.run (fun sw -> - let tasks = ref [] - and posted = ref [] in - let deps = - dependencies - ~environment - ~support - ~fork:(fun ~sw:_ task -> tasks := task :: !tasks) - () - in - let runner = - Runner.create ~sw deps ~post:(fun event -> posted := event :: !posted) - |> Result.get_ok - in - let selected, scope = Core_contract.selected_graph Core_contract.graph in - let run_tasks () = - while !tasks <> [] do - let batch = List.rev !tasks in - tasks := []; - List.iter (fun task -> task ()) batch - done - in - let complete transition = - posted := []; - List.iter - (function - | Core.Run runnable -> Runner.submit runner runnable - | _ -> ()) - transition.Core.effects; - run_tasks (); - match List.rev !posted with - | [ event ] -> Core.step transition.next event - | _ -> fail "resource command did not post one completion" - in - let source_file = Filename.concat support "replay-source.bin" in - Out_channel.with_open_bin source_file (fun out -> output_string out "replay"); - let stage = - complete - (Core.step - selected.next - (Core.Asset_requested - { scope - ; operation = "replay-stage" - ; action = - Core.Stage_asset_file - { operation = graph_id (); file_type = "bin"; source_file } - })) - in - let file = - List.find_map - (function - | Core.Publish - (Core.Asset_finished { result = Ok (Core.Asset_staged { file; _ }); _ }) - -> Some file - | _ -> None) - stage.effects - |> Option.get - in - let retaining = - Core.step - stage.next - (Core.Asset_requested - { scope - ; operation = "replay-retain" - ; action = Core.Retain_staged_file file - }) - in - let runnable = - List.find_map - (function - | Core.Run (Core.Asset_io _ as runnable) -> Some runnable - | _ -> None) - retaining.effects - |> Option.get - in - let retained = complete retaining in - let lease, path = - List.find_map - (function - | Core.Publish - (Core.Asset_finished - { result = Ok (Core.Asset_retained (Some retained)); _ }) -> - Some retained - | _ -> None) - retained.effects - |> Option.get - in - posted := []; - Gc.full_major (); - Runner.submit runner runnable; - run_tasks (); - Alcotest.(check int) - "accepted effect replay creates no new completion or lease" - 0 - (List.length !posted); - let released = - complete - (Core.step - retained.next - (Core.Asset_requested - { scope - ; operation = "replay-release-lease" - ; action = Core.Release_asset_file lease - })) - in - ignore - (complete - (Core.step - released.next - (Core.Asset_requested - { scope - ; operation = "replay-release-stage" - ; action = Core.Release_staged_file file - }))); - Alcotest.(check bool) - "original lease release permits final staging cleanup" - false - (Sys.file_exists path); - Runner.shutdown runner))) -;; - -let scenarios = - scenarios - @ [ Alcotest.test_case - "asset submit replay cannot mint a second lease" - `Quick - test_asset_submit_replay_does_not_mint_a_second_lease - ] -;; - -let test_asset_cleanup_survives_shutdown_before_queued_fork () = - with_support (fun support -> - Eio_main.run (fun environment -> - Eio.Switch.run (fun sw -> - let tasks = ref [] - and posted = ref [] in - let deps = - dependencies - ~environment - ~support - ~fork:(fun ~sw:_ task -> tasks := task :: !tasks) - () - in - let runner = - Runner.create ~sw deps ~post:(fun event -> posted := event :: !posted) - |> Result.get_ok - in - let selected, scope = Core_contract.selected_graph Core_contract.graph in - let core = ref selected.next in - let drain () = - while !tasks <> [] do - let batch = List.rev !tasks in - tasks := []; - List.iter (fun task -> task ()) batch - done - in - let request operation action = - posted := []; - let transition = - Core.step !core (Core.Asset_requested { scope; operation; action }) - in - core := transition.next; - List.iter - (function - | Core.Run runnable -> Runner.submit runner runnable - | _ -> ()) - transition.effects; - drain (); - match List.rev !posted with - | [ event ] -> - let completed = Core.step !core event in - core := completed.next; - List.find_map - (function - | Core.Publish (Core.Asset_finished output) -> Some output.result - | _ -> None) - completed.effects - |> Option.get - | _ -> fail "cleanup fixture command did not complete" - in - let source_file = Filename.concat support "shutdown-source.bin" in - Out_channel.with_open_bin source_file (fun out -> output_string out "shutdown"); - let file = - match - request - "shutdown-stage" - (Core.Stage_asset_file - { operation = graph_id (); file_type = "bin"; source_file }) - with - | Ok (Core.Asset_staged { file; _ }) -> file - | _ -> fail "shutdown fixture staging failed" - in - let lease, path = - match request "shutdown-retain" (Core.Retain_staged_file file) with - | Ok (Core.Asset_retained (Some retained)) -> retained - | _ -> fail "shutdown fixture retention failed" - in - ignore (request "shutdown-release-lease" (Core.Release_asset_file lease)); - posted := []; - let cleanup = - Core.step - !core - (Core.Asset_requested - { scope - ; operation = "shutdown-release-stage" - ; action = Core.Release_staged_file file - }) - in - List.iter - (function - | Core.Run runnable -> Runner.submit runner runnable - | _ -> ()) - cleanup.effects; - Runner.shutdown runner; - drain (); - Alcotest.(check bool) - "submitted staging cleanup survives shutdown before queued tasks" - false - (Sys.file_exists path); - let completed = - match List.rev !posted with - | [ event ] -> Core.step cleanup.next event - | _ -> fail "submitted cleanup did not post exactly one completion" - in - Alcotest.(check bool) - "cleanup completion reaches its reducer output" - true - (List.exists - (function - | Core.Publish (Core.Asset_finished output) -> - output.request.operation = "shutdown-release-stage" - && output.result = Ok Core.Asset_unit - | _ -> false) - completed.effects)))) -;; - -let scenarios = - scenarios - @ [ Alcotest.test_case - "asset queued cleanup survives runner shutdown" - `Quick - test_asset_cleanup_survives_shutdown_before_queued_fork - ] -;; - -(* Migrated asset cache public runner scenarios. *) -let with_local_asset_cache ?(budget = 8L) ?(pending_budget = 8L) ?(with_causes = false) - ?on_asset_diagnostic ?(acquire = fun _ -> Ok "token") test = - with_support (fun support -> - Eio_main.run (fun environment -> - Eio.Switch.run (fun sw -> - let tasks = Queue.create () - and posted = Queue.create () in - let deps = - let transport = - Runner.transport - ~websocket_liveness:Runner.Disabled - ~tls_authenticator:(Runner.system_tls_authenticator () |> Result.get_ok) - ~network:(Eio.Stdenv.net environment) - ~clock:(Eio.Stdenv.clock environment) - |> Result.get_ok - in - Runner.dependencies - ~runtime: - (Runner.runtime - ~fork:(fun ~sw:_ task -> Queue.add task tasks) - ~sleep:(fun _ -> ()) - |> Result.get_ok) - ~transport - ~local_store: - (Runner.local_store - ~asset_cache_budget_bytes:budget - ~asset_maximum_file_bytes:8 - ~asset_pending_budget_bytes:pending_budget - ~application_support_directory:support - () - |> Result.get_ok) - ~artifact_store: - (Runner.artifact_store - ~staging_directory:(Filename.concat support "staging") - |> Result.get_ok) - ~secrets:(secrets ()) - ~crypto:(crypto ()) - ~id_token_provider: - (Runner.id_token_provider - ~acquire - ~invalidate:(fun _ ~token:_ -> ())) - |> Result.get_ok - in - let runner = - Runner.create ?on_asset_diagnostic ~sw deps ~post:(fun event -> Queue.add event posted) - |> Result.get_ok - in - Fun.protect - ~finally:(fun () -> Runner.shutdown runner) - (fun () -> - let selected, scope = Core_contract.selected_graph Core_contract.graph in - let state = ref selected.next - and sequence = ref 0 in - let rec consume outputs effects = - List.fold_left - (fun outputs -> function - | Core.Run runner_effect -> - Runner.submit runner runner_effect; - outputs - | Core.Publish (Core.Asset_finished output) -> output :: outputs - | _ -> outputs) - outputs - effects - and pump outputs = - match Queue.take_opt tasks with - | Some task -> - task (); - pump outputs - | None -> - (match Queue.take_opt posted with - | None -> List.rev outputs - | Some event -> - let transition = Core.step !state event in - state := transition.next; - pump (consume outputs transition.effects)) - in - let request action = - incr sequence; - let transition = - Core.step - !state - (Core.Asset_requested - { scope; operation = "cache-" ^ string_of_int !sequence; action }) - in - state := transition.next; - match pump (consume [] transition.effects) with - | [ output ] -> if with_causes then output.Core.result - else Result.map_error Core.asset_failure_kind output.Core.result - | _ -> fail "cache operation did not resolve through reducer output" - in - test support request)))) -;; - -let staged_value = function - | Ok (Core.Asset_staged { file; checksum; size }) -> file, checksum, size - | _ -> fail "expected staged resource metadata" -;; - -let retained_value = function - | Ok (Core.Asset_retained (Some resource)) -> resource - | _ -> fail "expected retained resource output" -;; - -let cache_unit result = - Alcotest.(check bool) "cleanup completed" true (result = Ok Core.Asset_unit) -;; - -let cache_bool = Alcotest.check Alcotest.bool - -let cache_stage - request - ?(operation = Core_contract.graph_id) - ?(file_type = "bin") - source_file - = - request (Core.Stage_asset_file { operation; file_type; source_file }) |> staged_value -;; - -let test_cache_durable_staging () = - with_local_asset_cache (fun support request -> - let source = Filename.concat support "picker.bin" in - Out_channel.with_open_bin source (fun output -> output_string output "source"); - let file, checksum, size = cache_stage request source in - Alcotest.(check string) - "staging checksum" - (Logseq_sync_effect_runner.Asset_codec.checksum "source") - checksum; - Alcotest.(check int64) "staging size" 6L size; - Sys.remove source; - let lease, path = request (Core.Retain_staged_file file) |> retained_value in - Alcotest.(check string) - "picker is no longer needed" - "source" - (In_channel.with_open_bin path In_channel.input_all); - cache_unit (request (Core.Release_asset_file lease)); - let interrupted = path ^ ".part" in - Out_channel.with_open_bin interrupted (fun output -> output_string output "partial"); - cache_unit (request Core.Close_asset_scope); - let lease, restored = request (Core.Retain_staged_file file) |> retained_value in - cache_bool "restart preserves pending source" true (String.equal restored path); - cache_bool "interrupted staging discarded" false (Sys.file_exists interrupted); - cache_bool - "path traversal rejected" - true - (request (Core.Retain_staged_file "../picker.bin") = Ok (Core.Asset_retained None)); - cache_unit (request (Core.Release_asset_file lease)); - cache_unit (request (Core.Release_staged_file file)); - cache_unit (request (Core.Release_staged_file file)); - cache_bool - "completion releases pending source" - true - (request (Core.Retain_staged_file file) = Ok (Core.Asset_retained None))) -;; - -let test_cache_deletion_waits_for_native_lease () = - with_local_asset_cache (fun support request -> - let source = Filename.concat support "preview.bin" in - Out_channel.with_open_bin source (fun output -> output_string output "preview"); - let file, _, _ = cache_stage request source in - let lease, path = request (Core.Retain_staged_file file) |> retained_value in - cache_bool "deletion waits or reports resources in use" true - (Result.is_error (retire_cache request)); - cache_bool "live preview keeps its file" true (Sys.file_exists path); - cache_unit (request (Core.Release_asset_file lease)); - (match retire_cache request with - | Ok Core.Asset_unit -> () - | Error (Core.Asset_invalid_content message) -> Alcotest.fail message - | Error Core.Asset_cancelled -> Alcotest.fail "cleanup unexpectedly cancelled" - | _ -> Alcotest.fail "cleanup returned another result"); - cache_bool "retry removes released file" false (Sys.file_exists path)) -;; - -let test_asset_retirement_cancel_and_replacement () = - with_local_asset_cache (fun support request -> - let source = Filename.concat support "retirement-source.bin" in - Out_channel.with_open_bin source (fun output -> output_string output "source"); - let file, _, _ = cache_stage request source in - let lease, original = request (Core.Retain_staged_file file) |> retained_value in - cache_unit (request (Core.Release_asset_file lease)); - let id text = Core.retirement_id_of_string text |> Result.get_ok in - let retirement = id "96000000-0000-4000-8000-000000000011" in - let deletion = match request (Core.Prepare_graph_asset_deletion retirement) with - | Ok (Core.Asset_deletion_prepared deletion) -> deletion - | _ -> Alcotest.fail "retirement did not prepare" in - cache_bool "prepared source is only retired" false (Sys.file_exists original); - cache_unit (request (Core.Cancel_graph_asset_deletion deletion)); - cache_bool "cancel restores exact source" true (Sys.file_exists original); - cache_unit (request (Core.Commit_graph_asset_deletion deletion)); - cache_bool "late commit cannot delete aborted source" true (Sys.file_exists original); - let retirement = id "96000000-0000-4000-8000-000000000012" in - let deletion = match request (Core.Prepare_graph_asset_deletion retirement) with - | Ok (Core.Asset_deletion_prepared deletion) -> deletion - | _ -> Alcotest.fail "second retirement did not prepare" in - cache_unit (request (Core.Commit_graph_asset_deletion deletion)); - cache_bool "committed old source removed" false (Sys.file_exists original); - let fresh_file, _, _ = cache_stage request source in - let lease, fresh_path = request (Core.Retain_staged_file fresh_file) |> retained_value in - cache_unit (request (Core.Cancel_graph_asset_deletion deletion)); - cache_unit (request (Core.Commit_graph_asset_deletion deletion)); - cache_bool "old terminal replay preserves replacement source" true (Sys.file_exists fresh_path); - cache_unit (request (Core.Release_asset_file lease))) -;; - -let test_asset_retirement_restore_conflict () = - with_local_asset_cache (fun support request -> - let source = Filename.concat support "restore-source.bin" in - Out_channel.with_open_bin source (fun output -> output_string output "source"); - let file, _, _ = cache_stage request source in - let lease, original = request (Core.Retain_staged_file file) |> retained_value in - cache_unit (request (Core.Release_asset_file lease)); - let retirement = Core.retirement_id_of_string "96000000-0000-4000-8000-000000000013" |> Result.get_ok in - let deletion = match request (Core.Prepare_graph_asset_deletion retirement) with - | Ok (Core.Asset_deletion_prepared deletion) -> deletion - | _ -> Alcotest.fail "retirement did not prepare" in - let rec first_missing_ancestor location = - let parent = Filename.dirname location in - if Sys.file_exists parent then location else first_missing_ancestor parent in - let canonical = first_missing_ancestor (Filename.dirname original) in - Unix.mkdir canonical 0o700; - let sentinel = Filename.concat canonical "new-synthetic-copy" in - Out_channel.with_open_bin sentinel (fun output -> output_string output "replacement"); - cache_bool "restore conflict is reported" true - (Result.is_error (request (Core.Cancel_graph_asset_deletion deletion))); - cache_bool "restore does not overwrite replacement" true (Sys.file_exists sentinel); - Sys.remove sentinel; Unix.rmdir canonical; - cache_unit (request (Core.Cancel_graph_asset_deletion deletion)); - cache_bool "retry restores original bytes" true - (In_channel.with_open_bin original In_channel.input_all = "source")) -;; - -let test_cache_staging_limits () = - with_local_asset_cache ~pending_budget:4L (fun support request -> - let source = Filename.concat support "picker.bin" in - let write bytes = - Out_channel.with_open_bin source (fun output -> output_string output bytes) - in - write "large-file"; - cache_bool - "oversize staging rejected" - true - (Result.is_error - (request - (Core.Stage_asset_file - { operation = Core_contract.graph_id - ; file_type = "bin" - ; source_file = source - }))); - write "file"; - let file, _, _ = cache_stage request source in - cache_bool - "duplicate cannot overwrite immutable staging" - true - (Result.is_error - (request - (Core.Stage_asset_file - { operation = Core_contract.graph_id - ; file_type = "bin" - ; source_file = source - }))); - cache_bool - "pending namespace has a separate budget" - true - (request - (Core.Stage_asset_file - { operation = Core_contract.other_graph_id - ; file_type = "bin" - ; source_file = source - }) - = Error Core.Asset_storage_full); - let lease, path = request (Core.Retain_staged_file file) |> retained_value in - cache_bool "failed staging leaves original intact" true (Sys.file_exists path); - cache_unit (request (Core.Release_asset_file lease)); - cache_unit (retire_cache request); - cache_bool "graph deletion removes staging" false (Sys.file_exists path)) -;; - -let test_cache_orphan_staging () = - with_local_asset_cache ~pending_budget:16L (fun support request -> - let source = Filename.concat support "picker.bin" in - Out_channel.with_open_bin source (fun output -> output_string output "file"); - let retained, _, _ = cache_stage request source in - let orphan, _, _ = - cache_stage request ~operation:Core_contract.other_graph_id source - in - let first_lease, retained_path = - request (Core.Retain_staged_file retained) |> retained_value - in - let second_lease, orphan_path = - request (Core.Retain_staged_file orphan) |> retained_value - in - cache_unit (request (Core.Release_asset_file first_lease)); - cache_unit (request (Core.Release_asset_file second_lease)); - let directory = Filename.dirname orphan_path in - let stranger = Filename.concat directory "unrecognized.txt" in - Out_channel.with_open_bin stranger (fun output -> output_string output "preserve"); - let link = Filename.concat directory "00000000-0000-4000-8000-000000000003.bin" in - Unix.symlink source link; - Alcotest.(check bool) - "only orphan removed" - true - (request (Core.Prune_asset_staging [ retained ]) = Ok (Core.Asset_pruned 1)); - cache_bool "durable staging retained" true (Sys.file_exists retained_path); - cache_bool "orphan gone" false (Sys.file_exists orphan_path); - cache_bool "unknown file retained" true (Sys.file_exists stranger); - cache_bool - "symlink not followed or removed" - true - ((Unix.lstat link).st_kind = Unix.S_LNK && Sys.file_exists source); - cache_bool - "repeat cleanup" - true - (request (Core.Prune_asset_staging [ retained ]) = Ok (Core.Asset_pruned 0)); - for index = 1 to 4096 do - Out_channel.with_open_bin - (Filename.concat directory ("unknown-" ^ string_of_int index)) - (fun _ -> ()) - done; - cache_bool - "oversized directory fails closed" - true - (request (Core.Prune_asset_staging []) = Error Core.Asset_storage_full); - cache_bool - "capacity failure preserves staged data" - true - (Sys.file_exists retained_path)) -;; - -let test_cache_staged_preview () = - with_local_asset_cache (fun support request -> - let source = Filename.concat support "picker.bin" in - Out_channel.with_open_bin source (fun output -> output_string output "file"); - let file, _, _ = cache_stage request ~file_type:"pdf" source in - let lease, path = request (Core.Retain_staged_file file) |> retained_value in - cache_bool - "native preview retains document extension" - true - (Filename.check_suffix path ".pdf"); - List.iter - (fun file_type -> - cache_bool - "unsafe file types are rejected" - true - (Result.is_error - (request - (Core.Stage_asset_file - { operation = Core_contract.graph_id - ; file_type - ; source_file = source - })))) - [ ""; "../pdf"; "x.pdf"; String.make 33 'a' ]; - let second, _ = request (Core.Retain_asset_file lease) |> retained_value in - cache_bool - "orphan cleanup preserves active previews" - true - (request (Core.Prune_asset_staging []) = Ok (Core.Asset_pruned 0)); - cache_unit (request (Core.Release_staged_file file)); - cache_bool "completion keeps preview bytes" true (Sys.file_exists path); - cache_bool - "completed staging refuses new preview" - true - (request (Core.Retain_staged_file file) = Ok (Core.Asset_retained None)); - cache_unit (request (Core.Release_asset_file lease)); - cache_bool "one remaining lease keeps file" true (Sys.file_exists path); - cache_unit (request (Core.Release_asset_file second)); - cache_bool "last lease finishes cleanup" false (Sys.file_exists path); - cache_unit (request (Core.Release_asset_file second)); - cache_unit (request (Core.Release_staged_file file)); - let file, _, _ = cache_stage request ~file_type:"pdf" source in - let lease, path = request (Core.Retain_staged_file file) |> retained_value in - cache_unit (request (Core.Release_staged_file file)); - cache_bool "close cannot invalidate a live preview" true - (Result.is_error(request Core.Close_asset_scope)); - cache_bool "failed close preserves retained bytes" true(Sys.file_exists path); - cache_unit(request(Core.Release_asset_file lease)); - cache_unit(request Core.Close_asset_scope); - cache_bool "checked release finishes deferred cleanup" false(Sys.file_exists path); - let file, _, _ = cache_stage request ~file_type:"pdf" source in - let lease, path = request (Core.Retain_staged_file file) |> retained_value in - cache_unit(request(Core.Release_asset_file lease)); - cache_unit (request Core.Close_asset_scope); - cache_bool "close preserves nonterminal staging" true (Sys.file_exists path)) -;; - -let scenarios = - scenarios - @ [ Alcotest.test_case "graph deletion waits for native preview lease" `Quick test_cache_deletion_waits_for_native_lease ] - @ List.map - (fun (name, test) -> Alcotest.test_case name `Quick test) - [ "cache durable staging via submit", test_cache_durable_staging - ; "cache staging limits via submit", test_cache_staging_limits - ; "cache orphan staging via submit", test_cache_orphan_staging - ; "cache staged preview lifetime via submit", test_cache_staged_preview - ] -;; - -(* The runner owns distinct physical staging instances; Core only fences the - cancelled completion and asks to release that exact resource reference. *) -let test_late_stage_cleanup_cannot_delete_a_replacement () = - with_support (fun support -> - Eio_main.run (fun environment -> - Eio.Switch.run (fun sw -> - let tasks = ref [] - and posted = ref [] in - let deps = - dependencies - ~environment - ~support - ~fork:(fun ~sw:_ task -> tasks := task :: !tasks) - () - in - let runner = - Runner.create ~sw deps ~post:(fun event -> posted := event :: !posted) - |> Result.get_ok - in - let selected, scope = Core_contract.selected_graph Core_contract.graph in - let core = ref selected.next in - let drain () = - while !tasks <> [] do - let batch = List.rev !tasks in - tasks := []; - List.iter (fun task -> task ()) batch - done - in - let submit operation action = - posted := []; - let transition = - Core.step !core (Core.Asset_requested { scope; operation; action }) - in - core := transition.next; - List.iter - (function - | Core.Run runnable -> Runner.submit runner runnable - | _ -> ()) - transition.effects; - drain (); - List.rev !posted - in - let request operation action = - match submit operation action with - | [ event ] -> - let completed = Core.step !core event in - core := completed.next; - List.find_map - (function - | Core.Publish (Core.Asset_finished output) -> Some output.result - | _ -> None) - completed.effects - |> Option.get - | _ -> fail "stage instance fixture did not complete" - in - let source_file = Filename.concat support "instance-source.bin" in - let write bytes = - Out_channel.with_open_bin source_file (fun out -> output_string out bytes) - in - let stage = - Core.Stage_asset_file - { operation = graph_id (); file_type = "bin"; source_file } - in - write "old"; - let old_event, old_file = - match submit "old-instance" stage with - | [ (Core.Runner_completed - (Core.Asset_completion (_, Ok (Core.Asset_staged { file; _ }))) as event) - ] -> event, file - | _ -> fail "old stage did not produce a resource completion" - in - Alcotest.(check bool) - "prune removes old unaccepted staging instance" - true - (request "prune-old-instance" (Core.Prune_asset_staging []) - = Ok (Core.Asset_pruned 1)); - write "new"; - let new_file = - match request "new-instance" stage with - | Ok (Core.Asset_staged { file; _ }) -> file - | _ -> fail "replacement staging failed" - in - Alcotest.(check bool) - "same import operation gets a distinct physical resource" - false - (String.equal old_file new_file); - ignore - (request "cancel-old-instance" (Core.Cancel_asset_operation "old-instance")); - let late = Core.step !core old_event in - core := late.next; - Alcotest.(check bool) - "cancelled completion releases the original exact reference" - true - (List.exists - (function - | Core.Run (Core.Asset_io (_, io)) -> - io.action = Core.Release_staged_file old_file - | _ -> false) - late.effects); - posted := []; - List.iter - (function - | Core.Run runnable -> Runner.submit runner runnable - | _ -> ()) - late.effects; - drain (); - List.iter (fun event -> core := (Core.step !core event).next) (List.rev !posted); - let lease, path = - match request "retain-replacement" (Core.Retain_staged_file new_file) with - | Ok (Core.Asset_retained (Some retained)) -> retained - | _ -> fail "old cleanup deleted or completed replacement" - in - Alcotest.(check string) - "replacement bytes survive old cleanup" - "new" - (In_channel.with_open_bin path In_channel.input_all); - ignore (request "release-replacement-lease" (Core.Release_asset_file lease)); - ignore (request "release-replacement-stage" (Core.Release_staged_file new_file)); - Runner.shutdown runner))) -;; - -let test_cache_prune_keeps_exact_staging_instance_and_legacy_files () = - with_local_asset_cache ~pending_budget:16L (fun support request -> - let source = Filename.concat support "exact-source.bin" in - Out_channel.with_open_bin source (fun out -> output_string out "file"); - let older, _, _ = cache_stage request source in - let newest, _, _ = cache_stage request source in - cache_bool - "same operation can stage a fresh instance without replacing old bytes" - false - (String.equal older newest); - cache_bool - "prune preserves exact durable instance only" - true - (request (Core.Prune_asset_staging [ newest ]) = Ok (Core.Asset_pruned 1)); - cache_bool - "older instance removed" - true - (request (Core.Retain_staged_file older) = Ok (Core.Asset_retained None)); - let lease, path = request (Core.Retain_staged_file newest) |> retained_value in - cache_bool "new durable instance survives prune" true (Sys.file_exists path); - let legacy = - Logseq_db_types.Graph_types.Uuid.to_string Core_contract.other_graph_id ^ ".bin" - in - let legacy_path = Filename.concat (Filename.dirname path) legacy in - Out_channel.with_open_bin legacy_path (fun out -> output_string out "legacy"); - let legacy_lease, restored = - request (Core.Retain_staged_file legacy) |> retained_value - in - Alcotest.(check string) - "legacy UUID.type resource remains recoverable" - "legacy" - (In_channel.with_open_bin restored In_channel.input_all); - cache_unit (request (Core.Release_asset_file legacy_lease)); - cache_unit (request (Core.Release_staged_file legacy)); - cache_unit (request (Core.Release_asset_file lease)); - cache_unit (request (Core.Release_staged_file newest))) -;; - -let scenarios = - scenarios - @ [ Alcotest.test_case - "late staging cleanup cannot delete replacement" - `Quick - test_late_stage_cleanup_cannot_delete_a_replacement - ; Alcotest.test_case - "staging prune keeps exact instance and legacy recovery" - `Quick - test_cache_prune_keeps_exact_staging_instance_and_legacy_files - ] -;; - -(* Deletion must cancel registered work before any cache instance exists; Core's - cancelled pending state alone cannot stop a queued callback from doing I/O. *) -let test_graph_asset_deletion_cancels_queued_fetch_before_cache_open () = - with_support (fun support -> - Eio_main.run (fun environment -> - Eio.Switch.run (fun sw -> - let tasks = ref [] - and posted = ref [] - and acquisitions = ref 0 in - let deps = - dependencies - ~environment - ~support - ~fork:(fun ~sw:_ task -> tasks := task :: !tasks) - ~id_token_dependency: - (Runner.id_token_provider - ~acquire:(fun _ -> - incr acquisitions; - Error "network must not be entered") - ~invalidate:(fun _ ~token:_ -> ())) - () - in - let runner = - Runner.create ~sw deps ~post:(fun event -> posted := event :: !posted) - |> Result.get_ok - in - let selected, scope = Core_contract.selected_graph Core_contract.graph in - let version = - Logseq_db_types.Asset_descriptor.version - ~checksum:(String.make 64 'a') - ~file_type:"bin" - |> Result.get_ok - in - let fetching = - Core.step - selected.next - (Core.Asset_requested - { scope - ; operation = "queued-fetch-before-cache" - ; action = - Core.Fetch_asset - { asset = graph_id (); version; maximum_plaintext_bytes = 8 } - }) - in - let expected_ticket = - List.find_map - (function - | Core.Run (Core.Asset_io (ticket, _) as runnable) -> - Runner.submit runner runnable; - Some ticket - | _ -> None) - fetching.effects - |> Option.get - in - Alcotest.(check int) - "fetch has not entered authentication before queued work starts" - 0 - !acquisitions; - let deleting = - Core.step - fetching.next - (Core.Asset_requested - { scope - ; operation = "delete-before-cache" - ; action = Core.Prepare_graph_asset_deletion (Core.retirement_id_of_string "96000000-0000-4000-8000-000000000002" |> Result.get_ok) - }) - in - List.iter - (function - | Core.Run runnable -> Runner.submit runner runnable - | _ -> ()) - deleting.effects; - while !tasks <> [] do - let batch = List.rev !tasks in - tasks := []; - List.iter (fun task -> task ()) batch - done; - Alcotest.(check int) - "deleted graph's queued fetch never enters authentication or network" - 0 - !acquisitions; - Alcotest.(check bool) - "queued callback reports its own cancelled completion" - true - (List.exists - (function - | Core.Runner_completed - (Core.Asset_completion (ticket, Error Core.Asset_cancelled)) -> - ticket = expected_ticket - | _ -> false) - !posted); - List.iter (fun event -> ignore (Core.step deleting.next event)) (List.rev !posted); - Runner.shutdown runner))) -;; - -let scenarios = - scenarios - @ [ Alcotest.test_case - "asset graph deletion cancels queued fetch before cache opens" - `Quick - test_graph_asset_deletion_cancels_queued_fetch_before_cache_open - ] -;; - -(* Replay bookkeeping belongs to the runner, and Core cannot reproduce retained - runner allocations. Drive issued effects and their real completions through - the public boundary, retaining only the current reducer state. *) -let test_asset_submission_history_does_not_retain_completed_effects () = - with_support (fun support -> - Eio_main.run (fun environment -> - Eio.Switch.run (fun sw -> - let selected, scope = Core_contract.selected_graph Core_contract.graph in - let state = ref selected.next in - let completions = ref 0 in - let deps = - dependencies - ~environment - ~support - ~fork:(fun ~sw:_ _ -> fail "closed runner must not fork") - () - in - let runner = - Runner.create ~sw deps ~post:(fun event -> - incr completions; - state := (Core.step !state event).next) - |> Result.get_ok - in - Runner.shutdown runner; - let version = - Logseq_db_types.Asset_descriptor.version - ~checksum:(String.make 64 'a') - ~file_type:"bin" - |> Result.get_ok - in - let run count = - for index = 1 to count do - let requested = - Core.step - !state - (Core.Asset_requested - { scope - ; operation = "submission-memory-history" - ; action = Core.Check_asset_cache (graph_id (), version) - }) - in - state := requested.next; - List.iter - (function - | Core.Run runnable -> Runner.submit runner runnable - | _ -> ()) - requested.effects; - if index mod 256 = 0 then Gc.full_major () - done - in - run 512; - Gc.full_major (); - let before = (Gc.stat ()).live_words in - run 8192; - Gc.full_major (); - let growth = (Gc.stat ()).live_words - before in - Runner.shutdown runner; - Alcotest.(check int) "every issued request completed" 8704 !completions; - if growth > 32768 - then fail "completed submission history retained %d live words" growth))) -;; - -let scenarios = - scenarios - @ [ Alcotest.test_case - "completed asset submission history stays compact" - `Quick - test_asset_submission_history_does_not_retain_completed_effects - ] -;; - -(* Missing transport ownership is a runner fact, which a pure Core event cannot - reproduce. Exercise its public submission boundary without opening a socket. *) -let test_pull_without_registered_socket_reports_closed () = - with_support (fun support -> - Eio_main.run (fun environment -> - Eio.Switch.run (fun sw -> - let opened, connection, _ = Core_contract.pull_fixture () in - let send = - List.find_map - (function - | Core.Run (Core.Send_websocket _ as runnable) -> Some runnable - | _ -> None) - opened.effects - |> Option.get - in - let posted = ref [] in - let deps = - dependencies - ~environment - ~support - ~fork:(fun ~sw:_ _ -> fail "a send must not schedule a hidden retry") - () - in - let runner = - Runner.create ~sw deps ~post:(fun event -> posted := event :: !posted) - |> Result.get_ok - in - Runner.submit runner send; - Alcotest.(check int) - "missing socket posts one transport failure" - 1 - (List.length !posted); - Alcotest.(check bool) - "transport failure retains its connection identity" - true - (match !posted with - | [ Core.Websocket_closed (scope, Core.Recoverable_close { message = Some _; _ }) ] -> - scope = connection - | _ -> false); - Runner.shutdown runner))) -;; - -let scenarios = - scenarios - @ [ Alcotest.test_case - "F5 missing socket send posts a closed fact" - `Quick - test_pull_without_registered_socket_reports_closed - ] -;; - -(* Unsupported internal manifests are a filesystem fact owned by the runner. *) -let test_legacy_manifest_is_rejected_without_cleanup () = - with_local_asset_cache (fun support request -> - let version = - Logseq_db_types.Asset_descriptor.version - ~checksum:(String.make 64 'a') - ~file_type:"bin" - |> Result.get_ok - in - cache_bool - "first cache starts empty" - true - (request (Core.Check_asset_cache (graph_id (), version)) - = Ok (Core.Asset_cached None)); - cache_unit (request Core.Close_asset_scope); - let rec graph_directory path = - if Filename.basename path = "11111111-1111-4111-8111-111111111111" - then Some path - else if Sys.is_directory path - then - Sys.readdir path - |> Array.to_list - |> List.find_map (fun entry -> graph_directory (Filename.concat path entry)) - else None - in - let directory = - graph_directory (Filename.concat support "logseq-db-worker/assets") |> Option.get - in - let manifest = Filename.concat directory "legacy.json" in - let source = - {|{"asset":"11111111-1111-4111-8111-111111111111","type":"bin","checksum":"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa","size":"1"}|} - in - Out_channel.with_open_bin manifest (fun out -> output_string out source); - cache_bool - "unsupported manifest fails cache open" - true - (match request (Core.Check_asset_cache (graph_id (), version)) with - | Error (Core.Asset_invalid_content message) -> - message = "Asset cache manifest shape is invalid" - | _ -> false); - cache_bool "unsupported manifest is retained" true (Sys.file_exists manifest); - Alcotest.(check string) - "unsupported bytes are not rewritten" - source - (In_channel.with_open_bin manifest In_channel.input_all)) -;; - -let scenarios = - scenarios - @ [ Alcotest.test_case - "legacy manifest is rejected without cleanup" - `Quick - test_legacy_manifest_is_rejected_without_cleanup - ] -;; - -let scenarios = scenarios @ [Alcotest.test_case - "asset retirement cancel and replacement isolation" `Quick test_asset_retirement_cancel_and_replacement] - -let scenarios = scenarios @ [Alcotest.test_case - "asset retirement restore conflict is retryable" `Quick test_asset_retirement_restore_conflict] - -(* Cache publication owns verification, atomic residency, and lease lifetime; - the worker regression separately owns when an import requests publication. *) -let test_cache_publish_staged_original () = - with_local_asset_cache (fun support request -> - let source = Filename.concat support "picker.bin" in - Out_channel.with_open_bin source (fun out -> output_string out "file"); - let file, checksum, _ = cache_stage request ~file_type:"pdf" source in - let version = - Logseq_db_types.Asset_descriptor.version ~checksum ~file_type:"pdf" |> Result.get_ok - in - let staged_lease, staged_path = - request (Core.Retain_staged_file file) |> retained_value - in - cache_unit - (request (Core.Publish_staged_asset { asset = graph_id (); version; file })); - let handle = - match request (Core.Check_asset_cache (graph_id (), version)) with - | Ok (Core.Asset_cached (Some handle)) -> handle - | _ -> fail "published original missing from ordinary cache" - in - let lease, path = request (Core.Retain_asset_file handle) |> retained_value in - cache_bool - "ordinary cache keeps document extension" - true - (Filename.check_suffix path ".pdf"); - Alcotest.(check string) - "published original bytes" - "file" - (In_channel.with_open_bin path In_channel.input_all); - cache_unit (request (Core.Release_staged_file file)); - cache_bool - "publication preserves native staging lease" - true - (Sys.file_exists staged_path); - cache_unit (request (Core.Release_asset_file staged_lease)); - cache_bool "last staging lease permits cleanup" false (Sys.file_exists staged_path); - cache_bool "normal renderer survives staging cleanup" true (Sys.file_exists path); - cache_unit (request (Core.Release_asset_file lease)); - cache_unit (request (Core.Release_asset_file handle)); - cache_unit (request Core.Close_asset_scope); - let restored = - match request (Core.Check_asset_cache (graph_id (), version)) with - | Ok (Core.Asset_cached (Some handle)) -> handle - | _ -> fail "published original must survive cache reopen" - in - cache_unit (request (Core.Release_asset_file restored)); - let file, _, _ = cache_stage request source in - let wrong = - Logseq_db_types.Asset_descriptor.version - ~checksum:(String.make 64 'a') - ~file_type:"bin" - |> Result.get_ok - in - cache_bool - "wrong checksum cannot publish" - true - (request (Core.Publish_staged_asset { asset = graph_id (); version = wrong; file }) - = Error Core.Asset_checksum_mismatch); - let held, held_path = request (Core.Retain_staged_file file) |> retained_value in - cache_bool "failed publication preserves source" true (Sys.file_exists held_path); - cache_bool - "foreign filename is rejected" - true - (request - (Core.Publish_staged_asset - { asset = graph_id (); version; file = "../picker.bin" }) - = Error Core.Asset_missing_source); - cache_unit (request (Core.Release_asset_file held)); - cache_unit (request (Core.Release_staged_file file))) -;; - -let scenarios = - scenarios - @ [ Alcotest.test_case - "publish staged original into cache" - `Quick - test_cache_publish_staged_original - ] -;; - -(* Core already accepts an Error completion; cache initialization is physical IO, - so this regression exercises the public runner/reducer boundary. *) -let test_staged_preview_cache_failure_is_not_missing_file () = - with_local_asset_cache (fun support request -> - Out_channel.with_open_bin (Filename.concat support "logseq-db-worker") - (fun output -> output_string output "synthetic path obstruction"); - match request (Core.Retain_staged_file "missing.bin") with - | Error (Core.Asset_invalid_content message) -> - Alcotest.(check string) "preview failure excludes raw filesystem details" - "Attachment preview storage is unavailable" message - | Error _ -> fail "cache IO failure must not become cancellation or another failure class" - | Ok _ -> fail "cache initialization failure is not a successful missing preview") -;; -let scenarios = scenarios @ [Alcotest.test_case - "staged preview cache IO failure stays an error" `Quick test_staged_preview_cache_failure_is_not_missing_file] -;; - -(* The runner owns checksum verification after loading the staged source. A pure - completion would already contain the lost diagnostic rather than cause it. *) -let test_upload_checksum_cause_is_preserved () = - with_local_asset_cache (fun support request -> - let source = Filename.concat support "SENSITIVE-fixture.bin" in - Out_channel.with_open_bin source (fun out -> output_string out "file"); - let file, _, _ = cache_stage request source in - let version = Logseq_db_types.Asset_descriptor.version - ~checksum:(String.make 64 'a') ~file_type:"bin" |> Result.get_ok in - match request (Core.Put_asset_file - {asset=graph_id (); version; file; maximum_plaintext_bytes=4}) with - | Error (Core.Asset_invalid_content message) -> - Alcotest.(check string) "checksum stage survives" - "Asset upload checksum mismatch" message - | _ -> fail "checksum failure must retain invalid-content category") -;; -let scenarios = scenarios @ [Alcotest.test_case - "upload checksum cause is preserved" `Quick test_upload_checksum_cause_is_preserved] -;; - -let test_catalog_io_causes_exclude_paths () = - let execute ~prepare ~instruction ~expected = - with_support (fun support -> - prepare support; - Eio_main.run (fun environment -> Eio.Switch.run (fun sw -> - let tasks = ref [] and posted = ref [] in - let deps = dependencies ~environment ~support - ~fork:(fun ~sw:_ task -> tasks := task :: !tasks) () in - let runner = Runner.create ~sw deps - ~post:(fun event -> posted := event :: !posted) |> Result.get_ok in - Runner.submit runner (instruction ()); - List.iter (fun task -> task ()) (List.rev !tasks); - match !posted with - | [Core.Runner_completed (Core.Completion (_, Error (Core.Effect_failed message)))] -> - let message = match Logseq_db_types.Failure_diagnostic.decode message with - | Some detail -> Logseq_db_types.Failure_diagnostic.message detail - | None -> fail "catalog cause must be a safe forwarding carrier" in - Alcotest.(check bool) "safe operation and reason" true - (String.starts_with ~prefix:expected message); - Alcotest.(check bool) "no synthetic private path" false - (String.contains message '/'); - Runner.shutdown runner - | _ -> fail "expected one catalog failure"))) in - execute - ~prepare:(fun support -> - let worker = Filename.concat support "logseq-db-worker" in Unix.mkdir worker 0o700; - let catalogs = Filename.concat worker "sync-catalogs" in Unix.mkdir catalogs 0o700; - Unix.mkdir (Filename.concat catalogs - "748ac0b0c274b30bc6fc1da756958deab4ebdef06a2d6373a377e2b4d8cec6df.json") 0o700) - ~instruction:load_catalog_effect ~expected:"loadCatalog: Resource could not be opened"; - execute - ~prepare:(fun support -> Out_channel.with_open_bin - (Filename.concat support "logseq-db-worker") (fun out -> output_string out "private")) - ~instruction:(fun () -> - let authenticated = Core.step (core ()) (Core.Account_authenticated {user_id=Some "user-1"}) in - let catalog = List.find_map (function - | Core.Run (Core.Request (ticket, Core.Fetch_catalog _)) -> - Some (Core.step authenticated.next - (Core.Runner_completed (Core.Completion (ticket, Ok [encrypted_graph ()])))) - | _ -> None) authenticated.effects |> Option.get in - let selected = Core.step catalog.next (Core.Graph_selected (graph_id ())) in - List.find_map (function - | Core.Run (Core.Request (_, Core.Save_catalog _) as action) -> Some action - | _ -> None) selected.effects |> Option.get) - ~expected:"prepareCatalog: Storage path is invalid" -;; -let scenarios = scenarios @ [Alcotest.test_case - "catalog IO causes exclude paths" `Quick test_catalog_io_causes_exclude_paths] -;; - -let test_platform_crypto_unavailable_has_safe_cause () = - with_support (fun support -> Eio_main.run (fun environment -> Eio.Switch.run (fun sw -> - let tasks = ref [] and posted = ref [] in - let deps = dependencies ~secrets_dependency:(Runner.apple_secrets () |> Result.get_ok) - ~environment ~support ~fork:(fun ~sw:_ task -> tasks := task :: !tasks) () in - let runner = Runner.create ~sw deps ~post:(fun event -> posted := event :: !posted) - |> Result.get_ok in - let _, instruction = cached_key_effect () in - Runner.submit runner instruction; - List.iter (fun task -> task ()) (List.rev !tasks); - (match !posted with - | [Core.Runner_completed (Core.Completion (_, Error (Core.Effect_failed message)))] -> - Alcotest.(check bool) "provider finite cause survives public runner" true - (String.starts_with ~prefix:"platformCrypto: Resource is unavailable" message) - | _ -> fail "expected unavailable host crypto failure"); - Runner.shutdown runner))) -;; -let scenarios = scenarios @ [Alcotest.test_case - "platform crypto unavailable has safe cause" `Quick test_platform_crypto_unavailable_has_safe_cause] -;; - -let test_asset_authentication_cause_is_scoped () = - let reports = ref [] in - let on_asset_diagnostic ~scope ~operation ~request_operation detail = - reports := (scope, operation, request_operation, detail) :: !reports in - with_local_asset_cache ~with_causes:true ~on_asset_diagnostic - ~acquire:(fun _ -> Error "SENSITIVE-fixture-token-error") - (fun _ request -> - let version = Logseq_db_types.Asset_descriptor.version - ~checksum:(String.make 64 'a') ~file_type:"bin" |> Result.get_ok in - let result = request (Core.Fetch_asset {asset=graph_id (); version; maximum_plaintext_bytes=4}) in - Alcotest.(check bool) "original authentication category" true - (Result.map_error Core.asset_failure_kind result = Error Core.Asset_authentication); - (match result with - | Error failure -> - let cause = Core.asset_failure_cause failure in - Alcotest.(check (option string)) "result carries cause without side-channel dependency" - (Some "authenticationFailed") - (Option.map Logseq_db_types.Failure_diagnostic.reason cause) - | Ok _ -> fail "expected authentication failure"); - match !reports with - | [(scope, operation, request_operation, detail)] -> - let module D = Logseq_db_types.Failure_diagnostic in - Alcotest.(check bool) "exact graph identity" true (scope.graph_id = graph_id ()); - Alcotest.(check string) "finite operation" "assetDownload" operation; - Alcotest.(check string) "internal request identity" "cache-1" request_operation; - Alcotest.(check string) "source category" "authenticationFailed" (D.reason detail); - Alcotest.(check bool) "no raw auth text" false (String.contains (D.message detail) 'S') - | _ -> fail "expected one scoped authentication cause") -;; -let scenarios = scenarios @ [Alcotest.test_case - "asset authentication cause is scoped" `Quick test_asset_authentication_cause_is_scoped] -;; - -(* Physical socket admission is owned by Runner, not the pure instruction producer. *) -let test_resident_websocket_owners_are_independent () = - with_support(fun support->Eio_main.run(fun environment->Eio.Switch.run(fun sw-> - let ids=[Core_contract.graph_id;Core_contract.other_graph_id] in - let configured=Core.step (Core_contract.resident_catalog ()) (Core.Resident_graphs_changed ids) in - let cache=Core_contract.save_catalog_cache configured.effects in - let restoring=Core.step (Core_contract.cold_start ()) (Core.Restore_local_account {user_id="user"}) in - let restored=Core.step restoring.next (Core_contract.load_catalog_completion restoring.effects (Some cache)) in - let attached=List.fold_left(fun current id-> - let scope=Core.scope_for_graph current ~graph_id:id |> Option.get in - let request=List.find_map(function Core.Delegate(Core.Inspect_mirror request) when request.scope=scope -> Some request|_->None) restored.effects |> Option.get in - let attaching=Core.step current (Core.Mirror_inspected(Core.Mirror_available request)) in - let token=Logseq_overlay_db.Types.sync_token_of_string "sync-token:v1:resident-runner" |> Result.get_ok in - let sync=Logseq_overlay_db.Types.sync_view ~token ~checkpoint:(Core_contract.server_cursor_of_int 0) ~submissions:[] in - (Core.step attaching.next (Core.Graph_attached {scope;sync})).next) restored.next ids in - let shown=Core.step attached (Core.Timeline_presented(Core_contract.timeline_ticket attached)) in - let online=Core.step shown.next (Core.Account_authenticated {user_id=Some "user"}) in - let current=ref online.next in - let starts=List.filter_map(function Core.Run(Core.Start_websocket _ as instruction)->Some instruction|_->None) online.effects in - Alcotest.(check int) "reducer requests both physical connections" 2 (List.length starts); - let queued=Queue.create () and posted=ref [] and acquired=ref 0 in - let token=Runner.id_token_provider ~acquire:(fun _->incr acquired;Error "Synthetic blocked network") ~invalidate:(fun _ ~token:_->()) in - let deps=dependencies ~environment ~support ~fork:(fun ~sw:_ callback->Queue.add callback queued) ~id_token_dependency:token () in - let runner=Runner.create ~sw deps ~post:(fun event->posted:=event::!posted) |> Result.get_ok in - List.iter(Runner.submit runner) starts; - while not(Queue.is_empty queued) do (Queue.take queued) () done; - Alcotest.(check int) "both graph owners reach independent token/connect path" 2 !acquired; - Alcotest.(check bool) "no graph blocks another physical owner" false - (List.exists(function Core.Websocket_closed(_,Core.Protocol_failure _)->true|_->false) !posted); - List.iter(fun instruction->match instruction with - | Core.Start_websocket request -> - let transition=Core.step !current (Core.Websocket_closed(request.scope,Core.Authentication_failure "Synthetic blocked network")) in - current:=transition.next;List.iter(function Core.Run runnable->Runner.submit runner runnable|_->()) transition.effects - | _->()) starts; - while not(Queue.is_empty queued) do (Queue.take queued) () done; - Alcotest.(check int) "both graph owners retire separately" 2 - (List.length(List.filter(function Core.Websocket_retired _->true|_->false) !posted)); - Runner.shutdown runner))) -;; -let scenarios=scenarios @ [Alcotest.test_case "resident physical sockets admit and retire independently" `Quick test_resident_websocket_owners_are_independent] -;; - -(* Provider exceptions occur in the effect runner, not the pure reducer. Reproduce - through its public dependencies and completion output, never a fabricated result. *) -let test_asset_provider_exception_keeps_safe_cause () = - let module D = Logseq_db_types.Failure_diagnostic in - List.iter (fun error -> - let reports = ref [] in - let expected = D.of_source ~operation:"assetDownload" ~reason:"invalidContent" - (Printexc.to_string error) in - with_local_asset_cache ~with_causes:true - ~on_asset_diagnostic:(fun ~scope:_ ~operation:_ ~request_operation:_ detail -> - reports := detail :: !reports) - ~acquire:(fun _ -> raise error) - (fun _ request -> - let version = Logseq_db_types.Asset_descriptor.version - ~checksum:(String.make 64 'a') ~file_type:"bin" |> Result.get_ok in - (match request (Core.Fetch_asset {asset=graph_id (); version; maximum_plaintext_bytes=4}) with - | Error failure -> - let source = match Core.asset_failure_kind failure with - | Core.Asset_invalid_content source -> source - | _ -> fail "expected invalid content exception category" in - Alcotest.(check (option string)) "exception reaches caller as safe bounded cause" - (Some (D.message expected)) (Option.map D.message (D.decode source)) - | _ -> fail "expected classified provider exception"); - Alcotest.(check (list string)) "exception also reaches scoped diagnostics" - [D.message expected] (List.map D.message !reports))) - [ Unix.Unix_error (Unix.EACCES, "PRIVATE_OPERATION", "PRIVATE_PATH") - ; Unix.Unix_error (Unix.ENOSPC, "PRIVATE_OPERATION", "PRIVATE_PATH") - ; Failure "Bearer PRIVATE_TOKEN https://user:PRIVATE_PASSWORD@host/PRIVATE_GRAPH" ] -;; -let scenarios = scenarios @ [Alcotest.test_case - "asset provider exception keeps safe cause" `Quick test_asset_provider_exception_keeps_safe_cause] -;; - -let test_catalog_provider_exception_completes () = - with_support (fun support -> Eio_main.run (fun environment -> Eio.Switch.run (fun sw -> - let tasks = Queue.create () and posted = ref [] in - let deps = dependencies ~environment ~support - ~fork:(fun ~sw:_ task -> Queue.add task tasks) - ~id_token_dependency:(Runner.id_token_provider - ~acquire:(fun _ -> raise (Unix.Unix_error (Unix.EACCES, "PRIVATE_OPERATION", "PRIVATE_PATH"))) - ~invalidate:(fun _ ~token:_ -> ())) () in - let runner = Runner.create ~sw deps ~post:(fun event -> posted := event :: !posted) - |> Result.get_ok in - let requested = Core.step (core ()) (Core.Account_authenticated {user_id=Some "user-1"}) in - let runnable = List.find_map (function - | Core.Run (Core.Request (_, Core.Fetch_catalog _) as runnable) -> Some runnable - | _ -> None) requested.effects |> Option.get in - Runner.submit runner runnable; - while not (Queue.is_empty tasks) do (Queue.take tasks) () done; - (match !posted with - | [Core.Runner_completed (Core.Completion (_, Error (Core.Effect_failed source)))] -> - let module D = Logseq_db_types.Failure_diagnostic in - let diagnostic = D.decode source |> Option.get in - Alcotest.(check string) "provider exception retains errno" "io-permission" (D.source_class diagnostic); - let failed = Core.step requested.next (List.hd !posted) in - Alcotest.(check bool) "pending caller completes with explicit failure" true - ((Core.state failed.next).snapshot.sync_phase = Core.Failed) - | _ -> fail "provider exception must complete the caller"); - Runner.shutdown runner))) -;; -let scenarios = scenarios @ [Alcotest.test_case - "catalog provider exception completes" `Quick test_catalog_provider_exception_completes] -;; - -let scenarios = scenarios @ [Alcotest.test_case - "protected provider exception keeps safe cause" `Quick - (test_cached_wrapped_key_is_unlocked_before_protected_value_decryption ~decryption_exception:true)] -;; - -let scenarios = scenarios @ [Alcotest.test_case - "protected encryption failure keeps safe cause" `Quick - (test_cached_wrapped_key_is_unlocked_before_protected_value_decryption ~encryption_failure:true)] -;; diff --git a/logseq_sync/test/service_lifecycle_contract.ml b/logseq_sync/test/service_lifecycle_contract.ml new file mode 100644 index 00000000..e0014c42 --- /dev/null +++ b/logseq_sync/test/service_lifecycle_contract.ml @@ -0,0 +1,85 @@ +module State = Logseq_sync_service_lifecycle + +let accepted = function + | Ok state -> state + | Error _ -> Alcotest.fail "expected accepted close" +;; + +let test_reentrant_preserves_right () = + let state = State.initial in + Alcotest.(check bool) + "reentrant rejected" + true + (State.request_close ~inside_callback:true state = Error State.Reentrant); + let closing = State.request_close ~inside_callback:false state |> accepted in + Alcotest.(check bool) "right preserved" true (State.lifecycle closing = State.Closing) +;; + +let test_close_once () = + let closing = State.request_close ~inside_callback:false State.initial |> accepted in + Alcotest.(check bool) + "repeat during close fails" + true + (State.request_close ~inside_callback:false closing = Error State.Already_called); + let closed = State.complete closing in + Alcotest.(check bool) + "repeat after retirement fails" + true + (State.request_close ~inside_callback:false closed = Error State.Already_called) +;; + +let test_failed_cleanup_does_not_restore_right () = + let closing = State.request_close ~inside_callback:false State.initial |> accepted in + let failed = State.begin_cleanup closing in + Alcotest.(check bool) + "failure stays closing" + true + (State.lifecycle failed = State.Closing); + Alcotest.(check bool) + "no implicit repair retry" + true + (State.request_close ~inside_callback:false failed = Error State.Already_called) +;; + +let test_owner_cleanup_preserves_close_right () = + let closing = State.begin_cleanup State.initial in + let accepted = State.request_close ~inside_callback:false closing |> accepted in + Alcotest.(check bool) + "owner cleanup does not consume caller right" + true + (State.lifecycle accepted = State.Closing) +;; + +let test_reentrant_precedence () = + let closing = State.request_close ~inside_callback:false State.initial |> accepted in + Alcotest.(check bool) + "callback check precedes close right" + true + (State.request_close ~inside_callback:true closing = Error State.Reentrant) +;; + +let test_closed_late_close () = + let retired = State.complete (State.begin_cleanup State.initial) in + let closed = State.request_close ~inside_callback:false retired |> accepted in + Alcotest.(check bool) + "retirement stays terminal" + true + (State.lifecycle closed = State.Closed) +;; + +let () = + Alcotest.run + "service lifecycle" + [ ( "close ownership" + , List.map + (fun (name, test) -> Alcotest.test_case name `Quick test) + [ "reentrant preserves right", test_reentrant_preserves_right + ; "strict once only", test_close_once + ; ( "failed cleanup retains right consumption" + , test_failed_cleanup_does_not_restore_right ) + ; "owner cleanup", test_owner_cleanup_preserves_close_right + ; "reentrant precedence", test_reentrant_precedence + ; "late close", test_closed_late_close + ] ) + ] +;; diff --git a/logseq_sync/test/sync_dsl_contract.ml b/logseq_sync/test/sync_dsl_contract.ml new file mode 100644 index 00000000..53bfbf8a --- /dev/null +++ b/logseq_sync/test/sync_dsl_contract.ml @@ -0,0 +1,262 @@ +module D = Sync_dsl.Core + +let get = function + | Ok value -> value + | Error error -> failwith (D.error_message error) +;; + +let identity () = + get + (D.identity + ~origin:"https://example.com" + ~account:"a" + ~graph:"g" + ~account_generation:0L + ~graph_generation:0L) +;; + +let check condition message = Alcotest.(check bool) message true condition + +let ownership () = + let owner = D.Producer.create (identity ()) in + let foreign = D.Producer.create (identity ()) in + let scope = D.Producer.scope owner in + check + (Result.is_error (D.Producer.check_scope foreign scope)) + "equal identity is not ownership"; + let calls = ref 0 in + let child = + get + (D.Producer.child owner scope (D.task_return 7) ~poll:(fun () -> + incr calls; + Some (Ok 7))) + in + check + (Result.is_error (D.Producer.child_result foreign scope child)) + "foreign child rejected"; + check (!calls = 0) "foreign callback not called"; + check (D.Producer.child_result owner scope child = Ok (Some (Ok 7))) "typed result"; + ignore (get (D.Producer.forget_child owner scope child)); + ignore (get (D.Producer.forget_child owner scope child)); + check + (Result.is_error (D.Producer.child_result owner scope child)) + "consumed child rejected"; + let changing = ref 0 in + let cached = + get + (D.Producer.child owner scope (D.task_return 0) ~poll:(fun () -> + incr changing; + Some (Ok !changing))) + in + check (D.Producer.child_result owner scope cached = Ok (Some (Ok 1))) "first completion"; + check + (D.Producer.child_result owner scope cached = Ok (Some (Ok 1))) + "completion is repeatable"; + check (!changing = 1) "completed poll pruned"; + let receipt = get (D.Producer.demand_receipt owner scope) in + let id = get (D.demand_id "asset") in + let output = get (D.Producer.demand_output owner scope receipt ~id D.Demand_queued) in + ignore (get (D.Producer.close_scope owner scope)); + check + (Result.is_error (D.Producer.check_demand_receipt owner scope receipt)) + "closed receipt rejected"; + check + (Result.is_ok (D.Producer.check_delivery owner output.delivery)) + "cleanup survives close" +;; + +let stack_safe () = + let program = ref (D.return 0) in + for _ = 1 to 200000 do + program := D.bind !program (fun value -> D.return (value + 1)) + done; + match D.view !program with + | D.Return value -> Alcotest.(check int) "deep binds" 200000 value + | _ -> Alcotest.fail "unexpected effect" +;; + +let validation () = + check + (Result.is_error + (D.identity + ~origin:"http://example.com" + ~account:"a" + ~graph:"g" + ~account_generation:0L + ~graph_generation:0L)) + "HTTPS required"; + check + (Result.is_error + (D.retry_policy + ~initial_seconds:nan + ~maximum_seconds:2. + ~multiplier:2. + ~maximum_attempts:2)) + "NaN rejected"; + check (Result.is_error (D.demand_id (String.make 257 'x'))) "bounded ID" +;; + +let make_limits + ?(maximum_items = 8) + ?(maximum_tasks = 8) + ?(maximum_pending = 8) + ?(foreground_demand_reserve = 1) + ?(maximum_messages = 8) + () + = + D.limits + ~maximum_items + ~maximum_bytes:1024 + ~maximum_pending + ~maximum_resources:8 + ~steps_per_turn:32 + ~maximum_tasks + ~maximum_messages + ~maximum_message_bytes:1024 + ~maximum_demands:8 + ~maximum_active_assets:2 + ~foreground_reserve:1 + ~foreground_demand_reserve + ~foreground_descriptor_reserve_bytes:128 + ~asset_cache_budget_bytes:0L + ~asset_maximum_file_bytes:0 + ~asset_pending_budget_bytes:0L + ~snapshot_maximum_artifact_bytes:1073741824L + ~snapshot_staging_budget_bytes:1L + ~snapshot_scratch_budget_bytes:1L +;; + +let limits_validation () = + check (Result.is_ok (make_limits ())) "independent snapshot budgets and disabled assets"; + List.iter + (fun result -> check (Result.is_error result) "invalid bounds rejected") + [ make_limits ~maximum_items:4097 () + ; make_limits ~maximum_tasks:5 () + ; make_limits ~maximum_pending:5 () + ; make_limits ~foreground_demand_reserve:9 () + ; make_limits ~maximum_messages:3 () + ] +;; + +let identity_validation () = + let create origin = + D.identity ~origin ~account:"a" ~graph:"g" ~account_generation:0L ~graph_generation:0L + in + let normalized = get (create "HTTPS://EXAMPLE.COM:443/") in + Alcotest.(check string) "canonical origin" "https://example.com" normalized.origin; + List.iter + (fun origin -> check (Result.is_error (create origin)) "not an origin") + [ "https://example.com/path" + ; "https://user@example.com" + ; "https://example.com?query=1" + ; "https://example.com#fragment" + ; "https://exam ple.com" + ] +;; + +let scoped_handles () = + let owner = D.Producer.create (identity ()) in + let scope = D.Producer.scope owner in + let other = D.Producer.scope owner in + let unlock = get (D.Producer.unlock_receipt owner scope) in + let outbox = get (D.Producer.outbox_check owner scope) in + let channel = get (D.Producer.notifications owner scope) in + let revision = get (D.Producer.capacity_revision owner scope ~sequence:12L) in + check + (D.Producer.capacity_sequence owner scope revision = Ok 12L) + "old sequence remains valid"; + check + (Result.is_error (D.Producer.capacity_sequence owner other revision)) + "foreign scope sequence"; + check + (Result.is_error (D.Producer.check_unlock_receipt owner other unlock)) + "foreign unlock"; + check + (Result.is_error (D.Producer.check_outbox_check owner other outbox)) + "foreign outbox"; + check + (Result.is_error (D.Producer.check_notifications owner other channel)) + "foreign notifications"; + ignore (get (D.Producer.close_scope owner scope)); + ignore (get (D.Producer.close_scope owner scope)); + check + (Result.is_error (D.Producer.check_unlock_receipt owner scope unlock)) + "closed unlock" +;; + +let task_projection () = + let calls = ref 0 in + let task = + D.with_scope (fun _ -> + incr calls; + D.task_return 1) + in + (match D.task_view task with + | D.Task_scope _ -> () + | _ -> Alcotest.fail "scope projection"); + check (!calls = 0) "scope body not run by projection"; + let task = + D.task_bind (D.task_return 1) (fun value -> + incr calls; + D.task_return (value + 1)) + in + (match D.task_view task with + | D.Task_bind _ -> () + | _ -> Alcotest.fail "bind projection"); + check (!calls = 0) "task continuation not run by projection" +;; + +let authoritative_errors () = + let module T = Logseq_overlay_db.Types in + let module Diagnostic = Logseq_db_types.Authoritative_diagnostic in + let render error = D.error_message (D.Authoritative_failure error) in + Alcotest.(check string) + "checksum classification" + "Authoritative transition failed: checksum mismatch" + (render T.Authoritative_checksum_mismatch); + Alcotest.(check string) + "origin classification" + "Authoritative transition failed: origin unresolved" + (render T.Authoritative_origin_unresolved); + let diagnostic = Diagnostic.create ~stage:Decode ~reason:Reference_shape () in + Alcotest.(check string) + "typed decode context survives" + (Diagnostic.message diagnostic) + (render (T.Authoritative_decode_failed (Diagnostic.encode diagnostic))); + let secret = String.make 4096 '\001' ^ " secret-payload-token" in + List.iter + (fun error -> + let message = render error in + check (String.length message < 512) "bounded message"; + check (not (String.contains message '\001')) "raw source excluded") + [ T.Authoritative_decode_failed secret + ; T.Authoritative_integrity_failure secret + ; T.Authoritative_commit_persistence_failed secret + ; T.Authoritative_commit_fatal_state secret + ]; + Alcotest.(check string) + "legacy decode source fails closed" + "Authoritative transition failed: transaction decode rejected" + (render (T.Authoritative_decode_failed secret)) +;; + +let () = + Alcotest.run + "sync DSL" + [ "policy", Sync_dsl_policy_contract.tests + ; ( "ownership" + , [ Alcotest.test_case "original owners" `Quick ownership + ; Alcotest.test_case "scoped handles" `Quick scoped_handles + ] ) + ; ( "composition" + , [ Alcotest.test_case "stack safe" `Quick stack_safe + ; Alcotest.test_case "task projection" `Quick task_projection + ] ) + ; ( "validation" + , [ Alcotest.test_case "authoritative errors" `Quick authoritative_errors + ; Alcotest.test_case "invalid input" `Quick validation + ; Alcotest.test_case "limits" `Quick limits_validation + ; Alcotest.test_case "identity" `Quick identity_validation + ] ) + ] +;; diff --git a/logseq_sync/test/sync_dsl_interpreter.ml b/logseq_sync/test/sync_dsl_interpreter.ml new file mode 100644 index 00000000..4a0fb0fc --- /dev/null +++ b/logseq_sync/test/sync_dsl_interpreter.ml @@ -0,0 +1,217 @@ +module D = Sync_dsl.Core + +type responder = + { respond : 'a. D.scope -> 'a D.task_operation -> ('a, D.error) result option } + +type region = + { scope : D.scope + ; mutable cancel : ((unit -> bool) * (unit -> unit)) list + } + +type engine = + { owner : D.Producer.owner + ; responder : responder + ; input : unit -> D.service_input option + ; mutable fibers : (unit -> bool) list + ; mutable cancelling : bool + ; admit : unit -> (unit, D.error) result + } + +type 'a fiber = + { mutable result : ('a, D.error) result option + ; mutable step : unit -> bool + } + +let rec fiber : type value. engine -> region -> value D.task -> value fiber = + fun engine region task -> + let fiber = { result = None; step = (fun () -> false) } in + let finish result = + fiber.result <- Some result; + fiber.step <- (fun () -> false) + in + let rec run + : type value. region -> value D.task -> (value -> unit) -> (D.error -> unit) -> unit + = + fun region task success failure -> + fiber.step + <- (fun () -> + match D.task_view task with + | D.Task_return value -> + success value; + true + | D.Task_fail error -> + failure error; + true + | D.Task_bind (first, continuation) -> + run + region + first + (fun value -> run region (continuation value) success failure) + failure; + true + | D.Task_perform (operation, accepted, rejected) -> + (match respond engine region operation with + | None -> false + | Some (Ok value) -> + run region (accepted value) success failure; + true + | Some (Error error) -> + run region (rejected error) success failure; + true) + | D.Task_scope body -> + let nested = { scope = D.Producer.scope engine.owner; cancel = [] } in + let close () = + List.iter (fun (_, cancel) -> cancel ()) nested.cancel; + ignore (D.Producer.close_scope engine.owner nested.scope) + in + run + nested + (body nested.scope) + (fun value -> + close (); + success value) + (fun error -> + close (); + failure error); + true + | D.Task_spawn (scope, body) -> + success (spawn engine region scope body); + true + | D.Task_spawn_asset (scope, receipt, body) -> + (match D.Producer.check_demand_receipt engine.owner scope receipt with + | Error error -> success (Error error) + | Ok () -> success (spawn engine region scope body)); + true + | D.Task_join child -> + (match D.Producer.child_result engine.owner region.scope child with + | Error error -> + failure error; + true + | Ok None -> false + | Ok (Some result) -> + ignore (D.Producer.forget_child engine.owner region.scope child); + success result; + true) + | D.Task_cancel child -> + engine.cancelling <- true; + let result = D.Producer.child_result engine.owner region.scope child in + engine.cancelling <- false; + (match result with + | Error error -> failure error + | Ok _ -> success ()); + true) + in + run region task (fun value -> finish (Ok value)) (fun error -> finish (Error error)); + engine.fibers + <- engine.fibers + @ [ (fun () -> + ignore (fiber.step ()); + fiber.result = None) + ]; + fiber + +and spawn + : type value. + engine -> region -> D.scope -> value D.task -> (value D.child, D.error) result + = + fun engine region scope task -> + match D.Producer.check_scope engine.owner scope with + | Error error -> Error error + | Ok () when scope != region.scope -> Error D.Stale_context + | Ok () -> + (match engine.admit () with + | Error error -> Error error + | Ok () -> + let running = fiber engine region task in + let cancel () = + if running.result = None + then ( + running.result <- Some (Error D.Cancelled); + running.step <- (fun () -> false)) + in + region.cancel + <- ((fun () -> running.result = None), cancel) + :: List.filter (fun (active, _) -> active ()) region.cancel; + D.Producer.child engine.owner scope task ~poll:(fun () -> + if engine.cancelling then cancel (); + running.result)) + +and respond + : type value. + engine -> region -> value D.task_operation -> (value, D.error) result option + = + fun engine region operation -> + match operation with + | D.Await_input_or_child children -> + (match engine.input () with + | Some input -> Some (Ok (D.Input input)) + | None -> + let rec ready index = + if index = Rrbvec.length children + then None + else ( + let child = Rrbvec.nth children index in + match D.Producer.child_result engine.owner region.scope child with + | Error error -> Some (Error error) + | Ok None -> ready (index + 1) + | Ok (Some _) -> Some (Ok (D.Child_ready child))) + in + ready 0) + | D.Next_input -> Option.map (fun input -> Ok input) (engine.input ()) + | operation -> engine.responder.respond region.scope operation +;; + +let run_task ?(admit = fun () -> Ok ()) ~identity ~responder ~input task = + let owner = D.Producer.create identity in + let engine = + { owner; responder = responder owner; input; fibers = []; cancelling = false; admit } + in + let root = { scope = D.Producer.scope owner; cancel = [] } in + let running = fiber engine root task in + let rec drive remaining = + match running.result with + | Some result -> result + | None when remaining = 0 -> + Alcotest.fail "deterministic task interpreter exhausted its step budget" + | None -> + let scheduled = engine.fibers in + engine.fibers <- []; + let pending = List.filter (fun step -> step ()) scheduled in + engine.fibers <- pending @ engine.fibers; + drive (remaining - 1) + in + drive 200000 +;; + +type task_runner = { run_task : 'a. 'a D.task -> ('a, D.error) result } + +let run_program + ~(restore : unit -> (D.restore_report, D.error) result) + ~(install : D.network_gate -> (unit, D.error) result) + ~runner + program + = + let dispatch + : type before after failed value. + (before, after, failed, value) D.operation -> (value, D.error) result + = function + | D.Restore -> restore () + | D.Install_initial_network network -> install network + | D.Run_task task -> runner.run_task task + | _ -> Alcotest.fail "unexpected policy lifecycle operation" + in + let rec run + : type before after value. (before, after, value) D.program -> (value, D.error) result + = + fun program -> + match D.view program with + | D.Return value -> Ok value + | D.Abort error -> Error error + | D.Perform (operation, success, failure) -> + let result = dispatch operation in + (match result with + | Ok value -> run (success value) + | Error error -> run (failure error)) + in + run program +;; diff --git a/logseq_sync/test/sync_dsl_policy_contract.ml b/logseq_sync/test/sync_dsl_policy_contract.ml new file mode 100644 index 00000000..7a291702 --- /dev/null +++ b/logseq_sync/test/sync_dsl_policy_contract.ml @@ -0,0 +1,1212 @@ +module D = Sync_dsl.Core +module T = Logseq_overlay_db.Types +module I = Sync_dsl_interpreter + +let ok = Result.get_ok + +let identity = + ok + (D.identity + ~origin:"https://example.com" + ~account:"account" + ~graph:"graph" + ~account_generation:0L + ~graph_generation:0L) +;; + +let cursor value = ok (T.Server_cursor.of_int64 value) +let generation = ok (T.Generation.of_string "generation:v1:dsl") +let projection = ok (T.Projection_revision.of_string "projection:v1:dsl") +let batch_id = ok (T.Submission_batch_id.of_string "submission-batch:v1:original") + +let mutation_id = + ok (Logseq_db_types.Graph_types.Uuid.of_string "11111111-1111-4111-8111-111111111111") +;; + +let fingerprint = ok (T.Mutation_fingerprint.of_string "mutation-fingerprint:v1:dsl") +let checksum = ok (T.Checksum.of_string "checksum:v1:dsl") + +let retry = + ok + (D.retry_policy + ~initial_seconds:0.25 + ~maximum_seconds:2. + ~multiplier:2. + ~maximum_attempts:2) +;; + +let config = + ok (D.policy_config ~sync_retry:retry ~asset_retry:retry ~no_progress_seconds:0.125) +;; + +let open_gate epoch = D.Open { authorization_epoch = epoch } + +let descriptor state : T.submission_descriptor = + { mutation_id + ; fingerprint + ; state + ; operation = T.Save_block_operation + ; dependency_eligible = true + ; attempt_count = 0 + ; plaintext_bytes = 8 + ; protected_bytes = Some 8 + } +;; + +let wire = + ok + (T.submission_wire + ~maximum_bytes:1024 + ~mutation_id + ~operation:T.Save_block_operation + ~protected_transaction:"original durable ciphertext") +;; + +let batch = + ok + (T.submission_batch + ~maximum_wires:1 + ~maximum_bytes:1024 + ~batch_id + ~t_before:(cursor 0L) + ~wires:[ wire ]) +;; + +type fixture = + { mutable trace : string list + ; mutable checkpoint : T.server_cursor option + ; mutable remote : T.server_cursor + ; mutable submissions : T.submission_descriptor list + ; mutable revision : int64 + ; mutable network : D.network_gate + ; mutable keys : D.key_status + ; mutable status : D.service_status + ; mutable statuses : D.service_status list + ; mutable connect_failures : D.error list + ; mutable head_failure : D.error option + ; mutable pull_checksum : T.checksum option + ; mutable expected_checksum : T.checksum option + ; mutable snapshot_through : T.server_cursor option + ; mutable unknown_pushes : int + ; mutable defer : bool + ; mutable pause_sleep : bool + ; mutable outbox_dirty : bool + ; mutable install_count : int + ; mutable pulls_without_progress : int + ; mutable snapshot_activation : D.activation option + ; mutable activation_failure : D.error option + ; mutable inspect_failure : D.error option + ; mutable scope : D.scope option + ; mutable owner : D.Producer.owner option + ; mutable asset_reports : D.demand_transfer list + ; mutable availability : D.demand_availability list + ; mutable publication_failure : D.error option + } + +let fixture ?(submissions = []) () = + { trace = [] + ; checkpoint = Some (cursor 0L) + ; remote = cursor 0L + ; submissions + ; revision = 0L + ; network = D.Closed + ; keys = D.Unencrypted + ; status = D.Service_idle + ; statuses = [] + ; connect_failures = [] + ; head_failure = None + ; pull_checksum = None + ; expected_checksum = None + ; snapshot_through = None + ; unknown_pushes = 0 + ; defer = false + ; pause_sleep = false + ; outbox_dirty = false + ; install_count = 0 + ; pulls_without_progress = 0 + ; snapshot_activation = None + ; activation_failure = None + ; inspect_failure = None + ; scope = None + ; owner = None + ; asset_reports = [] + ; availability = [] + ; publication_failure = None + } +;; + +let record fixture event = fixture.trace <- fixture.trace @ [ event ] +let count fixture event = List.length (List.filter (( = ) event) fixture.trace) + +let token fixture = + ok (T.sync_token_of_string ("sync-token:v1:" ^ Int64.to_string fixture.revision)) +;; + +let view fixture = + T.sync_view + ~token:(token fixture) + ~checkpoint:(Option.value fixture.checkpoint ~default:(cursor 0L)) + ~submissions:fixture.submissions +;; + +let observation fixture : D.observation = + { identity + ; keys = fixture.keys + ; network = fixture.network + ; checkpoint = fixture.checkpoint + ; pending_submissions = List.length fixture.submissions + ; pending_assets = 0 + ; cleanup_pending = 0 + ; last_failure = None + } +;; + +let commit fixture transition submission_batch : T.outbox_commit = + { generation + ; before_projection_revision = projection + ; after_projection_revision = projection + ; sync_token = token fixture + ; transition + ; activity = T.Logically_active + ; logical_change_summary = T.No_logical_change + ; submission_batch + } +;; + +let database : type value. fixture -> value D.database -> (value, D.error) result = + fun fixture operation -> + match operation with + | D.Inspect -> + record fixture "inspect"; + (match fixture.inspect_failure with + | Some error when List.mem "outbox-check" fixture.trace -> Error error + | _ -> Ok (view fixture)) + | D.Prepare_submission (expected, transition) -> + if not (T.sync_token_equal expected (token fixture)) + then Error (D.Outbox_failure T.Outbox_sync_token_conflict) + else ( + fixture.revision <- Int64.succ fixture.revision; + let submission_batch = + match transition with + | T.Submit_group ids -> + Alcotest.(check bool) "fresh exact IDs" true (ids = [ mutation_id ]); + record fixture "persist-submit"; + fixture.submissions <- [ descriptor (T.Submitted batch_id) ]; + Some batch + | T.Retry_group id -> + Alcotest.(check bool) + "retry original batch ID" + true + (T.Submission_batch_id.equal id batch_id); + record fixture "persist-retry"; + Some batch + | T.Accept_group _ -> + record fixture "persist-accept"; + fixture.submissions + <- [ descriptor (T.Accepted_pending_authoritative batch_id) ]; + None + | T.Reject_group _ -> + record fixture "persist-reject"; + None + in + Ok (commit fixture transition submission_batch)) + | D.Apply_authoritative (_, authoritative) -> + let submitted = + List.exists + (fun (item : T.submission_descriptor) -> + match item.state with + | T.Submitted _ -> true + | _ -> false) + fixture.submissions + in + if fixture.defer && submitted + then ( + record fixture "deferred"; + Ok (D.Deferred (T.Await_submission_outcome batch_id))) + else ( + record fixture "apply"; + fixture.checkpoint <- Some (T.authoritative_batch_through authoritative); + fixture.revision <- Int64.succ fixture.revision; + fixture.submissions + <- List.filter + (fun (item : T.submission_descriptor) -> item.state = T.Queued) + fixture.submissions; + Ok + (D.Applied + { generation + ; before_projection_revision = projection + ; after_projection_revision = projection + ; checkpoint = Option.get fixture.checkpoint + ; sync_token = token fixture + ; terminal_receipts = [] + ; replanned_queued_ids = [] + ; blocked_ids = [] + ; logical_change_summary = T.No_logical_change + })) + | D.Activate_snapshot staged -> + record fixture "activate"; + (match fixture.activation_failure, fixture.snapshot_activation with + | Some error, _ -> Error error + | None, Some result -> Ok result + | None, None -> + fixture.checkpoint <- Some staged.through; + Ok (D.Activated generation)) + | D.Verify_checksum (through, expected) -> + record fixture "verify-checksum"; + if + Option.exists (T.Server_cursor.equal through) fixture.checkpoint + && not + (List.mem + (T.Checksum.to_string expected) + [ "checksum:v1:verified"; T.Checksum.to_string checksum ]) + then Error (D.Protocol_failure "Remote checkpoint checksum mismatch") + else Ok () + | D.Release_snapshot _ -> + record fixture "release-snapshot"; + Ok () +;; + +let rec transport + : type value. fixture -> value D.transport -> (value, D.error) result option + = + fun fixture operation -> + match operation with + | D.Connect -> + record fixture "connect"; + (match fixture.connect_failures with + | error :: rest -> + fixture.connect_failures <- rest; + Some (Error error) + | [] -> + Some + (Ok + (ok + (D.connection + ~owner:identity + ~token:("connection-" ^ string_of_int (count fixture "connect")) + ~generation:fixture.revision)))) + | D.Head -> + record fixture "head"; + Some + (match fixture.head_failure with + | Some error -> Error error + | None -> + Ok + { D.through = fixture.remote + ; snapshot_available = true + ; expected_checksum = fixture.expected_checksum + }) + | D.On_connection (_, operation) -> transport fixture operation + | D.Receive _ -> None + | D.Retire_connection _ -> + record fixture "retire"; + Some (Ok ()) + | D.Fetch_snapshot through -> + record fixture "snapshot"; + let resource = ok (D.resource ~owner:identity ~token:"staging") in + Some + (Ok + (ok + (D.snapshot + ~resource + ~through:(Option.value fixture.snapshot_through ~default:through) + ~checksum:"checksum:v1:verified" + ~encrypted:false))) + | D.Push actual -> + Alcotest.(check bool) + "push exposes actual submitting workflow" + true + (fixture.status = D.Service_submitting); + record fixture "push"; + Alcotest.(check bool) + "exact frozen ID" + true + (T.Submission_batch_id.equal (T.submission_batch_id actual) batch_id); + Alcotest.(check string) + "exact frozen bytes" + "original durable ciphertext" + (T.submission_wire_protected_transaction + (List.hd (T.submission_batch_wires actual))); + fixture.remote <- cursor 1L; + if fixture.unknown_pushes > 0 + then ( + fixture.unknown_pushes <- fixture.unknown_pushes - 1; + Some (Error (D.Outcome_unknown "lost response"))) + else Some (Ok (D.Accepted { through = fixture.remote; checksum })) + | D.Pull since -> + Alcotest.(check bool) + "pull exposes actual pulling workflow" + true + (fixture.status = D.Service_pulling); + record fixture "pull"; + if fixture.pulls_without_progress > 0 || T.Server_cursor.equal since fixture.remote + then ( + fixture.pulls_without_progress <- max 0 (fixture.pulls_without_progress - 1); + Some + (Ok + (D.No_progress + { since + ; through = fixture.remote + ; expected_checksum = fixture.pull_checksum + }))) + else ( + let transaction = + T.authoritative_transaction + ~cursor:fixture.remote + ~transaction:(ok (T.encoded_transaction_of_string ~maximum_bytes:1024 "[[]]")) + in + let batch = + ok + (T.authoritative_batch + ~maximum_count:1 + ~maximum_bytes:1024 + ~transactions:[ transaction ] + ~through:fixture.remote + ~checksum:fixture.pull_checksum) + in + Some (Ok (D.Pulled batch))) +;; + +let responder fixture owner = + fixture.owner <- Some owner; + let respond + : type value. D.scope -> value D.task_operation -> (value, D.error) result option + = + fun scope operation -> + fixture.scope <- Some scope; + match operation with + | D.Task_observe -> Some (Ok (observation fixture)) + | D.Task_database operation -> Some (database fixture operation) + | D.Task_transport operation -> transport fixture operation + | D.Capacity_revision -> + Some (D.Producer.capacity_revision owner scope ~sequence:fixture.revision) + | D.Await_capacity_changed revision -> + (match D.Producer.capacity_sequence owner scope revision with + | Error error -> Some (Error error) + | Ok sequence -> if sequence <> fixture.revision then Some (Ok ()) else None) + | D.Publish_status status -> + if status = D.Service_online + then ( + Alcotest.(check bool) + "online has reached authoritative remote checkpoint" + true + (fixture.checkpoint = Some fixture.remote); + Alcotest.(check int) + "online has no unsettled submissions" + 0 + (List.length fixture.submissions)); + fixture.status <- status; + fixture.statuses <- status :: fixture.statuses; + record fixture "status"; + Some (Ok ()) + | D.Resample_outbox -> + fixture.outbox_dirty <- true; + record fixture "resample"; + Some (Ok ()) + | D.Await_outbox_changed -> + if fixture.outbox_dirty + then ( + record fixture "outbox-check"; + Some (D.Producer.outbox_check owner scope)) + else None + | D.Acknowledge_outbox_check _ -> + fixture.outbox_dirty <- false; + record fixture "ack-outbox"; + Some (Ok ()) + | D.Abandon_outbox_check _ -> + record fixture "abandon-outbox"; + Some (Ok ()) + | D.Sleep seconds -> + if fixture.pause_sleep + then ( + if not (List.mem "sleep" fixture.trace) then record fixture "sleep"; + None) + else ( + record fixture ("sleep:" ^ string_of_float seconds); + Some (Ok ())) + | D.Yield -> Some (Ok ()) + | D.Consume_unlock _ -> + fixture.keys <- D.Key_available; + record fixture "unlock"; + Some (Ok D.Key_available) + | D.Download_demand _ -> + record fixture "asset-lookup"; + let report = + match fixture.asset_reports with + | report :: rest -> + fixture.asset_reports <- rest; + report + | [] -> { D.result = Ok D.Waiting_remote; attempts = 0 } + in + for _ = 1 to report.attempts do + record fixture "asset-attempt" + done; + Some (Ok report) + | D.Task_assets (D.Retain file) -> + record fixture "retain"; + Some (D.asset_lease (ok (D.resource ~owner:identity ~token:"lease")) file) + | D.Task_assets (D.Release_file _) -> + record fixture "release-file"; + Some (Ok ()) + | D.Task_assets (D.Release_lease _) -> + record fixture "release-lease"; + Some (Ok ()) + | D.Publish_demand (_, availability) -> + (match availability, fixture.publication_failure with + | D.Demand_available _, Some error -> + record fixture "publish-rejected"; + Some (Error error) + | _ -> + fixture.availability <- fixture.availability @ [ availability ]; + record fixture "publish"; + Some (Ok ())) + | D.Finish_demand _ -> + record fixture "finish-demand"; + Some (Ok ()) + | _ -> Alcotest.fail "unexpected policy task effect" + in + { I.respond } +;; + +let run ?(policy = config) ?(initial_network = open_gate 1L) ?input ?admit fixture = + let input = + match input with + | Some input -> input + | None -> + fun () -> + (match fixture.status with + | D.Service_online | D.Service_blocked _ -> Some D.Stop_requested + | _ -> None) + in + I.run_program + ~restore:(fun () -> + record fixture "restore"; + Ok + { D.keys = fixture.keys + ; checkpoint = fixture.checkpoint + ; recovery_pending = fixture.submissions <> [] + }) + ~install:(fun gate -> + fixture.install_count <- fixture.install_count + 1; + fixture.network <- gate; + record fixture "install"; + Ok ()) + ~runner: + { I.run_task = + (fun task -> + I.run_task ?admit ~identity ~responder:(responder fixture) ~input task) + } + (D.sync_policy policy ~initial_network) +;; + +let check_success = function + | Ok () -> () + | Error error -> Alcotest.fail (D.error_message error) +;; + +let check_count fixture event expected = + Alcotest.(check int) event expected (count fixture event) +;; + +let normal () = + let fixture = fixture ~submissions:[ descriptor T.Queued ] () in + check_success (run fixture); + check_count fixture "connect" 1; + check_count fixture "persist-submit" 1; + check_count fixture "persist-accept" 1; + check_count fixture "apply" 1; + check_count fixture "retire" 1; + let index event = + let rec find index = function + | [] -> max_int + | head :: rest -> if head = event then index else find (index + 1) rest + in + find 0 fixture.trace + in + Alcotest.(check bool) "durable before send" true (index "persist-submit" < index "push"); + Alcotest.(check bool) + "remote acceptance is not apply" + true + (index "persist-accept" < index "apply"); + Alcotest.(check int) "one-shot initial gate" 1 fixture.install_count +;; + +let recovery () = + let fixture = fixture ~submissions:[ descriptor (T.Submitted batch_id) ] () in + check_success (run fixture); + check_count fixture "persist-submit" 0; + check_count fixture "persist-retry" 1; + check_count fixture "push" 1; + check_count fixture "apply" 1 +;; + +let unknown_recovery () = + let fixture = fixture ~submissions:[ descriptor T.Queued ] () in + fixture.unknown_pushes <- 1; + fixture.defer <- true; + check_success (run fixture); + check_count fixture "persist-submit" 1; + check_count fixture "persist-retry" 1; + check_count fixture "deferred" 1; + check_count fixture "push" 2; + check_count fixture "pull" 1; + check_count fixture "apply" 1 +;; + +let nonnetwork () = + List.iter + (fun error -> + let fixture = fixture () in + fixture.head_failure <- Some error; + check_success (run fixture); + check_count fixture "connect" 1; + check_count fixture "head" 1; + check_count fixture "retire" 1; + Alcotest.(check bool) + "no non-network retry timer" + false + (List.exists + (fun entry -> String.starts_with ~prefix:"sleep" entry) + fixture.trace)) + [ D.Protocol_failure "bad framing" + ; D.Authentication_failure "revoked" + ; D.Storage_failure "disk full" + ] +;; + +let network_retry () = + let fixture = fixture () in + fixture.connect_failures <- [ D.Network_failure "offline" ]; + check_success (run fixture); + check_count fixture "connect" 2; + check_count fixture "sleep:0.25" 1 +;; + +let stop_backoff () = + let fixture = fixture () in + fixture.connect_failures <- [ D.Network_failure "offline" ]; + fixture.pause_sleep <- true; + let input () = if List.mem "sleep" fixture.trace then Some D.Stop_requested else None in + check_success (run ~input fixture); + check_count fixture "connect" 1; + check_count fixture "sleep" 1 +;; + +let gates () = + let fixture = fixture () in + let stage = ref 0 in + let input () = + match !stage with + | 0 -> + check_count fixture "connect" 0; + stage := 1; + fixture.network <- open_gate 7L; + Some (D.Network_changed fixture.network) + | 1 when fixture.status = D.Service_online -> + stage := 2; + fixture.network <- D.Closed; + Some (D.Network_changed D.Closed) + | 2 when count fixture "retire" = 1 -> + stage := 3; + Some D.Stop_requested + | _ -> None + in + check_success (run ~initial_network:D.Closed ~input fixture); + check_count fixture "connect" 1; + check_count fixture "retire" 1; + Alcotest.(check int) "initial gate not replayed" 1 fixture.install_count +;; + +let locked () = + let fixture = fixture () in + fixture.keys <- D.Key_locked; + let sent = ref false in + let input () = + if (not !sent) && fixture.scope <> None + then ( + check_count fixture "connect" 0; + sent := true; + let receipt = + ok + (D.Producer.unlock_receipt + (Option.get fixture.owner) + (Option.get fixture.scope)) + in + Some (D.Unlock_requested receipt)) + else if fixture.status = D.Service_online + then Some D.Stop_requested + else None + in + check_success (run ~input fixture); + check_count fixture "unlock" 1; + check_count fixture "connect" 1 +;; + +let no_progress () = + let fixture = fixture () in + fixture.remote <- cursor 1L; + fixture.pulls_without_progress <- 1; + check_success (run fixture); + check_count fixture "pull" 2; + check_count fixture "sleep:0.125" 1 +;; + +let snapshot () = + let fixture = fixture () in + fixture.checkpoint <- None; + check_success (run fixture); + check_count fixture "snapshot" 1; + check_count fixture "activate" 1; + check_count fixture "release-snapshot" 1 +;; + +let stop_completed_connection () = + let fixture = fixture () in + let input () = if count fixture "connect" > 0 then Some D.Stop_requested else None in + check_success (run ~input fixture); + check_count fixture "connect" 1; + check_count fixture "retire" 1 +;; + +let stop_completed_check () = + let fixture = fixture () in + let input () = + if count fixture "outbox-check" > 0 then Some D.Stop_requested else None + in + check_success (run ~input fixture); + check_count fixture "outbox-check" 1; + check_count fixture "abandon-outbox" 1 +;; + +let unlock_capacity () = + let fixture = fixture () in + fixture.keys <- D.Key_locked; + let admissions = ref 0 in + let admit () = + incr admissions; + if !admissions = 2 then Error (D.Capacity_exceeded "temporary") else Ok () + in + let stage = ref 0 in + let input () = + if !stage = 0 && fixture.scope <> None + then ( + stage := 1; + let receipt = + ok + (D.Producer.unlock_receipt + (Option.get fixture.owner) + (Option.get fixture.scope)) + in + Some (D.Unlock_requested receipt)) + else if !stage = 1 && !admissions >= 2 + then ( + stage := 2; + fixture.revision <- Int64.succ fixture.revision; + Some (D.Network_changed fixture.network)) + else if fixture.status = D.Service_online + then Some D.Stop_requested + else None + in + check_success (run ~input ~admit fixture); + check_count fixture "unlock" 1 +;; + +let retry_capacity () = + let fixture = fixture () in + fixture.connect_failures <- [ D.Network_failure "lost" ]; + let rejected = ref false in + let admit () = + match fixture.status with + | D.Service_backoff _ when not !rejected -> + rejected := true; + Error (D.Capacity_exceeded "temporary") + | _ -> Ok () + in + let input () = + if !rejected && fixture.revision = 0L + then ( + fixture.revision <- 1L; + Some (D.Network_changed fixture.network)) + else if fixture.status = D.Service_online + then Some D.Stop_requested + else None + in + check_success (run ~input ~admit fixture); + check_count fixture "sleep:0.25" 1; + check_count fixture "connect" 2 +;; + +let retry_exhaustion () = + let fixture = fixture () in + fixture.connect_failures <- List.init 4 (fun _ -> D.Network_failure "lost"); + check_success (run fixture); + check_count fixture "connect" 3; + check_count fixture "sleep:0.25" 1; + check_count fixture "sleep:0.5" 1 +;; + +let asset = + let module A = Logseq_db_types.Asset_descriptor in + let version = ok (A.version ~checksum:(String.make 64 'a') ~file_type:"png") in + ok + (A.create + ~uuid:mutation_id + ~source:(A.Managed (Some version)) + ~current_checksum:None + ~size:(Some 8L) + ~dimensions:None) +;; + +let asset_file = ok (D.asset_file (ok (D.resource ~owner:identity ~token:"file")) asset) + +let demand_input fixture = + let receipt = + ok (D.Producer.demand_receipt (Option.get fixture.owner) (Option.get fixture.scope)) + in + let id = ok (D.demand_id "image") in + D.Demand_changed (receipt, D.Demand { id; asset; priority = D.Foreground }) +;; + +let asset_flow reports = + let fixture = fixture () in + fixture.asset_reports <- reports; + let sent = ref false in + let input () = + if (not !sent) && fixture.scope <> None + then ( + sent := true; + Some (demand_input fixture)) + else if count fixture "finish-demand" > 0 + then Some D.Stop_requested + else None + in + check_success (run ~input fixture); + fixture +;; + +let asset_available () = + let fixture = asset_flow [ { D.result = Ok (D.Available asset_file); attempts = 1 } ] in + check_count fixture "retain" 1; + check_count fixture "asset-attempt" 1; + check_count fixture "release-file" 0; + check_count fixture "release-lease" 0; + Alcotest.(check bool) + "transferred availability" + true + (List.exists + (function + | D.Demand_available _ -> true + | _ -> false) + fixture.availability) +;; + +let asset_retry () = + let fixture = + asset_flow + [ { D.result = Error (D.Asset_network "offline"); attempts = 1 } + ; { D.result = Ok (D.Available asset_file); attempts = 1 } + ] + in + check_count fixture "asset-attempt" 2; + check_count fixture "sleep:0.25" 1; + Alcotest.(check bool) + "retry is actually scheduled" + true + (List.exists + (function + | D.Demand_failed { attempts = 1; retry_scheduled = true; _ } -> true + | _ -> false) + fixture.availability) +;; + +let asset_nonnetwork () = + List.iter + (fun reason -> + let fixture = asset_flow [ { D.result = Error reason; attempts = 1 } ] in + check_count fixture "asset-attempt" 1; + Alcotest.(check bool) + "asset permanent failure has no timer" + false + (List.exists + (fun entry -> String.starts_with ~prefix:"sleep" entry) + fixture.trace)) + [ D.Asset_storage_full; D.Asset_authentication "revoked"; D.Asset_checksum_mismatch ] +;; + +let asset_publish_failure () = + let fixture = fixture () in + fixture.asset_reports <- [ { D.result = Ok (D.Available asset_file); attempts = 1 } ]; + fixture.publication_failure <- Some D.Stale_context; + let sent = ref false in + let input () = + if (not !sent) && fixture.scope <> None + then ( + sent := true; + Some (demand_input fixture)) + else if count fixture "release-file" > 0 + then Some D.Stop_requested + else None + in + check_success (run ~input fixture); + check_count fixture "release-file" 1; + check_count fixture "release-lease" 1 +;; + +let asset_gate_close () = + let fixture = fixture () in + fixture.asset_reports + <- [ { D.result = Error (D.Asset_network "offline"); attempts = 1 } + ; { D.result = Error D.Asset_network_closed; attempts = 0 } + ]; + fixture.pause_sleep <- true; + let stage = ref 0 in + let input () = + if !stage = 0 && fixture.scope <> None + then ( + stage := 1; + Some (demand_input fixture)) + else if !stage = 1 && List.mem "sleep" fixture.trace + then ( + stage := 2; + fixture.network <- D.Closed; + Some (D.Network_changed D.Closed)) + else if + List.exists + (function + | D.Demand_waiting_network -> true + | _ -> false) + fixture.availability + then Some D.Stop_requested + else None + in + check_success (run ~input fixture); + check_count fixture "asset-attempt" 1; + check_count fixture "sleep" 1 +;; + +let snapshot_failure () = + let fixture = fixture () in + fixture.checkpoint <- None; + fixture.activation_failure <- Some (D.Storage_failure "activation failed"); + check_success (run fixture); + check_count fixture "activate" 1; + check_count fixture "release-snapshot" 1 +;; + +let snapshot_crypto_capacity_failure () = + let fixture = fixture () in + fixture.checkpoint <- None; + let cause = D.Capacity_exceeded "Crypto items" in + fixture.activation_failure <- Some cause; + check_success (run fixture); + check_count fixture "activate" 1; + check_count fixture "release-snapshot" 1; + Alcotest.(check bool) + "owned crypto failure blocks with original cause" + true + (List.mem (D.Service_blocked cause) fixture.statuses) +;; + +let snapshot_checksum_policy () = + List.iter + (fun (expected, older, blocked) -> + let fixture = fixture () in + fixture.checkpoint <- None; + fixture.remote <- cursor 3L; + fixture.expected_checksum + <- Option.map (fun value -> ok (T.Checksum.of_string value)) expected; + if older then fixture.snapshot_through <- Some (cursor 1L); + check_success (run fixture); + Alcotest.(check bool) + "checksum policy blocks exactly when required" + blocked + (List.exists + (function + | D.Service_blocked (D.Protocol_failure _) -> true + | _ -> false) + fixture.statuses); + check_count fixture "release-snapshot" 1; + check_count fixture "activate" (if blocked && not older then 0 else 1)) + [ Some "checksum:v1:wrong", false, true + ; Some "checksum:v1:verified", false, false + ; None, false, false + ; Some "checksum:v1:verified", true, false + ; Some "checksum:v1:wrong", true, true + ] +;; + +let existing_checkpoint_checksum_policy () = + List.iter + (fun (expected, blocked) -> + let fixture = fixture () in + fixture.expected_checksum <- Some (ok (T.Checksum.of_string expected)); + check_success (run fixture); + Alcotest.(check bool) + "same-cursor head checksum" + blocked + (List.exists + (function + | D.Service_blocked (D.Protocol_failure _) -> true + | _ -> false) + fixture.statuses)) + [ "checksum:v1:wrong", true; "checksum:v1:verified", false ] +;; + +let pull_checksum_policy () = + List.iter + (fun (expected, blocked) -> + let fixture = fixture () in + fixture.remote <- cursor 1L; + fixture.pull_checksum + <- Option.map (fun value -> ok (T.Checksum.of_string value)) expected; + check_success (run fixture); + check_count fixture "apply" 1; + check_count fixture "verify-checksum" 0; + let relevant = + List.filter + (fun event -> List.mem event [ "apply"; "verify-checksum" ]) + fixture.trace + in + Alcotest.(check (list string)) + "checksum validation belongs to atomic authoritative application" + [ "apply" ] + relevant; + Alcotest.(check bool) + "authoritative Pull checksum" + blocked + (List.exists + (function + | D.Service_blocked (D.Protocol_failure _) -> true + | _ -> false) + fixture.statuses)) + [ Some "checksum:v1:verified", false; None, false ] +;; + +let no_progress_checksum_policy () = + List.iter + (fun (expected, ahead, blocked) -> + let fixture = + fixture + ~submissions:[ descriptor (T.Accepted_pending_authoritative batch_id) ] + () + in + fixture.pull_checksum + <- Option.map (fun value -> ok (T.Checksum.of_string value)) expected; + if ahead + then ( + fixture.remote <- cursor 1L; + fixture.pulls_without_progress <- 1); + let input () = + if + (match fixture.status with + | D.Service_blocked _ -> true + | _ -> false) + || count fixture "sleep:0.125" > 0 + then Some D.Stop_requested + else None + in + check_success (run ~input fixture); + check_count + fixture + "verify-checksum" + (if Option.is_some expected && not ahead then 1 else 0); + Alcotest.(check bool) + "empty Pull checksum applies only at its cursor" + blocked + (List.exists + (function + | D.Service_blocked (D.Protocol_failure _) -> true + | _ -> false) + fixture.statuses)) + [ Some "checksum:v1:wrong", false, true + ; Some "checksum:v1:verified", false, false + ; None, false, false + ; Some "checksum:v1:wrong", true, false + ] +;; + +let snapshot_deferred () = + let fixture = fixture () in + fixture.checkpoint <- None; + fixture.snapshot_activation <- Some (D.Activation_deferred "local intent"); + let waits = ref 0 in + let input () = + if count fixture "release-snapshot" > 0 && !waits < 10 + then ( + check_count fixture "snapshot" 1; + incr waits; + if !waits = 10 + then ( + fixture.snapshot_activation <- None; + fixture.outbox_dirty <- true; + fixture.revision <- Int64.succ fixture.revision); + None) + else if fixture.status = D.Service_online + then Some D.Stop_requested + else None + in + check_success (run ~input fixture); + check_count fixture "snapshot" 2; + check_count fixture "release-snapshot" 2 +;; + +let outbox_inspection_failure () = + let fixture = fixture () in + fixture.inspect_failure <- Some (D.Storage_failure "inspection failed"); + let input () = + match fixture.status with + | D.Service_blocked _ -> Some D.Stop_requested + | _ -> None + in + check_success (run ~input fixture); + check_count fixture "outbox-check" 1; + check_count fixture "abandon-outbox" 1 +;; + +let inspect_bound_classification () = + let make_fixture = fixture in + let fixture = make_fixture () in + let cause = D.Invalid_input "Sync view items" in + fixture.inspect_failure <- Some cause; + let input () = + match fixture.status with + | D.Service_blocked actual -> + Alcotest.(check bool) "permanent cause preserved" true (actual = cause); + Some D.Stop_requested + | _ -> None + in + check_success (run ~input fixture); + Alcotest.(check bool) + "permanent mismatch terminates inspection" + true + (count fixture "inspect" <= 2); + let fixture = make_fixture () in + fixture.inspect_failure <- Some (D.Capacity_exceeded "transient ledger"); + let input () = + if count fixture "inspect" > 0 && fixture.revision = 0L + then ( + fixture.inspect_failure <- None; + fixture.revision <- 1L; + Some (D.Network_changed fixture.network)) + else if fixture.status = D.Service_online + then Some D.Stop_requested + else None + in + check_success (run ~input fixture); + Alcotest.(check bool) "transient capacity resumes" true (count fixture "inspect" >= 2) +;; + +let demand_lifecycle_metadata () = + let fixture = fixture () in + let completed = ref 0 + and phase = ref `Demand in + let baseline = ref 0 + and growth = ref None in + let input () = + if fixture.scope = None + then None + else ( + let id = ok (D.demand_id (string_of_int !completed)) in + let receipt () = + ok + (D.Producer.demand_receipt + (Option.get fixture.owner) + (Option.get fixture.scope)) + in + let finished = count fixture "finish-demand" > 0 in + match !phase with + | `Demand -> + fixture.trace <- []; + fixture.availability <- []; + phase := `Finish; + Some + (D.Demand_changed (receipt (), D.Demand { id; asset; priority = D.Foreground })) + | `Finish when finished -> + fixture.trace <- []; + fixture.availability <- []; + phase := `Withdraw; + Some (D.Demand_changed (receipt (), D.Withdraw id)) + | `Withdraw when finished -> + fixture.trace <- []; + fixture.availability <- []; + incr completed; + if !completed = 1000 + then ( + Gc.full_major (); + baseline := (Gc.stat ()).live_words); + if !completed = 2000 + then ( + Gc.full_major (); + growth := Some ((Gc.stat ()).live_words - !baseline); + Some D.Stop_requested) + else ( + phase := `Demand; + None) + | _ -> None) + in + check_success (run ~initial_network:D.Closed ~input fixture); + let growth = Option.get !growth in + Alcotest.(check bool) + (Printf.sprintf "pure policy retained growth: %d words" growth) + true + (growth < 5000) +;; + +let retry_disabled () = + let fixture = fixture () in + fixture.connect_failures <- [ D.Network_failure "offline" ]; + let retry = + ok + (D.retry_policy + ~initial_seconds:1. + ~maximum_seconds:1. + ~multiplier:1. + ~maximum_attempts:0) + in + let policy = + ok (D.policy_config ~sync_retry:retry ~asset_retry:retry ~no_progress_seconds:1.) + in + check_success (run ~policy fixture); + check_count fixture "connect" 1; + Alcotest.(check bool) + "zero retries disables timers" + false + (List.exists (fun entry -> String.starts_with ~prefix:"sleep" entry) fixture.trace) +;; + +let tests = + List.map + (fun (name, test) -> Alcotest.test_case name `Quick test) + [ "normal durable flow", normal + ; "Submitted exact recovery", recovery + ; "unknown outcome and Deferred reapply", unknown_recovery + ; "non-network failures never retry", nonnetwork + ; "bounded network retry", network_retry + ; "stop cancels retry sleep", stop_backoff + ; "gate changes and stop", gates + ; "locked waits for unlock", locked + ; "no-progress monotonic delay", no_progress + ; "snapshot release", snapshot + ; "stop joins unconsumed connection", stop_completed_connection + ; "stop abandons unconsumed outbox check", stop_completed_check + ; "unlock retained through capacity", unlock_capacity + ; "retry timer retained through capacity", retry_capacity + ; "retry exhaustion", retry_exhaustion + ; "asset ownership transfer", asset_available + ; "asset network retry", asset_retry + ; "asset non-network failures", asset_nonnetwork + ; "asset publication cleanup", asset_publish_failure + ; "gate close cancels asset backoff", asset_gate_close + ; "authoritative Pull checksum policy", pull_checksum_policy + ; "empty Pull checksum policy", no_progress_checksum_policy + ; "snapshot checksum checkpoint policy", snapshot_checksum_policy + ; "existing checkpoint integrity", existing_checkpoint_checksum_policy + ; "snapshot crypto capacity fails with cause", snapshot_crypto_capacity_failure + ; "snapshot failure releases staging", snapshot_failure + ; "deferred activation waits for change", snapshot_deferred + ; "outbox inspection failure abandons reservation", outbox_inspection_failure + ; "inspection permanent versus transient bounds", inspect_bound_classification + ; "demand lifecycle metadata bounded", demand_lifecycle_metadata + ; "retry disabled", retry_disabled + ] +;; diff --git a/logseq_sync/test/sync_protocol_contract.ml b/logseq_sync/test/sync_protocol_contract.ml index 4bccfa94..c24975b0 100644 --- a/logseq_sync/test/sync_protocol_contract.ml +++ b/logseq_sync/test/sync_protocol_contract.ml @@ -2,7 +2,7 @@ let server_cursor_of_int value = Logseq_db_types.Server_cursor.of_int64 (Int64.of_int value) |> Result.get_ok ;; -module Protocol = Logseq_sync_pure_reducer.Sync_protocol +module Protocol = Logseq_sync_protocol.Sync_protocol module Client = Protocol.Client module Server = Protocol.Server diff --git a/logseq_sync/test/test_asset_cache.ml b/logseq_sync/test/test_asset_cache.ml index d97473fc..4af068e1 100644 --- a/logseq_sync/test/test_asset_cache.ml +++ b/logseq_sync/test/test_asset_cache.ml @@ -1,181 +1,578 @@ -(* Cache filesystem scenarios run through Core.step/Effect_runner.submit in - runner_contract and transport_contract. This existing test entry checks the - public compiler boundary, without including implementation/private CMIs. *) -let public_interfaces = - [ ( "logseq_sync/spec/pure_reducer/.logseq_sync_pure_reducer.objs/byte" - , "logseq_sync_pure_reducer.cmi" ) - ; ( "logseq_sync/spec/effect_runner/.logseq_sync_effect_runner.objs/byte" - , "logseq_sync_effect_runner.cmi" ) - ; "logseq_db_types/lib/.logseq_db_types.objs/byte", "logseq_db_types.cmi" - ] -;; - -(* Compiler inputs belong to this executable's build, including --build-dir - builds outside the source checkout. Only public interface paths are used. *) -let rec compilation_root directory = - if - List.for_all - (fun (relative, interface) -> - Sys.file_exists (Filename.concat (Filename.concat directory relative) interface)) - public_interfaces - then directory - else ( - let parent = Filename.dirname directory in - if parent = directory - then failwith "public compiler interfaces not found beside test executable" - else compilation_root parent) -;; - -let read_file path = In_channel.with_open_bin path In_channel.input_all - -let contains text part = - let rec loop offset = - offset + String.length part <= String.length text - && (String.sub text offset (String.length part) = part || loop (offset + 1)) - in - loop 0 +module Cache = Logseq_sync_assets.Cache +module Namespace = Logseq_sync_assets.Namespace +module Codec = Logseq_sync_native_crypto.Asset_codec +module Uuid = Logseq_db_types.Graph_types.Uuid + +let ok = function + | Ok value -> value + | Error _ -> Alcotest.fail "unexpected error" +;; + +let some = function + | Some value -> value + | None -> Alcotest.fail "expected resource" +;; + +let bool = Alcotest.check Alcotest.bool +let exists label expected path = bool label expected (Sys.file_exists path) +let read path = In_channel.with_open_bin path In_channel.input_all +let write path bytes = Out_channel.with_open_bin path (fun out -> output_string out bytes) + +let uuid index = + Uuid.of_string (Printf.sprintf "96000000-0000-4000-8000-%012d" index) |> ok +;; + +let operation = uuid 1 +let asset = uuid 2 + +let scope ?(origin = "https://sync.example.test") ?(user = "alice") ?(graph = 3) () + : Namespace.graph_scope + = + { account = { managed_sync_origin = Uri.of_string origin; user_id = user } + ; graph_id = uuid graph + } ;; -let compile root source = - let filename = Filename.temp_file "sync-public-boundary" ".ml" in - let log = filename ^ ".log" in - let output = filename ^ ".cmo" in - let interface = filename ^ ".cmi" in +let rec remove_tree path = + match Unix.lstat path with + | { Unix.st_kind = Unix.S_DIR; _ } -> + Array.iter (fun name -> remove_tree (Filename.concat path name)) (Sys.readdir path); + Unix.rmdir path + | _ -> Unix.unlink path + | exception Unix.Unix_error (Unix.ENOENT, _, _) -> () +;; + +let with_fixture test = + let root = Filename.temp_file ~temp_dir:(Sys.getcwd ()) "asset-cache-fixture-" "" in + Sys.remove root; + Unix.mkdir root 0o700; + let caches = ref [] in + let create ?(scope = scope ()) ?(budget = 8L) ?(maximum = 8) () = + let cache = + Cache.create ~root ~scope ~budget_bytes:budget ~maximum_file_bytes:maximum |> ok + in + caches := cache :: !caches; + cache + in + let source = Filename.concat root "source.bin" in + write source "file"; Fun.protect ~finally:(fun () -> - List.iter - (fun file -> if Sys.file_exists file then Sys.remove file) - [ filename; log; output; interface ]) - (fun () -> - Out_channel.with_open_bin filename (fun channel -> output_string channel source); - let directories = - List.map (fun (relative, _) -> Filename.concat root relative) public_interfaces - in - let arguments = - Array.of_list - (("ocamlc" - :: "-c" - :: "-o" - :: output - :: List.concat_map (fun directory -> [ "-I"; directory ]) directories) - @ [ filename ]) - in - let fd = Unix.openfile log [ Unix.O_CREAT; Unix.O_TRUNC; Unix.O_WRONLY ] 0o600 in - let process = Unix.create_process "ocamlc" arguments Unix.stdin fd fd in - Unix.close fd; - let _, status = Unix.waitpid [] process in - status, read_file log) -;; - -let positive root source = - let status, diagnostics = compile root source in - Alcotest.(check bool) diagnostics true (status = Unix.WEXITED 0) -;; - -let negative root source reason = - let status, diagnostics = compile root source in - Alcotest.(check bool) ("must fail: " ^ source) false (status = Unix.WEXITED 0); - Alcotest.(check bool) - ("rejected at intended boundary: " ^ diagnostics) - true - (contains diagnostics reason) -;; - -let sealed_public_interface () = - let executable = - if Filename.is_relative Sys.executable_name - then Filename.concat (Sys.getcwd ()) Sys.executable_name - else Sys.executable_name - in - let root = compilation_root (Filename.dirname executable) in - let prefix = "module C = Logseq_sync_pure_reducer.Core\n" in - positive - root - (prefix - ^ "let carry (x : C.runner_effect) = x\n\ - let inspect = C.runner_effect_diagnostic\n\ - let submit = Logseq_sync_effect_runner.Effect_runner.submit\n"); - positive - root - "module C = Logseq_sync_pure_reducer__Core\nlet carry (x : C.runner_effect) = x\n"; - negative root (prefix ^ "let bypass = C.staged_artifact\n") "Unbound value"; - negative root - (prefix ^ "let forge (artifact : C.staged_artifact) (scope : C.graph_scope) cursor : C.snapshot_activation_request = { artifact; scope; applied_server_t = cursor; key = None }\n") - "Unbound record field"; - negative root - (prefix ^ "let forge (value : bool) : C.unlock_attached = value\n") - "but an expression was expected of type"; - negative root - (prefix ^ "let forge (value : string) : C.unlock_challenge = value\n") - "but an expression was expected of type"; - negative root - "let forge (value : string) : Logseq_sync_effect_runner.Asset_codec.verified = value\n" - "but an expression was expected of type"; - negative root (prefix ^ "let forge scope = C.Cancel_effects scope\n") "private type"; - negative - root - (prefix - ^ "let forge x = match x with C.Asset_io (ticket, request) -> C.Asset_io (ticket, \ - request) | _ -> x\n") - "private type"; - negative - root - (prefix - ^ "type alias = C.runner_effect\nlet forge scope : alias = C.Cancel_effects scope\n" - ) - "private type"; - negative - root - "module C = Logseq_sync_pure_reducer__Core\n\ - let forge scope = C.Cancel_effects scope\n" - "private type"; - negative - root - (prefix ^ "let forge : C.asset_ticket = { asset_serial = 1; asset_request = () }\n") - "Unbound record field"; - negative - root - (prefix ^ "let forge : unit C.effect_ticket = { id = 1; scope = (); kind = () }\n") - "Unbound record field"; - negative - root - "let bypass = Logseq_sync_effect_runner.Asset_cache.create\n" - "Unbound value"; - negative root "module B = Logseq_sync_effect_runner.Bootstrap\n" "Unbound module"; - negative - root - "module B = Logseq_sync_effect_runner__logseq_sync_effect_runner_impl__Bootstrap\n" - "Unbound module"; - negative root "module B = Logseq_sync_effect_runner_impl.Bootstrap\n" "Unbound module"; - List.iter - (fun operation -> - negative - root - ("let bypass = Logseq_sync_effect_runner.Effect_runner." ^ operation ^ "\n") - "Unbound value") - [ "submit_asset" - ; "run_scoped_asset" - ; "upload_asset" - ; "stage_asset" - ; "retain_asset_file" - ; "release_asset_file" - ; "staged_asset_path" - ; "close_asset_scope" - ; "delete_graph_assets" - ; "release_staged_asset" - ; "prune_staged_assets" - ; "retain_staged_file" - ; "encrypt_protected_values" - ; "decrypt_protected_value" - ; "authenticated_operation" - ] + List.iter Cache.close !caches; + remove_tree root) + (fun () -> test root source create) +;; + +let stage ?(budget = 64L) ?(file_type = "bin") cache source = + Cache.stage cache ~operation ~file_type ~source_file:source ~pending_budget_bytes:budget +;; + +let staged_path cache (staged : Cache.staged) = + Cache.staged_path cache ~file:staged.file |> some +;; + +let release cache handle = Cache.release_checked cache handle |> ok +let close cache = Cache.close_checked cache |> ok + +let invalid = function + | Error (Cache.Invalid _) -> () + | _ -> Alcotest.fail "expected Invalid" +;; + +let full result = bool "storage full" true (result = Error Cache.Full) + +let durable_staging () = + with_fixture (fun _ source create -> + let cache = create ~budget:0L () in + let staged = stage cache source |> ok in + Alcotest.(check string) "checksum" (Codec.checksum "file") staged.checksum; + Alcotest.(check int64) "size" 4L staged.size; + let path = staged_path cache staged in + Sys.remove source; + Alcotest.(check string) "independent copy" "file" (read path); + write (path ^ ".part") "interrupted"; + close cache; + let reopened = create ~budget:0L () in + Alcotest.(check string) "same durable path" path (staged_path reopened staged); + exists "partial removed" false (path ^ ".part"); + ignore (Cache.prune reopened |> ok); + exists "pending not evicted" true path; + Cache.release_staged reopened ~file:staged.file |> ok; + Cache.release_staged reopened ~file:staged.file |> ok; + exists "completion removes copy" false path) +;; + +let staging_limits () = + with_fixture (fun _ source create -> + let cache = create ~maximum:4 () in + write source "large"; + invalid (stage cache source); + write source "file"; + let staged = stage ~budget:4L cache source |> ok in + full (stage ~budget:4L cache source); + exists "budget rejection preserves source" true (staged_path cache staged); + Cache.release_staged cache ~file:staged.file |> ok; + ignore (stage ~budget:4L cache source |> ok)) +;; + +let empty_staging_budget () = + with_fixture (fun _ source create -> + let cache = create () in + write source ""; + full (stage ~budget:0L cache source); + let staged = stage ~budget:1L cache source |> ok in + full (stage ~budget:1L cache source); + Alcotest.(check int64) "empty size" 0L staged.size) +;; + +let invalid_inputs () = + with_fixture (fun root source create -> + let cache = create () in + invalid (stage ~file_type:"../bin" cache source); + invalid (stage ~budget:(-1L) cache source); + invalid (stage cache root); + bool + "traversal not readable" + true + (Cache.staged_path cache ~file:"../source.bin" = None); + bool + "traversal not retainable" + true + (Cache.retain_staged cache ~file:"../source.bin" = None); + invalid (Cache.release_staged cache ~file:"../source.bin"); + invalid + (Cache.create ~root ~scope:(scope ()) ~budget_bytes:(-1L) ~maximum_file_bytes:8); + invalid + (Cache.create + ~root + ~scope:(scope ()) + ~budget_bytes:8L + ~maximum_file_bytes:((100 * 1024 * 1024) + 1))) +;; + +let preview_leases () = + with_fixture (fun _ source create -> + let cache = create () in + let staged = stage cache source |> ok in + let first = Cache.retain_staged cache ~file:staged.file |> some in + let second = Cache.retain cache first |> some in + bool "distinct leases" true (first <> second); + let path = Cache.path cache first |> some in + full (Cache.release_staged_if_unused cache ~file:staged.file); + Cache.release_staged cache ~file:staged.file |> ok; + bool + "retired refuses new preview" + true + (Cache.retain_staged cache ~file:staged.file = None); + let report = Cache.report cache |> ok in + Alcotest.(check int64) "retained bytes counted once" 4L report.retained_bytes; + Alcotest.(check int) "deferred cleanup" 1 report.cleanup_pending; + invalid (Cache.close_checked cache); + release cache first; + exists "second lease preserves bytes" true path; + release cache second; + release cache second; + exists "last release cleans" false path; + bool "released lease invalid" true (Cache.path cache second = None); + close cache) +;; + +let nonterminal_preview () = + with_fixture (fun _ source create -> + let cache = create () in + let staged = stage cache source |> ok in + let lease = Cache.retain_staged cache ~file:staged.file |> some in + let path = Cache.path cache lease |> some in + release cache lease; + close cache; + exists "close preserves pending ownership" true path; + bool + "closed scope rejects retain" + true + (Cache.retain_staged cache ~file:staged.file = None); + bool "closed stage stale" true (stage cache source = Error Cache.Stale)) +;; + +let checked_release_retry () = + with_fixture (fun _ source create -> + let cache = create () in + let staged = stage cache source |> ok in + let lease = Cache.retain_staged cache ~file:staged.file |> some in + let path = Cache.path cache lease |> some in + Cache.release_staged cache ~file:staged.file |> ok; + let saved = path ^ ".saved" in + Unix.rename path saved; + Unix.mkdir path 0o700; + (match Cache.release_checked cache lease with + | Error (Cache.Io _) -> () + | _ -> Alcotest.fail "cleanup failure must be typed"); + bool "failed release retains exact handle" true (Cache.path cache lease = Some path); + invalid (Cache.close_checked cache); + Unix.rmdir path; + Unix.rename saved path; + release cache lease; + exists "retry completes cleanup" false path; + close cache) +;; + +let exact_instances () = + with_fixture (fun _ source create -> + let cache = create () in + let older = stage cache source |> ok in + let newer = stage cache source |> ok in + bool "same operation fresh physical identity" true (older.file <> newer.file); + let old_path = staged_path cache older in + let new_path = staged_path cache newer in + let seen = ref [] in + let removed = + Cache.prune_staged cache ~keep:(fun file -> + seen := file :: !seen; + Ok (file = newer.file)) + |> ok + in + Alcotest.(check int) "one orphan" 1 removed; + Alcotest.(check (list string)) + "exact names checked" + (List.sort String.compare [ older.file; newer.file ]) + (List.sort String.compare !seen); + exists "old removed" false old_path; + Cache.release_staged cache ~file:older.file |> ok; + exists "late old completion preserves replacement" true new_path; + Alcotest.(check int) + "idempotent prune" + 0 + (Cache.prune_staged cache ~keep:(fun _ -> Ok true) |> ok)) +;; + +let prune_legacy_and_strangers () = + with_fixture (fun _ source create -> + let cache = create () in + let staged = stage cache source |> ok in + let directory = Filename.dirname (staged_path cache staged) in + let legacy = Uuid.to_string (uuid 4) ^ ".bin" in + let legacy_path = Filename.concat directory legacy in + write legacy_path "legacy"; + let lease = Cache.retain_staged cache ~file:legacy |> some in + Alcotest.(check string) + "legacy preview" + "legacy" + (read (Cache.path cache lease |> some)); + release cache lease; + let stranger = Filename.concat directory "unknown.txt" in + write stranger "keep"; + let link = Filename.concat directory (Uuid.to_string (uuid 5) ^ ".bin") in + Unix.symlink source link; + let seen = ref [] in + Alcotest.(check int) + "legacy pruned" + 1 + (Cache.prune_staged cache ~keep:(fun file -> + seen := file :: !seen; + Ok (file = staged.file)) + |> ok); + bool "legacy exact identity checked" true (List.mem legacy !seen); + exists "legacy removed" false legacy_path; + exists "stranger preserved" true stranger; + bool "symlink preserved" true ((Unix.lstat link).st_kind = Unix.S_LNK); + exists "symlink target preserved" true source) +;; + +let prune_failure_atomic () = + with_fixture (fun _ source create -> + let cache = create () in + let first = stage cache source |> ok in + let second = stage cache source |> ok in + let calls = ref 0 in + (match + Cache.prune_staged cache ~keep:(fun _ -> + incr calls; + if !calls = 2 then Error "ownership unavailable" else Ok false) + with + | Error (Cache.Io _) -> () + | _ -> Alcotest.fail "ownership failure must abort prune"); + exists "first survives aborted prune" true (staged_path cache first); + exists "second survives aborted prune" true (staged_path cache second)) +;; + +let prune_pinned () = + with_fixture (fun _ source create -> + let cache = create () in + let staged = stage cache source |> ok in + let lease = Cache.retain_staged cache ~file:staged.file |> some in + Alcotest.(check int) + "pinned skipped" + 0 + (Cache.prune_staged cache ~keep:(fun _ -> + Alcotest.fail "pinned ownership must not be queried") + |> ok); + release cache lease; + Alcotest.(check int) + "released orphan removed" + 1 + (Cache.prune_staged cache ~keep:(fun _ -> Ok false) |> ok)) +;; + +let directory_limit () = + with_fixture (fun _ source create -> + let cache = create () in + let staged = stage cache source |> ok in + let path = staged_path cache staged in + let directory = Filename.dirname path in + for index = 1 to 4096 do + write (Filename.concat directory ("unknown-" ^ string_of_int index)) "" + done; + full (stage cache source); + full (Cache.prune_staged cache ~keep:(fun _ -> Ok false)); + exists "scan overflow removes nothing" true path) +;; + +let crypto : Codec.crypto = + { encrypt = (fun ~key:_ ~plaintext:_ -> Error "unused") + ; decrypt = (fun ~key:_ ~iv:_ ~ciphertext:_ -> Error "unused") + } +;; + +let version bytes = + Logseq_db_types.Asset_descriptor.version + ~checksum:(Codec.checksum bytes) + ~file_type:"bin" + |> ok +;; + +let verified bytes = + Codec.decode + ~maximum_plaintext_bytes:64 + ~crypto + ~key:None + ~expected:(version bytes) + bytes + |> ok +;; + +let publish cache bytes = + Cache.publish cache ~asset ~current:(fun () -> true) ~verified:(verified bytes) +;; + +let published_durable () = + with_fixture (fun _ _ create -> + let cache = create () in + let lease = publish cache "file" |> ok in + let path = Cache.path cache lease |> some in + let duplicate = Cache.retain cache lease |> some in + invalid (Cache.close_checked cache); + release cache lease; + exists "retained published bytes" true path; + release cache duplicate; + close cache; + let reopened = create () in + let restored = Cache.lookup reopened ~asset ~version:(version "file") |> ok |> some in + Alcotest.(check string) + "manifest-backed recovery" + "file" + (read (Cache.path reopened restored |> some)); + release reopened restored) +;; + +let published_limits () = + with_fixture (fun _ _ create -> + let cache = create ~budget:4L () in + let lease = publish cache "file" |> ok in + full (publish cache "next"); + release cache lease; + let next = publish cache "next" |> ok in + bool + "unpinned entry evicted" + true + (Cache.lookup cache ~asset ~version:(version "file") |> ok = None); + release cache next; + invalid (publish cache "123456789"); + bool + "stale publication refused" + true + (Cache.publish cache ~asset ~current:(fun () -> false) ~verified:(verified "next") + = Error Cache.Stale)) +;; + +let corrupt_publication () = + with_fixture (fun _ _ create -> + let cache = create () in + let lease = publish cache "file" |> ok in + let path = Cache.path cache lease |> some in + write path "evil"; + bool + "corrupt bytes not served" + true + (Cache.lookup cache ~asset ~version:(version "file") |> ok = None); + full (publish cache "file"); + release cache lease; + bool + "corrupt unpinned entry removed" + true + (Cache.lookup cache ~asset ~version:(version "file") |> ok = None); + exists "corrupt file removed" false path; + let replacement = publish cache "file" |> ok in + release cache replacement) +;; + +let scope_isolation () = + with_fixture (fun _ source create -> + let original = create () in + let staged = stage original source |> ok in + let published = publish original "file" |> ok in + List.iter + (fun scoped -> + let other = create ~scope:scoped () in + bool "staging isolated" true (Cache.staged_path other ~file:staged.file = None); + bool + "published isolated" + true + (Cache.lookup other ~asset ~version:(version "file") |> ok = None); + bool "lease isolated" true (Cache.path other published = None); + release other published; + bool "foreign release harmless" true (Cache.path original published <> None)) + [ scope ~origin:"https://other.example.test" () + ; scope ~user:"bob" () + ; scope ~graph:4 () + ]; + release original published) +;; + +let retirement index = + Namespace.retirement_id_of_string (Uuid.to_string (uuid index)) |> ok +;; + +let deletion scope retirement = Namespace.asset_deletion_reference ~scope ~retirement + +let graph_abort () = + with_fixture (fun root source create -> + let scoped = scope () in + let cache = create () in + let staged = stage cache source |> ok in + let path = staged_path cache staged in + close cache; + let retirement = retirement 10 in + let deletion = deletion scoped retirement in + Cache.prepare_graph_deletion ~root ~scope:scoped ~retirement |> ok; + Cache.prepare_graph_deletion ~root ~scope:scoped ~retirement |> ok; + exists "prepared namespace moved" false path; + bool + "not terminal while prepared" + true + (Cache.graph_deletion_terminal ~root ~scope:scoped ~retirement |> ok = None); + Cache.finish_graph_deletion ~root ~deletion ~commit:false |> ok; + exists "abort restores bytes" true path; + Cache.finish_graph_deletion ~root ~deletion ~commit:true |> ok; + exists "late commit cannot reverse abort" true path; + bool + "abort terminal" + true + (Cache.graph_deletion_terminal ~root ~scope:scoped ~retirement |> ok = Some `Aborted)) +;; + +let graph_commit_replacement () = + with_fixture (fun root source create -> + let scoped = scope () in + let cache = create () in + let staged = stage cache source |> ok in + let path = staged_path cache staged in + close cache; + let retirement = retirement 11 in + let deletion = deletion scoped retirement in + Cache.prepare_graph_deletion ~root ~scope:scoped ~retirement |> ok; + let replacement = create () in + let fresh = stage replacement source |> ok in + let fresh_path = staged_path replacement fresh in + close replacement; + Cache.finish_graph_deletion ~root ~deletion ~commit:true |> ok; + Cache.finish_graph_deletion ~root ~deletion ~commit:false |> ok; + Cache.finish_graph_deletion ~root ~deletion ~commit:true |> ok; + exists "old source removed" false path; + exists "replacement survives terminal replay" true fresh_path; + bool + "commit terminal" + true + (Cache.graph_deletion_terminal ~root ~scope:scoped ~retirement + |> ok + = Some `Committed)) +;; + +let graph_restore_conflict () = + with_fixture (fun root source create -> + let scoped = scope () in + let cache = create () in + let staged = stage cache source |> ok in + let path = staged_path cache staged in + close cache; + let retirement = retirement 12 in + let deletion = deletion scoped retirement in + Cache.prepare_graph_deletion ~root ~scope:scoped ~retirement |> ok; + let replacement = create () in + let fresh = stage replacement source |> ok in + let fresh_path = staged_path replacement fresh in + close replacement; + bool + "restore conflict reported" + true + (Result.is_error (Cache.finish_graph_deletion ~root ~deletion ~commit:false)); + exists "replacement not overwritten" true fresh_path; + remove_tree (Filename.dirname (Filename.dirname fresh_path)); + Cache.finish_graph_deletion ~root ~deletion ~commit:false |> ok; + exists "retry restores original" true path) +;; + +let account_deletion () = + with_fixture (fun root source create -> + let scopes = + [ scope () + ; scope ~graph:4 () + ; scope ~user:"bob" () + ; scope ~origin:"https://other.example.test" () + ] + in + let paths = + List.map + (fun scoped -> + let cache = create ~scope:scoped () in + let staged = stage cache source |> ok in + let path = staged_path cache staged in + close cache; + path) + scopes + in + Cache.delete_account ~root ~account:(scope ()).account |> ok; + List.iter2 + (fun expected path -> exists "account/origin deletion boundary" expected path) + [ false; false; true; true ] + paths; + Cache.delete_account ~root ~account:(scope ()).account |> ok) ;; let () = Alcotest.run - "sync public interface" - [ ( "external compilation" - , [ Alcotest.test_case "sealed execution and cache" `Quick sealed_public_interface ] - ) + "native asset cache" + [ ( "Cache" + , List.map + (fun (name, test) -> Alcotest.test_case name `Quick test) + [ "durable staging and recovery", durable_staging + ; "staging file and pending limits", staging_limits + ; "empty staging consumes capacity", empty_staging_budget + ; "invalid inputs and traversal", invalid_inputs + ; "preview leases and deferred cleanup", preview_leases + ; "nonterminal preview survives close", nonterminal_preview + ; "checked release retries exact handle", checked_release_retry + ; "prune exact physical identities", exact_instances + ; "legacy staging and strangers", prune_legacy_and_strangers + ; "ownership failure makes prune atomic", prune_failure_atomic + ; "prune protects preview leases", prune_pinned + ; "bounded staging directory scan", directory_limit + ; "published bytes survive reopen", published_durable + ; "published limits and eviction", published_limits + ; "corrupt pinned publication", corrupt_publication + ; "origin account graph isolation", scope_isolation + ; "graph retirement abort", graph_abort + ; "graph retirement commit and replacement", graph_commit_replacement + ; "graph restore conflict and retry", graph_restore_conflict + ; "account deletion isolation", account_deletion + ] ) ] ;; diff --git a/logseq_sync/test/test_asset_codec.ml b/logseq_sync/test/test_asset_codec.ml index c1e56325..42b4efa6 100644 --- a/logseq_sync/test/test_asset_codec.ml +++ b/logseq_sync/test/test_asset_codec.ml @@ -1,4 +1,4 @@ -module C = Logseq_sync_effect_runner.Asset_codec +module C = Logseq_sync_native_crypto.Asset_codec module T = Transit_native.Transit.Json let get = function diff --git a/logseq_sync/test/test_asset_transfer.ml b/logseq_sync/test/test_asset_transfer.ml index e681dca2..93f4efbe 100644 --- a/logseq_sync/test/test_asset_transfer.ml +++ b/logseq_sync/test/test_asset_transfer.ml @@ -1,5 +1,5 @@ -module T = Logseq_sync_pure_reducer.Asset_transfer -module C = Logseq_sync_pure_reducer.Core +module T = Logseq_sync_asset_transfer.Core +module C = Logseq_sync_domain.Core module A = Logseq_db_types.Asset_descriptor module U = Logseq_db_types.Graph_types.Uuid diff --git a/logseq_sync/test/test_sync.ml b/logseq_sync/test/test_sync.ml deleted file mode 100644 index f7321e54..00000000 --- a/logseq_sync/test/test_sync.ml +++ /dev/null @@ -1,12 +0,0 @@ -let () = - Alcotest.run - "logseq sync" - [ "sync protocol", Sync_protocol_contract.scenarios - ; ( "pure core" - , Core_contract.scenarios @ [ Local_restore_failure_reconciliation.scenario ] ) - ; "sync recovery reproductions", Core_contract.sync_recovery_reproductions - ; "asset execution core", Core_contract.asset_scenarios - ; "effect runner", Runner_contract.scenarios - ; "transport", Transport_contract.scenarios - ] -;; diff --git a/logseq_sync/test/test_wire_protocol.ml b/logseq_sync/test/test_wire_protocol.ml new file mode 100644 index 00000000..6c05b9a3 --- /dev/null +++ b/logseq_sync/test/test_wire_protocol.ml @@ -0,0 +1,3 @@ +let () = + Alcotest.run "logseq sync protocol" [ "wire codec", Sync_protocol_contract.scenarios ] +;; diff --git a/logseq_sync/test/tls_peer.ml b/logseq_sync/test/tls_peer.ml new file mode 100644 index 00000000..eeef4ec4 --- /dev/null +++ b/logseq_sync/test/tls_peer.ml @@ -0,0 +1,227 @@ +let rec remove path = + match Unix.lstat path with + | { st_kind = Unix.S_DIR; _ } -> + Array.iter (fun name -> remove (Filename.concat path name)) (Sys.readdir path); + Unix.rmdir path + | _ -> Unix.unlink path + | exception Unix.Unix_error (Unix.ENOENT, _, _) -> () +;; + +let with_directory callback = + let path = Filename.temp_file "native-http-peer-" "" in + Sys.remove path; + Unix.mkdir path 0o700; + Fun.protect ~finally:(fun () -> remove path) (fun () -> callback path) +;; + +let peer = + {python| +import base64, gzip, hashlib, json, os, select, socket, ssl, struct, subprocess, sys, time, threading +root = sys.argv[1] +with open(root+'/cases.json') as f: cases = json.load(f) +key, cert = root + '/key.pem', root + '/cert.pem' +subprocess.run(['openssl','req','-x509','-newkey','ec','-pkeyopt','ec_paramgen_curve:P-256', + '-nodes','-keyout',key,'-out',cert,'-days','1','-subj','/CN=localhost'], + check=True, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL) +ctx = ssl.SSLContext(ssl.PROTOCOL_TLS_SERVER) +ctx.load_cert_chain(cert,key) +for attempt in range(16): + listener = socket.socket(socket.AF_INET6) + listener.setsockopt(socket.IPPROTO_IPV6, socket.IPV6_V6ONLY, 1) + listener4 = socket.socket() + try: + listener.bind(('::1',0)) + listener4.bind(('127.0.0.1', listener.getsockname()[1])) + break + except OSError: + listener.close(); listener4.close() + if attempt == 15: raise +listener.listen(8); listener.settimeout(5); listener4.listen(8) +with open(root+'/ready.tmp','w') as f: f.write(str(listener.getsockname()[1])) +os.replace(root+'/ready.tmp', root+'/ready') +report=[]; stalled=[] +def observe_stalled_tls(raw, item, index): + try: + hello=b'' + while True: + part=raw.recv(16384) + if not part: + item['closed']=True; break + if not item['client_hello']: + hello+=part + if len(hello)>5: + record_length=int.from_bytes(hello[3:5],'big') + if len(hello)>=5+record_length and hello[0]==22 and hello[5]==1: + item['client_hello']=True + path=root+'/client-hello-'+str(index) + with open(path+'.tmp','w') as f: f.write('ClientHello observed') + os.replace(path+'.tmp',path) + except (ConnectionResetError, BrokenPipeError): item['closed']=True + except socket.timeout: pass + finally: raw.close() +def serve_case(index,case,raw): + raw.settimeout(2) + if case.get('stall_tls'): + item={'closed':False,'client_hello':False}; report.append(item) + worker=threading.Thread(target=observe_stalled_tls,args=(raw,item,index),daemon=True) + worker.start(); stalled.append(worker); return + time.sleep(case.get('tls_delay',0)) + try: conn=ctx.wrap_socket(raw,server_side=True) + except (ssl.SSLError,OSError): + raw.close(); report.append({'closed':True,'frames':[]}); return + request=b'' + while b'\r\n\r\n' not in request: + part=conn.recv(16384) + if not part: raise RuntimeError('no request') + request+=part + header_end=request.index(b'\r\n\r\n')+4 + request_headers=dict(line.split(b':',1) for line in request[:header_end].split(b'\r\n')[1:] if b':' in line) + length=next((int(v.strip()) for k,v in request_headers.items() if k.lower()==b'content-length'),0) + while len(request)=2: + length=pending[1]&127; offset=2 + if length==126: + if len(pending)<4: break + length=int.from_bytes(pending[2:4],'big'); offset=4 + elif length==127: + if len(pending)<10: break + length=int.from_bytes(pending[2:10],'big'); offset=10 + masked=bool(pending[1]&128) + if len(pending) Seq.map (fun c -> Printf.sprintf "%02x" (Char.code c)) + |> List.of_seq + |> String.concat "" +;; + +let case ?(ws = false) ?(extra = "") ?(segment = 16384) ?(eof = false) wire = + `Assoc + [ "wire", `String (hex wire) + ; "ws", `Bool ws + ; "extra", `String extra + ; "segment", `Int segment + ; "eof", `Bool eof + ] +;; + +let wait clock predicate = + Eio.Time.with_timeout_exn clock 5. (fun () -> + while not (predicate ()) do + Eio.Time.sleep clock 0.005 + done) +;; + +let read_file path = + let input = open_in_bin path in + Fun.protect + ~finally:(fun () -> close_in_noerr input) + (fun () -> really_input_string input (in_channel_length input)) +;; + +let scaled_clock clock scale = + let module Clock = struct + type t = unit + type time = float + + let now () = Eio.Time.now clock *. scale + let sleep_until () deadline = Eio.Time.sleep_until clock (deadline /. scale) + end + in + Eio.Resource.T ((), Eio.Time.Pi.clock (module Clock)) +;; + +let with_peer cases f = + with_directory (fun support -> + Eio_main.run (fun environment -> + Eio.Switch.run (fun sw -> + let clock = Eio.Stdenv.clock environment in + Yojson.Basic.to_file (Filename.concat support "cases.json") (`List cases); + let process = + Eio.Process.spawn + ~sw + (Eio.Stdenv.process_mgr environment) + [ "python3"; "-c"; peer; support ] + in + wait clock (fun () -> Sys.file_exists (Filename.concat support "ready")); + let port = int_of_string (read_file (Filename.concat support "ready")) in + let origin = Uri.of_string (Printf.sprintf "https://localhost:%d" port) in + f ~environment ~sw ~support ~origin; + Eio.Time.with_timeout_exn clock 5. (fun () -> Eio.Process.await_exn process); + Yojson.Basic.from_file (Filename.concat support "report")))) +;; diff --git a/logseq_sync/test/transport_contract.ml b/logseq_sync/test/transport_contract.ml deleted file mode 100644 index af178aa3..00000000 --- a/logseq_sync/test/transport_contract.ml +++ /dev/null @@ -1,4935 +0,0 @@ -module Core = Logseq_sync_pure_reducer.Core -module Runner = Logseq_sync_effect_runner.Effect_runner - -let with_support = Runner_contract.with_support -let secrets = Runner_contract.secrets -let crypto = Runner_contract.crypto - -(* The peer uses Python's independent TLS/framing implementation. Tests enter - production transports only through the public Runner API. *) -let peer = - {python| -import base64, hashlib, json, os, select, socket, ssl, struct, subprocess, sys, time, threading -root = sys.argv[1] -with open(root+'/cases.json') as f: cases = json.load(f) -key, cert = root + '/key.pem', root + '/cert.pem' -subprocess.run(['openssl','req','-x509','-newkey','ec','-pkeyopt','ec_paramgen_curve:P-256', - '-nodes','-keyout',key,'-out',cert,'-days','1','-subj','/CN=localhost'], - check=True, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL) -ctx = ssl.SSLContext(ssl.PROTOCOL_TLS_SERVER) -ctx.load_cert_chain(cert,key) -for attempt in range(16): - listener = socket.socket(socket.AF_INET6) - listener.setsockopt(socket.IPPROTO_IPV6, socket.IPV6_V6ONLY, 1) - listener4 = socket.socket() - try: - listener.bind(('::1',0)) - listener4.bind(('127.0.0.1', listener.getsockname()[1])) - break - except OSError: - listener.close(); listener4.close() - if attempt == 15: raise -listener.listen(8); listener.settimeout(5); listener4.listen(8) -with open(root+'/ready.tmp','w') as f: f.write(str(listener.getsockname()[1])) -os.replace(root+'/ready.tmp', root+'/ready') -report=[]; stalled=[] -def observe_stalled_tls(raw, item, index): - try: - hello=b'' - while True: - part=raw.recv(16384) - if not part: - item['closed']=True; break - if not item['client_hello']: - hello+=part - if len(hello)>5: - record_length=int.from_bytes(hello[3:5],'big') - if len(hello)>=5+record_length and hello[0]==22 and hello[5]==1: - item['client_hello']=True - path=root+'/client-hello-'+str(index) - with open(path+'.tmp','w') as f: f.write('ClientHello observed') - os.replace(path+'.tmp',path) - except (ConnectionResetError, BrokenPipeError): item['closed']=True - except socket.timeout: pass - finally: raw.close() -try: - for index,case in enumerate(cases): - ready,_,_=select.select([listener,listener4],[],[],5) - if not ready: raise RuntimeError('no client connection') - raw,_ = ready[0].accept() - raw.settimeout(2) - if case.get('stall_tls'): - item={'closed':False,'client_hello':False}; report.append(item) - worker=threading.Thread(target=observe_stalled_tls,args=(raw,item,index),daemon=True) - worker.start(); stalled.append(worker); continue - time.sleep(case.get('tls_delay',0)) - try: conn=ctx.wrap_socket(raw,server_side=True) - except (ssl.SSLError,OSError): - raw.close(); report.append({'closed':True,'frames':[]}); continue - request=b'' - while b'\r\n\r\n' not in request: - part=conn.recv(16384) - if not part: raise RuntimeError('no request') - request+=part - header_end=request.index(b'\r\n\r\n')+4 - request_headers=dict(line.split(b':',1) for line in request[:header_end].split(b'\r\n')[1:] if b':' in line) - length=next((int(v.strip()) for k,v in request_headers.items() if k.lower()==b'content-length'),0) - while len(request)=2: - length=pending[1]&127; offset=2 - if length==126: - if len(pending)<4: break - length=int.from_bytes(pending[2:4],'big'); offset=4 - elif length==127: - if len(pending)<10: break - length=int.from_bytes(pending[2:10],'big'); offset=10 - masked=bool(pending[1]&128) - if len(pending) Seq.map (fun c -> Printf.sprintf "%02x" (Char.code c)) - |> List.of_seq - |> String.concat "" -;; - -let case ?(ws = false) ?(extra = "") ?(segment = 16384) ?(eof = false) wire = - `Assoc - [ "wire", `String (hex wire) - ; "ws", `Bool ws - ; "extra", `String extra - ; "segment", `Int segment - ; "eof", `Bool eof - ] -;; - -let wait clock predicate = - Eio.Time.with_timeout_exn clock 5. (fun () -> - while not (predicate ()) do - Eio.Time.sleep clock 0.005 - done) -;; - -let read_file path = - let input = open_in_bin path in - Fun.protect - ~finally:(fun () -> close_in_noerr input) - (fun () -> really_input_string input (in_channel_length input)) -;; - -let scaled_clock clock scale = - let module Clock = struct - type t = unit - type time = float - - let now () = Eio.Time.now clock *. scale - let sleep_until () deadline = Eio.Time.sleep_until clock (deadline /. scale) - end - in - Eio.Resource.T ((), Eio.Time.Pi.clock (module Clock)) -;; - -let with_peer cases f = - with_support (fun support -> - Eio_main.run (fun environment -> - Eio.Switch.run (fun sw -> - let clock = Eio.Stdenv.clock environment in - Yojson.Basic.to_file (Filename.concat support "cases.json") (`List cases); - let process = - Eio.Process.spawn - ~sw - (Eio.Stdenv.process_mgr environment) - [ "python3"; "-c"; peer; support ] - in - wait clock (fun () -> Sys.file_exists (Filename.concat support "ready")); - let port = int_of_string (read_file (Filename.concat support "ready")) in - let origin = Uri.of_string (Printf.sprintf "https://localhost:%d" port) in - f ~environment ~sw ~support ~origin; - Eio.Time.with_timeout_exn clock 5. (fun () -> Eio.Process.await_exn process); - Yojson.Basic.from_file (Filename.concat support "report")))) -;; - -let graph = { Core_contract.graph with name = "Transport fixture" } - -let bootstrap ?(mirror_absent = true) ?(user_id = "fixture") origin = - let initial = - Core.config ~managed_sync_origin:origin ~limits:(Core_contract.limits ()) - |> Result.get_ok - |> Core.initial - |> Result.get_ok - |> fun core -> (Core.step core (Core.Sign_in_requested {account_generation=0;presentation_generation=0})).next - in - let auth = Core.step initial (Core.Account_authenticated { user_id = Some user_id }) in - let catalog = - List.find_map - (function - | Core.Run (Core.Request (ticket, Core.Fetch_catalog _)) -> - Some - (Core.step - auth.next - (Core.Runner_completed (Core.Completion (ticket, Ok [ graph ])))) - | _ -> None) - auth.effects - |> Option.get - in - let selected = Core.step catalog.next (Core.Graph_selected graph.graph_id) in - let scope = - List.find_map - (function - | Core.Delegate (Core.Inspect_mirror request) -> Some request.scope - | _ -> None) - selected.effects - |> Option.get - in - ( (if mirror_absent - then Core.step selected.next (Core.Mirror_inspected (Core.Mirror_absent scope)) - else selected) - , scope ) -;; - -let operation ?artifact_uri ~download origin = - let transition, _scope = bootstrap origin in - if not download - then - List.find_map - (function - | Core.Run (Core.Request (_, Core.Fetch_snapshot_metadata _) as op) -> Some op - | _ -> None) - transition.effects - |> Option.get - else ( - let artifact_uri = - Option.value - artifact_uri - ~default:(Uri.with_path origin "/artifact" |> Uri.to_string) - in - let response = - Yojson.Basic.to_string (`Assoc [ "ok", `Bool true; "t", `Int 42; "url", `String artifact_uri ]) - in - let downloading = - List.find_map - (function - | Core.Run (Core.Request (ticket, Core.Fetch_snapshot_metadata _)) -> - Some - (Core.step - transition.next - (Core.Runner_completed (Core.Completion (ticket, Ok response)))) - | _ -> None) - transition.effects - |> Option.get - in - List.find_map - (function - | Core.Run (Core.Request (_, Core.Download_snapshot _) as runnable) -> - Some runnable - | _ -> None) - downloading.effects - |> Option.get) -;; - -let websocket_cores = ref [] - -let websocket_start ?(checkpoint = 0) origin = - let selected, scope = bootstrap ~mirror_absent:false origin in - let inspected = - Core.step - selected.next - (Core.Mirror_inspected (Core.Mirror_available { graph; scope })) - in - let sync = - Logseq_overlay_db.Types.sync_view - ~token: - (Logseq_overlay_db.Types.sync_token_of_string "sync-token:v1:transport" - |> Result.get_ok) - ~checkpoint: - (Logseq_overlay_db.Types.Server_cursor.of_int64 (Int64.of_int checkpoint) - |> Result.get_ok) - ~submissions:[] - in - let attached = Core.step inspected.next (Core.Graph_attached { scope; sync }) in - let runnable, connection = - List.find_map - (function - | Core.Run (Core.Start_websocket request as runnable) -> - Some (runnable, request.scope) - | _ -> None) - attached.effects - |> Option.get - in - websocket_cores := (connection, attached.next) :: !websocket_cores; - runnable -;; - -let websocket_core (scope : Core.connection_scope) = List.assoc scope !websocket_cores - -let websocket_close (scope : Core.connection_scope) = - let opened = Core.step (websocket_core scope) (Core.Websocket_opened scope) in - Core.step - opened.next - (Core.Foreground_changed - { foreground = false - ; lifecycle_generation = scope.graph.account.lifecycle_generation - }) - |> fun transition -> - List.find_map - (function - | Core.Run (Core.Close_websocket _ as runnable) -> Some runnable - | _ -> None) - transition.effects - |> Option.get -;; - -let websocket_send (scope : Core.connection_scope) checkpoint = - let _ = websocket_start ~checkpoint scope.Core.graph.account.managed_sync_origin in - Core.step (websocket_core scope) (Core.Websocket_opened scope) - |> fun transition -> - List.find_map - (function - | Core.Run (Core.Send_websocket _ as runnable) -> Some runnable - | _ -> None) - transition.effects - |> Option.get -;; - -let cancellation_effect ?(network_only = false) scope = - let core = - Core.config - ~managed_sync_origin:(Uri.of_string "https://localhost") - ~limits:(Core_contract.limits ()) - |> Result.get_ok - |> Core.initial - |> Result.get_ok - |> fun core -> (Core.step core (Core.Sign_in_requested {account_generation=0;presentation_generation=0})).next - in - let core = - (Core.step core (Core.Account_authenticated { user_id = Some "fixture" })).next - in - let transition = if network_only then ( - let startup = (Core.state core).snapshot.startup in - Core.step core (Core.Online_cancelled {account_generation=startup.account_generation; - presentation_generation=startup.presentation_generation})) - else Core.step core Core.Shutdown in - ignore scope; - List.find_map - (function - | Core.Run ((Core.Cancel_effects _ | Core.Cancel_network_effects _) as runnable) -> Some runnable - | _ -> None) - transition.effects - |> Option.get -;; - -(* Keep the fixture's real reducer owner across sequential commands. Each accepted - completion consumes a ticket; starting from its old snapshot would replay it. *) -let asset_fixture_owners = ref [] - -let asset_owner t = - match List.find_opt (fun (runner, _) -> runner == t) !asset_fixture_owners with - | Some (_, scopes) -> scopes - | None -> - let scopes = Hashtbl.create 8 in - asset_fixture_owners := (t, scopes) :: !asset_fixture_owners; - scopes -;; - -let asset_current_core t core scope = - if Core.admitted_graph_scope core <> Some scope - then core - else Option.value (Hashtbl.find_opt (asset_owner t) scope) ~default:core -;; - -let asset_remember_core t scope core = Hashtbl.replace (asset_owner t) scope core - -let asset_submit t core scope operation action = - let core = asset_current_core t core scope in - let transition = Core.step core (Core.Asset_requested { scope; operation; action }) in - List.iter - (function - | Core.Run runnable -> Runner.submit t runnable - | _ -> ()) - transition.effects; - asset_remember_core t scope transition.next; - transition.next -;; - -let asset_collect t core posted = - let rec collect core outputs = function - | [] -> - Option.iter - (fun scope -> asset_remember_core t scope core) - (Core.admitted_graph_scope core); - core, List.rev outputs - | event :: events -> - let transition = Core.step core event in - let outputs = - List.fold_left - (fun outputs -> function - | Core.Publish (Core.Asset_finished result) -> result :: outputs - | Core.Run runnable -> - Runner.submit t runnable; - outputs - | _ -> outputs) - outputs - transition.effects - in - collect transition.next outputs events - in - collect core [] posted -;; - -let asset_run ?(with_cause = false) t ~clock ~posted core scope operation action = - let core = asset_current_core t core scope in - posted := []; - let transition = Core.step core (Core.Asset_requested { scope; operation; action }) in - let immediate = - List.filter_map - (function - | Core.Publish (Core.Asset_finished output) -> Some output - | _ -> None) - transition.effects - in - List.iter - (function - | Core.Run runnable -> Runner.submit t runnable - | _ -> ()) - transition.effects; - let outputs = - if immediate <> [] - then ( - asset_remember_core t scope transition.next; - immediate) - else ( - wait clock (fun () -> !posted <> []); - let completed, outputs = asset_collect t transition.next !posted in - asset_remember_core t scope completed; - outputs) - in - match outputs with - | [ output ] -> if with_cause then output.Core.result - else Result.map_error Core.asset_failure_kind output.Core.result - | _ -> Alcotest.fail "asset request did not publish exactly one typed outcome" -;; - -let asset_stage t ~clock ~posted core scope source_file = - match - asset_run - t - ~clock - ~posted - core - scope - "stage" - (Core.Stage_asset_file - { operation = graph.graph_id; file_type = "bin"; source_file }) - with - | Ok (Core.Asset_staged { file; _ }) -> file - | _ -> Alcotest.fail "source staging failed" -;; - -let runner - ?(websocket_liveness = Runner.Disabled) - ?network - ?clock - ?(tls_authenticator = Runner.tls_authenticator (fun ?ip:_ ~host:_ _ -> Ok None)) - ?secrets_dependency - ?crypto_dependency - ?(acquire = fun _ -> Ok "fixture-token") - ?(on_invalidate = fun _ -> ()) - ?(on_post = fun _ -> ()) - ?on_asset_diagnostic - ?asset_cache_budget_bytes - ?asset_maximum_file_bytes - ?asset_pending_budget_bytes - ~environment - ~sw - ~support - ~posted - ~invalidations - () - = - let transport = - Runner.transport - ~websocket_liveness - ~tls_authenticator - ~network:(Option.value network ~default:(Eio.Stdenv.net environment)) - ~clock:(Option.value clock ~default:(Eio.Stdenv.clock environment)) - |> Result.get_ok - in - let dependencies = - Runner.dependencies - ~runtime: - (Runner.runtime - ~fork:Eio.Fiber.fork - ~sleep:(Eio.Time.sleep (Eio.Stdenv.clock environment)) - |> Result.get_ok) - ~transport - ~local_store: - (Runner.local_store - ?asset_cache_budget_bytes - ?asset_maximum_file_bytes - ?asset_pending_budget_bytes - ~application_support_directory:support - () - |> Result.get_ok) - ~artifact_store: - (Runner.artifact_store ~staging_directory:(Filename.concat support "staging") - |> Result.get_ok) - ~secrets:(Option.value secrets_dependency ~default:(secrets ())) - ~crypto:(Option.value crypto_dependency ~default:(crypto ())) - ~id_token_provider: - (Runner.id_token_provider ~acquire ~invalidate:(fun _ ~token -> - incr invalidations; - on_invalidate token)) - |> Result.get_ok - in - Runner.create ?on_asset_diagnostic ~sw dependencies ~post:(fun event -> - on_post event; - posted := !posted @ [ event ]) - |> Result.get_ok -;; - -let completed events = - List.find_map - (function - | Core.Runner_completed (Core.Completion (_, result)) -> - Some (Result.map (fun _ -> ()) result) - | _ -> None) - events -;; - -let response ?(status = 200) ?(headers = "Content-Length: 1\r\n") body = - Printf.sprintf - "HTTP/1.1 %d Test\r\n\ - Content-Type: application/transit+json\r\n\ - X-Snapshot-Row-Count: 1\r\n\ - %s\r\n\ - %s" - status - headers - body -;; - -let check_retired report = - let rows = Yojson.Basic.Util.to_list report in - List.iter - (fun row -> - Alcotest.(check bool) - "peer observed local resource retirement" - true - Yojson.Basic.Util.(row |> member "closed" |> to_bool)) - rows -;; - -let captured_request_headers row = - let encoded = Yojson.Basic.Util.(row |> member "request" |> to_string) in - let request = - String.init (String.length encoded / 2) (fun n -> - Char.chr (int_of_string ("0x" ^ String.sub encoded (2 * n) 2))) - in - let rec headers = function - | [] | "" :: _ -> [] - | line :: rest -> line :: headers rest - in - String.split_on_char '\n' request - |> List.tl - |> List.map (fun line -> String.lowercase_ascii (String.trim line)) - |> headers -;; - -let check_default_connection report = - Yojson.Basic.Util.to_list report - |> List.iter (fun row -> - Alcotest.(check bool) - "ordinary request omits Connection" - false - (List.exists - (String.starts_with ~prefix:"connection:") - (captured_request_headers row))) -;; - -let http_test - ~download - ~success - ?(payload = "x") - ?retained_download - ?(segment = 16384) - ?(eof = false) - ?failure_message - wire - () - = - let report = - with_peer - [ case ~segment ~eof wire ] - (fun ~environment ~sw ~support ~origin -> - let posted = ref [] - and invalidations = ref 0 in - let t = runner ~environment ~sw ~support ~posted ~invalidations () in - Runner.submit t (operation ~download origin); - wait (Eio.Stdenv.clock environment) (fun () -> - Option.is_some (completed !posted)); - let result = Option.get (completed !posted) in - (match result with - | Error (Core.Effect_failed message | Core.Crypto_failed (_, message)) -> - prerr_endline message - | Ok () -> ()); - Alcotest.(check bool) "response outcome" success (Result.is_ok result); - Option.iter (fun expected -> match result with - | Error (Core.Effect_failed actual) -> - Alcotest.(check string) "finite decoder cause" expected actual - | _ -> Alcotest.fail "expected decoder failure") failure_message; - if download - then ( - let staging = Filename.concat support "staging" in - let names = Sys.readdir staging |> Array.to_list in - if success - then ( - Alcotest.(check int) "one artifact published" 1 (List.length names); - Alcotest.(check string) - "exact artifact payload" - payload - (read_file (Filename.concat staging (List.hd names)))) - else ( - match retained_download with - | None -> Alcotest.(check (list string)) "no partial artifact" [] names - | Some body -> - Alcotest.(check int) - "only completed download remains" - 1 - (List.length names); - let name = List.hd names in - Alcotest.(check bool) - "no decompressed artifact or staging file published" - true - (Filename.check_suffix name ".download"); - Alcotest.(check string) - "completed download is intact" - body - (read_file (Filename.concat staging name)))); - Runner.shutdown t) - in - check_retired report; - check_default_connection report -;; - -let frame ?(fin = true) opcode payload = - let size = String.length payload in - let prefix = - if size < 126 - then String.make 1 (Char.chr size) - else if size <= 65535 - then - String.init 3 (function - | 0 -> Char.chr 126 - | 1 -> Char.chr (size lsr 8) - | _ -> Char.chr (size land 255)) - else - String.init 9 (function - | 0 -> Char.chr 127 - | i -> - Char.chr - (Int64.to_int - (Int64.logand - 255L - (Int64.shift_right_logical (Int64.of_int size) ((8 - i) * 8))))) - in - String.make 1 (Char.chr (opcode lor if fin then 128 else 0)) ^ prefix ^ payload -;; - -let ws_test ?(extra = "") ?(segment = 16384) ~messages ~terminal wire () = - let report = - with_peer - [ case ~ws:true ~extra ~segment wire ] - (fun ~environment ~sw ~support ~origin -> - let posted = ref [] - and invalidations = ref 0 in - let t = runner ~environment ~sw ~support ~posted ~invalidations () in - let _, graph = bootstrap origin in - let scope = Core.{ graph; connection_generation = 1 } in - Runner.submit t (websocket_start origin); - let clock = Eio.Stdenv.clock environment in - let message_count () = - List.length - (List.filter - (function - | Core.Websocket_message _ -> true - | _ -> false) - !posted) - in - let closed () = - List.exists - (function - | Core.Websocket_closed _ -> true - | _ -> false) - !posted - in - wait clock (fun () -> - if terminal then closed () else message_count () >= messages); - Alcotest.(check int) - "ordered application message count" - messages - (message_count ()); - let rec ordered opened = function - | [] -> () - | Core.Websocket_opened _ :: rest -> ordered true rest - | Core.Websocket_message _ :: rest -> - Alcotest.(check bool) "open precedes data" true opened; - ordered opened rest - | _ :: rest -> ordered opened rest - in - ordered false !posted; - if not terminal - then ( - Runner.submit t (websocket_close scope); - wait clock closed); - Alcotest.(check int) - "one terminal notification" - 1 - (List.length - (List.filter - (function - | Core.Websocket_closed _ -> true - | _ -> false) - !posted)); - Runner.shutdown t) - in - check_retired report; - Yojson.Basic.Util.to_list report - |> List.iter (fun row -> - let headers = captured_request_headers row in - Alcotest.(check bool) "WebSocket keeps Connection Upgrade" true - (List.mem "connection: upgrade" headers); - Alcotest.(check bool) "WebSocket keeps Upgrade websocket" true - (List.mem "upgrade: websocket" headers)); - if not terminal - then ( - let frames = - Yojson.Basic.Util.(report |> to_list |> List.hd |> member "frames" |> to_list) - in - Alcotest.(check bool) - "peer received masked Close" - true - (List.exists - (fun frame -> - Yojson.Basic.Util.(frame |> member "opcode" |> to_int) = 8 - && String.length Yojson.Basic.Util.(frame |> member "mask" |> to_string) = 8) - frames)) -;; - -(* T12-T18 are accepted dependency limitations. These cases exercise supported - framing and wrapper cleanup, not a replacement protocol conformance suite. *) -let http_cases = - [ "complete length", true, false, response "x" - ; "truncated length", false, true, response ~headers:"Content-Length: 2\r\n" "x" - ; "close delimited", true, true, response ~headers:"" "x" - ; ( "plain chunked" - , true - , false - , response ~headers:"Transfer-Encoding: chunked\r\n" "1\r\nx\r\n0\r\n\r\n" ) - ; ( "truncated chunk" - , false - , true - , response ~headers:"Transfer-Encoding: chunked\r\n" "2\r\nx" ) - ; ( "malformed chunk after payload preserves failure" - , false - , false - , response ~headers:"Transfer-Encoding: chunked\r\n" "1\r\nx\r\nZ\r\n" ) - ; ( "bad content type" - , false - , false - , "HTTP/1.1 200 OK\r\nContent-Type: text/html\r\nContent-Length: 1\r\n\r\nx" ) - ; "forbidden", false, false, response ~status:403 "x" - ] -;; - -let ws_cases = - let message = {|{"type":"changed","t":42}|} in - [ ( "coalesced fragmented message" - , 1 - , false - , frame ~fin:false 1 (String.sub message 0 3) - ^ frame 0 (String.sub message 3 (String.length message - 3)) ) - ; "ordered multiple messages", 2, false, frame 1 message ^ frame 1 message - ; ( "interleaved ping" - , 1 - , false - , frame ~fin:false 1 (String.sub message 0 3) - ^ frame 9 "probe" - ^ frame 0 (String.sub message 3 (String.length message - 3)) ) - ; "peer close", 0, true, frame 8 "\x03\xe8" - ] -;; - -let scenarios = - List.concat_map - (fun download -> - List.map - (fun (name, success, eof, wire) -> - Alcotest.test_case - ((if download then "download " else "HTTP ") ^ name) - `Quick - (http_test ~download ~success ~eof wire)) - http_cases) - [ false; true ] - @ List.map - (fun (name, messages, terminal, wire) -> - Alcotest.test_case - ("WebSocket " ^ name) - `Quick - (ws_test ~messages ~terminal wire)) - ws_cases - @ [ Alcotest.test_case - "HTTP segmented supported chunked response" - `Quick - (http_test - ~download:false - ~success:true - ~segment:1 - (response ~headers:"Transfer-Encoding: chunked\r\n" "1\r\nx\r\n0\r\n\r\n")) - ; Alcotest.test_case - "WebSocket segmented Upgrade and frame" - `Quick - (ws_test - ~messages:1 - ~terminal:false - ~segment:1 - (frame 1 {|{"type":"changed","t":42}|})) - ; Alcotest.test_case - "WebSocket bounds retained incomplete Upgrade input" - `Quick - (ws_test - ~messages:0 - ~terminal:true - ~extra:("X-Incomplete: " ^ String.make 20000 'x') - "") - ] -;; - -let test_cancelled_attempt ~download () = - let report = - with_peer - [ case (response ~headers:"Content-Length: 100\r\n" "x") ] - (fun ~environment ~sw ~support ~origin -> - let posted = ref [] - and invalidations = ref 0 in - let t = runner ~environment ~sw ~support ~posted ~invalidations () in - let op = operation ~download origin in - let raw = - match op with - | Core.Request (ticket, _) -> - Filename.concat - (Filename.concat support "staging") - ("snapshot-" - ^ Core.effect_id_to_string (Core.effect_ticket_id ticket) - ^ ".download") - | _ -> assert false - in - if download - then ( - Unix.mkdir (Filename.dirname raw) 0o700; - let output = open_out_bin raw in - output_string output "committed"; - close_out output); - Runner.submit t op; - let clock = Eio.Stdenv.clock environment in - wait clock (fun () -> Sys.file_exists (Filename.concat support "sent")); - Runner.submit t (cancellation_effect (Core.runner_effect_scope op)); - Eio.Time.sleep clock 0.05; - let event_count = List.length !posted in - Eio.Time.sleep clock 0.05; - Alcotest.(check int) - "no callbacks after cancellation" - event_count - (List.length !posted); - Alcotest.(check bool) - "cancelled operation never completes" - true - (completed !posted = None); - if download - then ( - Alcotest.(check bool) - "committed destination preserved" - true - (Sys.file_exists raw); - Alcotest.(check string) "committed bytes preserved" "committed" (read_file raw); - Alcotest.(check int) - "no attempt temporary remains" - 1 - (Array.length (Sys.readdir (Filename.dirname raw)))); - Runner.shutdown t) - in - check_retired report -;; - -let test_http_retry ~download () = - let report = - with_peer - [ case (response ~status:401 "x"); case (response "x") ] - (fun ~environment ~sw ~support ~origin -> - let posted = ref [] - and invalidations = ref 0 in - let t = runner ~environment ~sw ~support ~posted ~invalidations () in - Runner.submit t (operation ~download origin); - wait (Eio.Stdenv.clock environment) (fun () -> - Option.is_some (completed !posted)); - Alcotest.(check bool) - "retry succeeds" - true - (Result.is_ok (Option.get (completed !posted))); - Alcotest.(check int) "one token invalidation" 1 !invalidations; - if download - then - Alcotest.(check int) - "one validated artifact" - 1 - (Array.length (Sys.readdir (Filename.concat support "staging"))); - Runner.shutdown t) - in - check_retired report -;; - -let test_redirect ~download ~cross_origin () = - let redirect = - response - ~status:302 - ~headers: - ((if cross_origin - then "Location: https://example.invalid/artifact\r\n" - else "Location: /follow\r\n") - ^ "Content-Length: 1\r\n") - "x" - in - let cases = - if cross_origin then [ case redirect ] else [ case redirect; case (response "x") ] - in - let report = - with_peer cases (fun ~environment ~sw ~support ~origin -> - let posted = ref [] - and invalidations = ref 0 in - let t = runner ~environment ~sw ~support ~posted ~invalidations () in - Runner.submit t (operation ~download origin); - wait (Eio.Stdenv.clock environment) (fun () -> Option.is_some (completed !posted)); - Alcotest.(check bool) - "redirect origin policy" - (not cross_origin) - (Result.is_ok (Option.get (completed !posted))); - if cross_origin && download - then - Alcotest.(check int) - "no redirect artifact" - 0 - (Array.length (Sys.readdir (Filename.concat support "staging"))); - Runner.shutdown t) - in - check_retired report -;; - -let test_ws_cancellation ~during_close () = - let report = - with_peer - [ case ~ws:true "" ] - (fun ~environment ~sw ~support ~origin -> - let posted = ref [] - and invalidations = ref 0 in - let t = runner ~environment ~sw ~support ~posted ~invalidations () in - let _, graph = bootstrap origin in - let scope = Core.{ graph; connection_generation = 1 } in - Runner.submit t (websocket_start origin); - let clock = Eio.Stdenv.clock environment in - wait clock (fun () -> - List.exists - (function - | Core.Websocket_opened _ -> true - | _ -> false) - !posted); - if during_close then Runner.submit t (websocket_close scope); - let start = Eio.Time.now clock in - Runner.submit t (cancellation_effect (Core.effect_scope_of_graph graph)); - Runner.shutdown t; - Alcotest.(check bool) - "abort dispatch does not await grace" - true - (Eio.Time.now clock -. start < 0.05)) - in - check_retired report -;; - -let scenarios = - scenarios - @ List.concat_map - (fun download -> - let label = if download then "download " else "HTTP " in - [ Alcotest.test_case - (label ^ "cancel retires attempt and preserves committed output") - `Quick - (test_cancelled_attempt ~download) - ; Alcotest.test_case - (label ^ "401 refresh and retry") - `Quick - (test_http_retry ~download) - ; Alcotest.test_case - (label ^ "same-origin redirect") - `Quick - (test_redirect ~download ~cross_origin:false) - ; Alcotest.test_case - (label ^ "cross-origin redirect rejected") - `Quick - (test_redirect ~download ~cross_origin:true) - ]) - [ false; true ] - @ [ Alcotest.test_case - "WebSocket cancellation aborts owner" - `Quick - (test_ws_cancellation ~during_close:false) - ; Alcotest.test_case - "WebSocket abort interrupts graceful close" - `Quick - (test_ws_cancellation ~during_close:true) - ] - @ List.map - (fun size -> - let message = {|{"type":"changed","t":42}|} in - let wire = String.make (size - String.length message) ' ' ^ message in - Alcotest.test_case - (Printf.sprintf "WebSocket minimal length boundary %d" size) - `Quick - (ws_test ~messages:1 ~terminal:false (frame 1 wire))) - [ 125; 126; 65535; 65536 ] - @ [ Alcotest.test_case - "WebSocket UTF8 codepoint split across fragments" - `Quick - (ws_test - ~messages:1 - ~terminal:false - (frame ~fin:false 1 "{\"type\":\"error\",\"message\":\"\xe4" - ^ frame 9 "probe" - ^ frame 0 "\xb8\xad\"}")) - ] -;; - -let test_artifact_credentials ~status () = - let report = - with_peer - [ case (response ~status "x") ] - (fun ~environment ~sw ~support ~origin -> - let posted = ref [] - and invalidations = ref 0 in - let t = runner ~environment ~sw ~support ~posted ~invalidations () in - let op = - operation - ~download:true - ~artifact_uri: - (Uri.with_path origin "/signed-artifact?signature=fixture" |> Uri.to_string) - (Uri.with_port origin (Some 1)) - in - Runner.submit t op; - wait (Eio.Stdenv.clock environment) (fun () -> - Option.is_some (completed !posted)); - Alcotest.(check bool) - "signed artifact status preserved" - (status = 200) - (Result.is_ok (Option.get (completed !posted))); - Alcotest.(check int) "signed URL never invalidates bearer token" 0 !invalidations; - Runner.shutdown t) - in - check_retired report; - let wire = - Yojson.Basic.Util.(report |> to_list |> List.hd |> member "request" |> to_string) - in - let decoded = - String.init - (String.length wire / 2) - (fun i -> Char.chr (int_of_string ("0x" ^ String.sub wire (i * 2) 2))) - in - Alcotest.(check bool) - "cross-origin request omits authorization" - false - (String.split_on_char '\n' decoded - |> List.exists (fun line -> - String.starts_with ~prefix:"authorization:" (String.lowercase_ascii line))) -;; - -let scenarios = - scenarios - @ [ Alcotest.test_case - "signed artifact omits account bearer" - `Quick - (test_artifact_credentials ~status:200) - ; Alcotest.test_case - "signed artifact 401 never refreshes account bearer" - `Quick - (test_artifact_credentials ~status:401) - ] -;; - -let test_liveness ~reply () = - let config = - match case ~ws:true "" with - | `Assoc fields -> `Assoc (("reply_ping", `Bool reply) :: fields) - | _ -> assert false - in - let report = - with_peer [ config ] (fun ~environment ~sw ~support ~origin -> - let posted = ref [] - and invalidations = ref 0 in - let t = - runner - ~websocket_liveness: - (Runner.Ping_pong { interval_seconds = 0.02; timeout_seconds = 0.03 }) - ~environment - ~sw - ~support - ~posted - ~invalidations - () - in - let _, graph = bootstrap origin in - let scope = Core.{ graph; connection_generation = 1 } in - Runner.submit t (websocket_start origin); - let clock = Eio.Stdenv.clock environment in - wait clock (fun () -> - List.exists - (function - | Core.Websocket_opened _ -> true - | _ -> false) - !posted); - Eio.Time.sleep clock 0.15; - let closed () = - List.exists - (function - | Core.Websocket_closed _ -> true - | _ -> false) - !posted - in - Alcotest.(check bool) "correlated Pong determines liveness" (not reply) (closed ()); - if reply - then ( - Runner.submit t (websocket_close scope); - wait clock closed); - Runner.shutdown t) - in - check_retired report; - let frames = - Yojson.Basic.Util.(report |> to_list |> List.hd |> member "frames" |> to_list) - in - Alcotest.(check bool) - "peer observes protocol Ping" - true - (List.exists - (fun frame -> Yojson.Basic.Util.(frame |> member "opcode" |> to_int) = 9) - frames) -;; - -let test_control_output () = - let config = - match case ~ws:true (frame 9 "server-probe") with - | `Assoc fields -> `Assoc (("reply_ping", `Bool true) :: fields) - | _ -> assert false - in - let report = - with_peer [ config ] (fun ~environment ~sw ~support ~origin -> - let posted = ref [] - and invalidations = ref 0 in - let t = - runner - ~websocket_liveness: - (Runner.Ping_pong { interval_seconds = 0.03; timeout_seconds = 0.1 }) - ~environment - ~sw - ~support - ~posted - ~invalidations - () - in - let _, graph = bootstrap origin in - let scope = Core.{ graph; connection_generation = 1 } in - Runner.submit t (websocket_start origin); - let clock = Eio.Stdenv.clock environment in - wait clock (fun () -> - List.exists - (function - | Core.Websocket_opened _ -> true - | _ -> false) - !posted); - Runner.submit t (websocket_send scope 0); - Eio.Time.sleep clock 0.15; - Runner.submit t (websocket_close scope); - wait clock (fun () -> - List.exists - (function - | Core.Websocket_closed _ -> true - | _ -> false) - !posted); - Runner.shutdown t) - in - let frames = - Yojson.Basic.Util.(report |> to_list |> List.hd |> member "frames" |> to_list) - in - List.iter - (fun opcode -> - Alcotest.(check bool) - (Printf.sprintf "opcode %d serialized" opcode) - true - (List.exists - (fun frame -> Yojson.Basic.Util.(frame |> member "opcode" |> to_int) = opcode) - frames)) - [ 1; 8; 9; 10 ]; - List.iter - (fun frame -> - Alcotest.(check int) - "client frame has a mask" - 8 - (String.length Yojson.Basic.Util.(frame |> member "mask" |> to_string))) - frames; - check_retired report -;; - -let test_invalid_liveness () = - Eio_main.run (fun env -> - List.iter - (fun seconds -> - let result = - Runner.transport - ~tls_authenticator: - (Runner.tls_authenticator (fun ?ip:_ ~host:_ _ -> Ok None)) - ~network:(Eio.Stdenv.net env) - ~clock:(Eio.Stdenv.clock env) - ~websocket_liveness: - (Runner.Ping_pong { interval_seconds = seconds; timeout_seconds = seconds }) - in - Alcotest.(check bool) - "invalid liveness is rejected" - true - (Result.is_error result)) - [ 0.; -1.; infinity; nan ]) -;; - -let scenarios = - scenarios - @ [ Alcotest.test_case - "WebSocket silent peer liveness deadline" - `Quick - (test_liveness ~reply:false) - ; Alcotest.test_case - "WebSocket matching Pong keeps connection alive" - `Quick - (test_liveness ~reply:true) - ; Alcotest.test_case - "WebSocket data and control frames reach peer" - `Quick - test_control_output - ; Alcotest.test_case - "WebSocket liveness configuration validation" - `Quick - test_invalid_liveness - ] -;; - -let test_output_admission () = - let report = - with_peer - [ case ~ws:true "" ] - (fun ~environment ~sw ~support ~origin -> - let posted = ref [] - and invalidations = ref 0 in - let t = runner ~environment ~sw ~support ~posted ~invalidations () in - let _, graph = bootstrap origin in - let scope = Core.{ graph; connection_generation = 1 } in - Runner.submit t (websocket_start origin); - let clock = Eio.Stdenv.clock environment in - wait clock (fun () -> - List.exists - (function - | Core.Websocket_opened _ -> true - | _ -> false) - !posted); - for index = 0 to 1999 do - Runner.submit t (websocket_send scope index) - done; - let count_path = Filename.concat support "frame-count" in - wait clock (fun () -> - Sys.file_exists count_path - && Option.value (int_of_string_opt (read_file count_path)) ~default:0 >= 128); - Runner.submit t (websocket_send scope 2000); - Runner.submit t (websocket_close scope); - wait clock (fun () -> - List.exists - (function - | Core.Websocket_closed _ -> true - | _ -> false) - !posted); - Runner.shutdown t) - in - let frames = - Yojson.Basic.Util.(report |> to_list |> List.hd |> member "frames" |> to_list) - in - let data = - List.filter - (fun frame -> Yojson.Basic.Util.(frame |> member "opcode" |> to_int) = 1) - frames - in - Alcotest.(check int) - "queue rejects overflow and releases completed budget" - 129 - (List.length data); - List.iteri - (fun index frame -> - let since = if index = 128 then 2000 else index in - let expected = - Logseq_sync_pure_reducer.Sync_protocol.encode_client_message - (Pull { since = Some (Core_contract.server_cursor_of_int since) }) - |> Result.get_ok - |> hex - in - Alcotest.(check string) - "accepted data retains order" - expected - Yojson.Basic.Util.(frame |> member "payload" |> to_string)) - data; - check_retired report -;; - -let scenarios = - scenarios - @ [ Alcotest.test_case - "WebSocket aggregate send admission and budget release" - `Quick - test_output_admission - ] -;; - -let test_setup_cancel ~websocket ~dns () = - let run ~environment ~sw ~support ~origin = - let posted = ref [] - and invalidations = ref 0 in - let t = runner ~environment ~sw ~support ~posted ~invalidations () in - let op = - if websocket then websocket_start origin else operation ~download:false origin - in - Runner.submit t op; - Eio.Time.sleep (Eio.Stdenv.clock environment) 0.03; - Runner.submit t (cancellation_effect (Core.runner_effect_scope op)); - Runner.shutdown t - in - let started = Unix.gettimeofday () in - if dns - then ( - let descriptors () = Array.length (Sys.readdir "/dev/fd") in - let before = descriptors () in - with_support (fun support -> - Eio_main.run (fun environment -> - Eio.Switch.run (fun sw -> - let host = - Printf.sprintf - "logseq-transport-%d-%d.local" - (Unix.getpid ()) - (Random.bits ()) - in - let origin = Uri.of_string ("https://" ^ host) in - run ~environment ~sw ~support ~origin))); - Alcotest.(check bool) - "DNS setup cancellation joins within bound" - true - (Unix.gettimeofday () -. started < 0.5); - Alcotest.(check bool) - "DNS cancellation releases owned descriptors" - true - (descriptors () <= before)) - else ( - let stalled = `Assoc [ "stall_tls", `Bool true ] in - let report=with_peer [stalled] (fun ~environment ~sw ~support ~origin-> - let posted=ref [] and invalidations=ref 0 in - let t=runner ~environment ~sw ~support ~posted ~invalidations () in - Fun.protect ~finally:(fun()->Runner.shutdown t)(fun()-> - let op=if websocket then websocket_start origin else operation ~download:false origin in - Runner.submit t op; - (* The peer has consumed a complete TLS record carrying ClientHello and - sends no handshake response. No process startup or polling guess is - part of the cancellation interval. *) - wait (Eio.Stdenv.clock environment)(fun()-> - Sys.file_exists(Filename.concat support "client-hello-0")); - Alcotest.(check bool) "TLS setup is still pending when cancellation starts" true - (if websocket then not(List.exists(function Core.Websocket_closed _->true|_->false)!posted) - else Option.is_none(completed !posted)); - let scope=Core.runner_effect_scope op in - let retired()=if websocket then - let connection=match op with Core.Start_websocket request->request.scope|_->assert false in - List.exists(function Core.Websocket_retired actual->actual=connection|_->false)!posted - else List.exists(function Core.IO_fence_completed(98001L,actual,Ok())->actual=scope|_->false)!posted in - let clock=Eio.Stdenv.mono_clock environment in - let began=Eio.Time.Mono.now clock in - let elapsed()=Mtime.Span.to_float_ns(Mtime.span began(Eio.Time.Mono.now clock))/.1e9 in - Runner.submit t(cancellation_effect scope); - if not websocket then ( - let initialized,_=bootstrap origin in - let fence=Core.step initialized.next(Core.IO_fence_requested(98001L,scope)) in - List.iter(function Core.Run instruction->Runner.submit t instruction|_->())fence.effects); - (try Eio.Time.Timeout.run_exn(Eio.Time.Timeout.seconds clock 1.)(fun()-> - while not(retired()) do Eio.Time.Mono.sleep clock 0.001 done) - with Eio.Time.Timeout->Alcotest.failf "%s TLS cancellation has no actual retirement after %.6fs" - (if websocket then "WebSocket" else "HTTP")(elapsed())); - Printf.printf "%s TLS cancel -> actual scope retirement: %.6fs\n%!" - (if websocket then "WebSocket" else "HTTP")(elapsed()); - Alcotest.(check bool) "TLS cancellation reaches actual retirement within bound" true(elapsed()<1.))) in - check_retired report; - List.iter(fun row->Alcotest.(check bool) "peer received ClientHello before cancellation" true - Yojson.Basic.Util.(row |> member "client_hello" |> to_bool))Yojson.Basic.Util.(to_list report)) -;; - -let test_alternate_address ~websocket () = - let config = if websocket then case ~ws:true "" else case (response "x") in - let report = - with_peer - [ `Assoc [ "stall_tls", `Bool true ]; config ] - (fun ~environment ~sw ~support ~origin -> - let posted = ref [] - and invalidations = ref 0 in - let clock = scaled_clock (Eio.Stdenv.clock environment) 10. in - let t = runner ~clock ~environment ~sw ~support ~posted ~invalidations () in - if websocket - then ( - let _, graph = bootstrap origin in - let scope = Core.{ graph; connection_generation = 1 } in - Runner.submit t (websocket_start origin); - wait (Eio.Stdenv.clock environment) (fun () -> - List.exists - (function - | Core.Websocket_opened _ | Core.Websocket_closed _ -> true - | _ -> false) - !posted); - Alcotest.(check bool) - "later address opens authenticated WebSocket" - true - (List.exists - (function - | Core.Websocket_opened _ -> true - | _ -> false) - !posted); - Runner.submit t (websocket_close scope); - wait (Eio.Stdenv.clock environment) (fun () -> - List.exists - (function - | Core.Websocket_closed _ -> true - | _ -> false) - !posted)) - else ( - Runner.submit t (operation ~download:false origin); - wait (Eio.Stdenv.clock environment) (fun () -> - Option.is_some (completed !posted)); - Alcotest.(check bool) - "later address completes HTTP" - true - (Result.is_ok (Option.get (completed !posted)))); - Runner.shutdown t) - in - check_retired report -;; - -let scenarios = - scenarios - @ List.concat_map - (fun websocket -> - let prefix = if websocket then "WebSocket " else "HTTP " in - [ Alcotest.test_case - (prefix ^ "DNS cancellation reclaims setup") - `Quick - (test_setup_cancel ~websocket ~dns:true) - ; Alcotest.test_case - (prefix ^ "TLS cancellation reclaims setup") - `Quick - (test_setup_cancel ~websocket ~dns:false) - ; Alcotest.test_case - (prefix ^ "tries alternate address after TLS failure") - `Quick - (test_alternate_address ~websocket) - ]) - [ false; true ] -;; - -let test_upgrade_authentication ~status () = - let cases = - [ case (response ~status "x") ] @ if status = 401 then [ case ~ws:true "" ] else [] - in - let report = - with_peer cases (fun ~environment ~sw ~support ~origin -> - let posted = ref [] - and invalidations = ref 0 in - let t = runner ~environment ~sw ~support ~posted ~invalidations () in - let _, graph = bootstrap origin in - let scope = Core.{ graph; connection_generation = 1 } in - Runner.submit t (websocket_start origin); - let clock = Eio.Stdenv.clock environment in - let closed () = - List.exists - (function - | Core.Websocket_closed _ -> true - | _ -> false) - !posted - in - let opened () = - List.exists - (function - | Core.Websocket_opened _ -> true - | _ -> false) - !posted - in - wait clock (fun () -> opened () || closed ()); - Alcotest.(check bool) - "only unauthorized Upgrade retries and opens" - (status = 401) - (opened ()); - Alcotest.(check int) - "Upgrade token invalidations" - (if status = 401 then 1 else 0) - !invalidations; - if opened () - then ( - Runner.submit t (websocket_close scope); - wait clock closed); - Runner.shutdown t) - in - check_retired report -;; - -let scenarios = - scenarios - @ List.map - (fun status -> - Alcotest.test_case - (Printf.sprintf "WebSocket Upgrade %d preserves authentication outcome" status) - `Quick - (test_upgrade_authentication ~status)) - [ 401; 403 ] -;; - -let delayed_network ?(destination = Fun.id) network before_connect = - let module Network = struct - type t = Eio_unix.Net.t * (unit -> unit) - - type tag = - [ `Generic - | `Unix - ] - - let listen (net, _) ~reuse_addr ~reuse_port ~backlog ~sw address = - Eio.Net.listen net ~reuse_addr ~reuse_port ~backlog ~sw address - ;; - - let connect (net, before) ~sw address = - before (); - Eio.Net.connect net ~sw (destination address) - ;; - - let datagram_socket (net, _) ~reuse_addr ~reuse_port ~sw address = - Eio.Net.datagram_socket net ~reuse_addr ~reuse_port ~sw address - ;; - - let getaddrinfo (net, _) ~service host = Eio.Net.getaddrinfo net ~service host - let getnameinfo (net, _) address = Eio.Net.getnameinfo net address - end - in - Eio.Resource.T ((network, before_connect), Eio.Net.Pi.network (module Network)) -;; - -let test_stalled_tcp ?(network_only = false) ~cancel () = - let report = - with_peer - (if cancel then [] else [ case (response "x") ]) - (fun ~environment ~sw ~support ~origin -> - let clock = Eio.Stdenv.clock environment in - let transport_clock = scaled_clock clock 10. in - let attempts = ref 0 - and released = ref false in - let network = - delayed_network (Eio.Stdenv.net environment) (fun () -> - incr attempts; - if !attempts = 1 - then - Fun.protect - ~finally:(fun () -> released := true) - (fun () -> Eio.Time.sleep transport_clock 30.)) - in - let posted = ref [] - and invalidations = ref 0 in - let t = - runner - ~network - ~clock:transport_clock - ~environment - ~sw - ~support - ~posted - ~invalidations - () - in - let op = operation ~download:false origin in - Runner.submit t op; - wait clock (fun () -> !attempts > 0); - let started = Eio.Time.now clock in - if cancel - then ( - Runner.submit t (cancellation_effect ~network_only (Core.runner_effect_scope op)); - wait clock (fun () -> !released); - Alcotest.(check bool) - "TCP cancellation is prompt" - true - (Eio.Time.now clock -. started < 0.5)) - else ( - wait clock (fun () -> Option.is_some (completed !posted)); - Alcotest.(check bool) - "blackholed first candidate does not starve the next" - true - (Result.is_ok (Option.get (completed !posted))); - Alcotest.(check bool) - "next candidate starts within bound" - true - (Eio.Time.now clock -. started < 2.); - Alcotest.(check int) "two address candidates" 2 !attempts); - Alcotest.(check bool) "first candidate work retired" true !released; - Runner.shutdown t) - in - check_retired report -;; - -let test_artifact_publication_failure () = - let report = - with_peer - [ case (response "x") ] - (fun ~environment ~sw ~support ~origin -> - let posted = ref [] - and invalidations = ref 0 in - let t = runner ~environment ~sw ~support ~posted ~invalidations () in - let op = operation ~download:true origin in - let id = - match op with - | Core.Request (ticket, _) -> - Core.effect_id_to_string (Core.effect_ticket_id ticket) - | _ -> assert false - in - let staging = Filename.concat support "staging" in - Unix.mkdir staging 0o700; - let destination = Filename.concat staging ("snapshot-" ^ id ^ ".download") in - Unix.mkdir destination 0o700; - Runner.submit t op; - wait (Eio.Stdenv.clock environment) (fun () -> - Option.is_some (completed !posted)); - Alcotest.(check bool) - "failed atomic publication never succeeds" - true - (Result.is_error (Option.get (completed !posted))); - Alcotest.(check bool) - "existing destination preserved" - true - (Sys.is_directory destination); - Alcotest.(check int) - "temporary removed after publication failure" - 1 - (Array.length (Sys.readdir staging)); - Runner.shutdown t) - in - check_retired report -;; - -let scenarios = - scenarios - @ [ Alcotest.test_case - "network-only cancellation aborts in-flight TCP setup" - `Quick - (test_stalled_tcp ~network_only:true ~cancel:true) - ; Alcotest.test_case - "HTTP stalled TCP cancellation" - `Quick - (test_stalled_tcp ~cancel:true) - ; Alcotest.test_case - "HTTP stalled TCP advances to next address" - `Quick - (test_stalled_tcp ~cancel:false) - ; Alcotest.test_case - "download publication failure preserves destination" - `Quick - test_artifact_publication_failure - ] -;; - -(* Generated independently with Python gzip.compress(..., mtime=0). *) -let gzip_downloads = - [ ( "one gzip layer" - , true - , "x" - , "\x1f\x8b\x08\x00\x00\x00\x00\x00\x02\xff\xab\x00\x00\x83\x16\xdc\x8c\x01\x00\x00\x00" - ) - ; ( "two gzip layers" - , true - , "x" - , "\x1f\x8b\x08\x00\x00\x00\x00\x00\x02\xff\x93\xef\xe6\x60\x00\x01\xa6\xff\xab\x19\x18\x9a\xc5\xee\xf4\x30\x02\x39\x00\xd5\xaa\x22\xe2\x15\x00\x00\x00" - ) - ; ( "three gzip layers rejected" - , false - , "" - , "\x1f\x8b\x08\x00\x00\x00\x00\x00\x02\xff\x93\xef\xe6\x60\x00\x01\xa6\xff\x93\xdf\x3f\x4b\x60\x60\x5c\xf6\x7f\xb5\xa4\xc4\xac\xa3\xef\xbe\x18\x30\x59\x32\x5c\x5d\xa5\xf4\x48\x14\x28\x09\x00\x34\x8c\x86\xf0\x25\x00\x00\x00" - ) - ; "truncated gzip rejected", false, "", "\x1f\x8b" - ; "empty plain artifact", true, "", "" - ; "one-byte signature prefix", true, "\x1f", "\x1f" - ; "reversed signature is plain", true, "\x8b\x1f", "\x8b\x1f" - ] -;; - -let test_close_with_pending_pong ~reply () = - let config = - match case ~ws:true "" with - | `Assoc fields -> - `Assoc (("reply_close", `Bool reply) :: ("close_delay", `Float 0.4) :: fields) - | _ -> assert false - in - let report = - with_peer [ config ] (fun ~environment ~sw ~support ~origin -> - let posted = ref [] - and invalidations = ref 0 in - let t = - runner - ~websocket_liveness: - (Runner.Ping_pong { interval_seconds = 0.02; timeout_seconds = 0.2 }) - ~environment - ~sw - ~support - ~posted - ~invalidations - () - in - let _, graph = bootstrap origin in - let scope = Core.{ graph; connection_generation = 1 } in - let clock = Eio.Stdenv.clock environment in - Runner.submit t (websocket_start origin); - (* The peer records a Ping only after receiving the complete frame, and - deliberately never answers it. Close starts with a Pong outstanding. *) - wait clock (fun () -> Sys.file_exists (Filename.concat support "frame-count")); - let started = Eio.Time.now clock in - Runner.submit t (websocket_close scope); - let closures () = - List.filter_map - (function - | Core.Websocket_closed (_, reason) -> Some reason - | _ -> None) - !posted - in - wait clock (fun () -> closures () <> []); - let elapsed = Eio.Time.now clock -. started in - Alcotest.(check bool) - "intentional Close is classified locally and terminates exactly once" - true - (closures () = [ Core.Local_close ]); - Alcotest.(check bool) - "pending Pong never shortens close grace" - true - (elapsed >= if reply then 0.35 else 0.9); - Runner.shutdown t; - Alcotest.(check int) "close callback occurs once" 1 (List.length (closures ()))) - in - check_retired report; - let frames = - Yojson.Basic.Util.(report |> to_list |> List.hd |> member "frames" |> to_list) - in - Alcotest.(check (list int)) - "one Ping followed by Close, no later heartbeat" - [ 9; 8 ] - (List.map - (fun frame -> Yojson.Basic.Util.(frame |> member "opcode" |> to_int)) - frames) -;; - -let scenarios = - scenarios - @ List.map - (fun (name, success, payload, body) -> - let headers = Printf.sprintf "Content-Length: %d\r\n" (String.length body) in - Alcotest.test_case - ("download " ^ name) - `Quick - (http_test - ~download:true - ~success - ~payload - ?failure_message:(if name = "truncated gzip rejected" then - Some "snapshot gzip decode failed" else None) - ?retained_download:(if success then None else Some body) - (response ~headers body))) - gzip_downloads - @ [ Alcotest.test_case - "WebSocket pending Pong permits delayed Close reply" - `Quick - (test_close_with_pending_pong ~reply:true) - ; Alcotest.test_case - "WebSocket pending Pong preserves silent-peer close grace" - `Quick - (test_close_with_pending_pong ~reply:false) - ] -;; - -let asset_download ~mismatch ~refresh () = - let module Asset = Logseq_db_types.Asset_descriptor in - let module Codec = Logseq_sync_effect_runner.Asset_codec in - let bytes = "\000\255file\128" in - let checksum = Codec.checksum (if mismatch then "different" else bytes) in - let version = Asset.version ~checksum ~file_type:"png" |> Result.get_ok in - let wire = - response - ~headers:(Printf.sprintf "Content-Length: %d\r\n" (String.length bytes)) - bytes - in - let cases = - (if refresh then [ case (response ~status:401 "x") ] else []) @ [ case wire ] - in - let report = - with_peer cases (fun ~environment ~sw ~support ~origin -> - let posted = ref [] - and invalidations = ref 0 in - let t = runner ~environment ~sw ~support ~posted ~invalidations () in - let selected, scope = bootstrap ~mirror_absent:false origin in - let clock = Eio.Stdenv.clock environment in - let result = - asset_run - t - ~clock - ~posted - selected.next - scope - "download" - (Core.Fetch_asset - { asset = scope.graph_id; version; maximum_plaintext_bytes = 32 }) - in - (match result with - | Ok (Core.Asset_downloaded file) when not mismatch -> - let retained = - asset_run - t - ~clock - ~posted - selected.next - scope - "retain" - (Core.Retain_asset_file file) - in - (match retained with - | Ok (Core.Asset_retained (Some (lease, path))) -> - Alcotest.(check string) "verified binary bytes" bytes (read_file path); - ignore - (asset_run - t - ~clock - ~posted - selected.next - scope - "release" - (Core.Release_asset_file lease)); - Alcotest.(check bool) - "download handle releases through completion" - true - (asset_run - t - ~clock - ~posted - selected.next - scope - "release-download" - (Core.Release_asset_file file) - = Ok Core.Asset_unit) - | _ -> Alcotest.fail "downloaded resource could not be retained") - | Error Core.Asset_checksum_mismatch when mismatch -> - let checked = - asset_run - t - ~clock - ~posted - selected.next - scope - "check" - (Core.Check_asset_cache (scope.graph_id, version)) - in - Alcotest.(check bool) - "bad object not published" - true - (checked = Ok (Core.Asset_cached None)) - | _ -> Alcotest.fail "unexpected asset download outcome"); - Alcotest.(check int) - "same token refresh owner" - (if refresh then 1 else 0) - !invalidations; - Runner.shutdown t) - in - check_retired report; - let request = - Yojson.Basic.Util.(report |> to_list |> List.hd |> member "request" |> to_string) - in - let expected = - "GET /assets/" - ^ Logseq_db_types.Graph_types.Uuid.to_string graph.graph_id - ^ "/" - ^ Logseq_db_types.Graph_types.Uuid.to_string graph.graph_id - ^ ".png HTTP/1.1" - in - Alcotest.(check bool) - "per-asset endpoint" - true - (String.starts_with ~prefix:(hex expected) request) -;; - -let scenarios = - scenarios - @ [ Alcotest.test_case - "asset GET publishes verified bytes" - `Quick - (asset_download ~mismatch:false ~refresh:false) - ; Alcotest.test_case - "asset GET rejects checksum mismatch" - `Quick - (asset_download ~mismatch:true ~refresh:false) - ; Alcotest.test_case - "asset GET refreshes authentication" - `Quick - (asset_download ~mismatch:false ~refresh:true) - ] -;; - -let asset_upload ~status ~refresh () = - let module Asset = Logseq_db_types.Asset_descriptor in - let bytes = "\000\255upload\128" in - let checksum = Logseq_sync_effect_runner.Asset_codec.checksum bytes in - let version = Asset.version ~checksum ~file_type:"png" |> Result.get_ok in - let cases = - (if refresh then [ case (response ~status:401 "x") ] else []) - @ [ case (response ~status "x") ] - in - let report = - with_peer cases (fun ~environment ~sw ~support ~origin -> - let posted = ref [] - and invalidations = ref 0 in - let diagnostics = ref [] in - let on_asset_diagnostic ~scope:_ ~operation ~request_operation detail = - diagnostics := (operation, request_operation, detail) :: !diagnostics in - let runner = runner ~on_asset_diagnostic ~environment ~sw ~support ~posted ~invalidations () in - let selected, scope = bootstrap ~mirror_absent:false origin in - let source_file = Filename.concat support "staged.bin" in - Out_channel.with_open_bin source_file (fun out -> output_string out bytes); - let clock = Eio.Stdenv.clock environment in - let file = asset_stage runner ~clock ~posted selected.next scope source_file in - let result = - asset_run ~with_cause:true - runner - ~clock - ~posted - selected.next - scope - "upload" - (Core.Put_asset_file - { asset = scope.graph_id; version; file; maximum_plaintext_bytes = 32 }) - in - let expected = - match status with - | 200 -> Ok Core.Asset_unit - | 403 -> Error Core.Asset_revoked_access - | 413 -> Error Core.Asset_size_rejected - | _ -> Error Core.Asset_network - in - Alcotest.(check bool) "explicit upload result" true (Result.map_error Core.asset_failure_kind result = expected); - (match status, !diagnostics with - | 200, [] -> () - | 200, _ -> Alcotest.fail "successful retry must not emit a failure diagnostic" - | _, [(operation, request_operation, detail)] -> - Alcotest.(check string) "finite diagnostic operation" "assetUpload" operation; - Alcotest.(check string) "exact request identity" "upload" request_operation; - let module D = Logseq_db_types.Failure_diagnostic in - let result_cause = match result with - | Error failure -> Core.asset_failure_cause failure - | Ok _ -> None in - Alcotest.(check (option string)) "typed completion preserves exact diagnostic" - (Some (D.encode detail)) (Option.map D.encode result_cause); - Alcotest.(check bool) "safe numeric HTTP status survives" true - (String.ends_with ~suffix:(Printf.sprintf "(http=%d)." status) (D.message detail)); - Alcotest.(check bool) "source status preserved by strict carrier" true - (D.decode (D.encode detail) |> Option.map D.message = Some (D.message detail)) - | _ -> Alcotest.fail "one terminal failure diagnostic expected"); - Alcotest.(check int) - "upload token refresh" - (if refresh then 1 else 0) - !invalidations; - Alcotest.(check bool) - "upload staging cleanup completes" - true - (asset_run - runner - ~clock - ~posted - selected.next - scope - "release-upload-stage" - (Core.Release_staged_file file) - = Ok Core.Asset_unit); - Runner.shutdown runner) - in - check_retired report; - check_default_connection report; - Yojson.Basic.Util.to_list report - |> List.iter (fun row -> - let encoded = Yojson.Basic.Util.(row |> member "request" |> to_string) in - let request = - String.init - (String.length encoded / 2) - (fun n -> Char.chr (int_of_string ("0x" ^ String.sub encoded (2 * n) 2))) - in - let endpoint = - Printf.sprintf - "PUT /assets/%s/%s.png HTTP/1.1" - (Logseq_db_types.Graph_types.Uuid.to_string graph.graph_id) - (Logseq_db_types.Graph_types.Uuid.to_string graph.graph_id) - in - Alcotest.(check bool) - "immutable asset endpoint" - true - (String.starts_with ~prefix:endpoint request); - let headers = String.split_on_char '\n' request |> List.map String.trim in - List.iter - (fun header -> Alcotest.(check bool) header true (List.mem header headers)) - [ "x-amz-meta-checksum: " ^ checksum - ; "x-amz-meta-type: png" - ; "content-length: " ^ string_of_int (String.length bytes) - ]; - Alcotest.(check bool) - "raw uploaded bytes" - true - (String.ends_with ~suffix:bytes request)) -;; - -let asset_upload_source () = - with_support (fun support -> - Eio_main.run (fun environment -> - Eio.Switch.run (fun sw -> - let posted = ref [] - and invalidations = ref 0 in - let runner = runner ~environment ~sw ~support ~posted ~invalidations () in - let selected, scope = - bootstrap ~mirror_absent:false (Uri.of_string "https://localhost:1") - in - let clock = Eio.Stdenv.clock environment in - let version = - Logseq_db_types.Asset_descriptor.version - ~checksum:(Logseq_sync_effect_runner.Asset_codec.checksum "file") - ~file_type:"bin" - |> Result.get_ok - in - let source_file = Filename.concat support "source.bin" in - let missing = - asset_run - runner - ~clock - ~posted - selected.next - scope - "missing" - (Core.Put_asset_file - { asset = scope.graph_id - ; version - ; file = "unavailable-stage" - ; maximum_plaintext_bytes = 32 - }) - in - Alcotest.(check bool) - "missing staging file" - true - (missing = Error Core.Asset_missing_source); - Out_channel.with_open_bin source_file (fun out -> output_string out "file"); - let file = asset_stage runner ~clock ~posted selected.next scope source_file in - let put core scope operation maximum_plaintext_bytes version = - asset_run - runner - ~clock - ~posted - core - scope - operation - (Core.Put_asset_file - { asset = scope.Core.graph_id; version; file; maximum_plaintext_bytes }) - in - Alcotest.(check bool) - "size bound before network" - true - (put selected.next scope "size" 3 version = Error Core.Asset_size_rejected); - let locked, locked_scope = - Core_contract.selected_graph Core_contract.encrypted_graph - in - let locked_file = - asset_stage runner ~clock ~posted locked.next locked_scope source_file - in - Alcotest.(check bool) - "locked graph never sends plaintext" - true - (asset_run - runner - ~clock - ~posted - locked.next - locked_scope - "locked" - (Core.Put_asset_file - { asset = locked_scope.graph_id - ; version - ; file = locked_file - ; maximum_plaintext_bytes = 32 - }) - = Error Core.Asset_locked); - let requested = - Core.step - selected.next - (Core.Asset_requested - { scope - ; operation = "cancelled" - ; action = - Core.Put_asset_file - { asset = scope.graph_id - ; version - ; file - ; maximum_plaintext_bytes = 32 - } - }) - in - let cancelled = - Core.step - requested.next - (Core.Asset_requested - { scope - ; operation = "cancel" - ; action = Core.Cancel_asset_operation "cancelled" - }) - in - Alcotest.(check bool) - "cancelled intent never uploads" - true - (List.exists - (function - | Core.Publish (Core.Asset_finished output) -> - output.request.operation = "cancelled" - && output.result = Error Core.Asset_cancelled - | _ -> false) - cancelled.effects); - let incorrect = - Logseq_db_types.Asset_descriptor.version - ~checksum:(Logseq_sync_effect_runner.Asset_codec.checksum "oops") - ~file_type:"bin" - |> Result.get_ok - in - Alcotest.(check bool) - "changed source rejected" - true - (match put selected.next scope "changed" 32 incorrect with - | Error (Core.Asset_invalid_content _) -> true - | _ -> false); - Runner.shutdown runner))) -;; - -let scenarios = - scenarios - @ [ Alcotest.test_case - "asset PUT bytes and metadata" - `Quick - (asset_upload ~status:200 ~refresh:false) - ; Alcotest.test_case - "asset PUT token refresh" - `Quick - (asset_upload ~status:200 ~refresh:true) - ; Alcotest.test_case - "asset PUT revoked access" - `Quick - (asset_upload ~status:403 ~refresh:false) - ; Alcotest.test_case - "asset PUT size rejection" - `Quick - (asset_upload ~status:413 ~refresh:false) - ; Alcotest.test_case - "asset PUT transient failure" - `Quick - (asset_upload ~status:503 ~refresh:false) - ; Alcotest.test_case "asset PUT validates staging" `Quick asset_upload_source - ] -;; - -let asset_upload_admission () = - with_support (fun support -> - Eio_main.run (fun environment -> - Eio.Switch.run (fun sw -> - let active = ref 0 - and maximum = ref 0 in - let acquire _ = - incr active; - maximum := max !maximum !active; - Eio.Fiber.yield (); - decr active; - Error "offline" - in - let posted = ref [] in - let t = - runner ~acquire ~environment ~sw ~support ~posted ~invalidations:(ref 0) () - in - let selected, scope = - bootstrap ~mirror_absent:false (Uri.of_string "https://localhost") - in - let clock = Eio.Stdenv.clock environment in - let source_file = Filename.concat support "pending.bin" in - Out_channel.with_open_bin source_file (fun out -> output_string out "file"); - let file = asset_stage t ~clock ~posted selected.next scope source_file in - let version = - Logseq_db_types.Asset_descriptor.version - ~checksum:(Logseq_sync_effect_runner.Asset_codec.checksum "file") - ~file_type:"bin" - |> Result.get_ok - in - posted := []; - let core = - List.fold_left - (fun core index -> - asset_submit - t - core - scope - ("upload-" ^ string_of_int index) - (Core.Put_asset_file - { asset = scope.graph_id; version; file; maximum_plaintext_bytes = 4 })) - selected.next - [ 1; 2; 3 ] - in - wait clock (fun () -> List.length !posted = 3); - let _, outputs = asset_collect t core !posted in - Alcotest.(check int) "one upload holds source and wire buffers" 1 !maximum; - Alcotest.(check bool) - "failed transfer releases admission" - true - (List.length outputs = 3 - && List.for_all (fun output -> Result.is_error output.Core.result) outputs); - Runner.shutdown t))) -;; - -let scenarios = - scenarios - @ [ Alcotest.test_case "asset upload resource admission" `Quick asset_upload_admission ] -;; - -let asset_download_budget maximum_plaintext_bytes expected_entered () = - let module Asset = Logseq_db_types.Asset_descriptor in - with_support (fun support -> - Eio_main.run (fun environment -> - Eio.Switch.run (fun sw -> - let entered = ref 0 in - let gate, release = Eio.Promise.create () in - let acquire _ = - incr entered; - Eio.Promise.await gate; - Error "offline" - in - let posted = ref [] in - let t = - runner ~acquire ~environment ~sw ~support ~posted ~invalidations:(ref 0) () - in - let version = - Asset.version ~checksum:(String.make 64 '0') ~file_type:"bin" |> Result.get_ok - in - let cores = - List.init 5 (fun index -> - let selected, scope = - bootstrap - ~mirror_absent:false - ~user_id:("fixture-" ^ string_of_int index) - (Uri.of_string "https://localhost") - in - asset_submit - t - selected.next - scope - ("download-" ^ string_of_int index) - (Core.Fetch_asset - { asset = scope.graph_id; version; maximum_plaintext_bytes })) - in - for _ = 1 to 10 do - Eio.Fiber.yield () - done; - let initial = !entered in - Eio.Promise.resolve release (); - wait (Eio.Stdenv.clock environment) (fun () -> List.length !posted = 5); - let outputs = - List.concat_map (fun core -> snd (asset_collect t core !posted)) cores - in - Alcotest.(check int) - "scopes share bounded download and byte permits" - expected_entered - initial; - Alcotest.(check int) "all failed requests release admission" 5 !entered; - Alcotest.(check int) - "all completions pass through reducer outputs" - 5 - (List.length outputs); - Runner.shutdown t))) -;; - -let asset_download_admission = asset_download_budget 4 3 - -let scenarios = - scenarios - @ [ Alcotest.test_case - "asset download resource admission" - `Quick - asset_download_admission - ] -;; - -let shared_asset_byte_budget = asset_download_budget 16777216 2 - -let scenarios = - scenarios - @ [ Alcotest.test_case "shared asset byte budget" `Quick shared_asset_byte_budget ] -;; - -let shared_asset_codec_admission () = - let module Asset = Logseq_db_types.Asset_descriptor in - let plaintext = "file" in - let version = - Asset.version - ~checksum:(Logseq_sync_effect_runner.Asset_codec.checksum plaintext) - ~file_type:"bin" - |> Result.get_ok - in - let envelope = - Transit_native.Transit.Json.to_string - (Transit_core.Json.Array - [ Binary (String.make 12 'i'); Binary (String.make 20 'c') ]) - in - let wire = - response - ~headers:(Printf.sprintf "Content-Length: %d\r\n" (String.length envelope)) - envelope - in - let report = - with_peer - [ case wire ] - (fun ~environment ~sw ~support ~origin -> - let gate, release = Eio.Promise.create () in - let active = ref 0 - and maximum = ref 0 - and encrypted = ref false - and decrypted = ref false in - let use flag result = - flag := true; - incr active; - maximum := max !maximum !active; - Eio.Promise.await gate; - decr active; - result - in - let crypto_dependency = - Runner.crypto - ~encrypt_aes_gcm:(fun ~key:_ ~plaintext:_ -> - use encrypted (Error "fixture encode failure")) - ~decrypt_aes_gcm:(fun ~key:_ ~iv:_ ~ciphertext:_ -> - use decrypted (Ok plaintext)) - |> Result.get_ok - in - let secrets_dependency = - Runner.secrets - ~unlock_private_key: - (fun - ~managed_sync_origin:_ ~user_id:_ ~password:_ ~private_key_package:_ -> - Ok ()) - ~unlock_graph_key: - (fun - ~managed_sync_origin:_ ~user_id:_ ~encrypted_graph_key:_ -> - Ok (String.make 32 'k')) - ~load_wrapped_graph_key:(fun ~managed_sync_origin:_ ~user_id:_ ~graph_id:_ -> - Ok "wrapped") - ~verify_and_save_wrapped_graph_key: - (fun - ~managed_sync_origin:_ ~user_id:_ ~graph_id:_ ~encrypted_graph_key:_ -> - Ok ()) - ~delete_account_secrets:(fun ~managed_sync_origin:_ ~user_id:_ -> Ok ()) - |> Result.get_ok - in - let posted = ref [] in - let t = - runner - ~crypto_dependency - ~secrets_dependency - ~environment - ~sw - ~support - ~posted - ~invalidations:(ref 0) - () - in - let before, instruction = Runner_contract.cached_key_effect ~origin () in - Runner.submit t instruction; - wait (Eio.Stdenv.clock environment) (fun () -> !posted <> []); - let unlocked = (Core.step before.next (List.hd !posted)).next in - let context = Core.asset_context unlocked |> Option.get in - let scope = context.scope in - let source_file = Filename.concat support "pending.bin" in - Out_channel.with_open_bin source_file (fun out -> output_string out plaintext); - let clock = Eio.Stdenv.clock environment in - let file = asset_stage t ~clock ~posted unlocked scope source_file in - posted := []; - let uploading = - asset_submit - t - unlocked - scope - "codec-upload" - (Core.Put_asset_file - { asset = scope.graph_id; version; file; maximum_plaintext_bytes = 4 }) - in - wait clock (fun () -> !encrypted); - let downloading = - asset_submit - t - uploading - scope - "codec-download" - (Core.Fetch_asset - { asset = scope.graph_id; version; maximum_plaintext_bytes = 4 }) - in - wait (Eio.Stdenv.clock environment) (fun () -> - Sys.file_exists (Filename.concat support "report")); - Eio.Promise.resolve release (); - wait (Eio.Stdenv.clock environment) (fun () -> List.length !posted = 2); - let _, outputs = asset_collect t downloading !posted in - Alcotest.(check int) - "both codec completions reach reducer" - 2 - (List.length outputs); - Alcotest.(check int) "GET and PUT share one codec permit" 1 !maximum; - Alcotest.(check bool) - "failed encoding releases the permit for decoding" - true - !decrypted; - Runner.shutdown t) - in - check_retired report -;; - -let scenarios = - scenarios - @ [ Alcotest.test_case - "shared asset codec admission" - `Quick - shared_asset_codec_admission - ] -;; - -(* Provider refresh contracts execute through reducer-created HTTP operations. *) -let submitted_authentication_policy - statuses - expected_result - expected_tokens - expected_invalidated - () - = - let acquired = ref [] - and invalidated = ref [] in - let tokens = Queue.create () in - Queue.add "token-1" tokens; - Queue.add "token-2" tokens; - let report = - with_peer - (List.map (fun status -> case (response ~status "x")) statuses) - (fun ~environment ~sw ~support ~origin -> - let posted = ref [] in - let acquire _ = - match Queue.take_opt tokens with - | Some token -> - acquired := token :: !acquired; - Ok token - | None -> Error "no token" - in - let t = - runner - ~acquire - ~on_invalidate:(fun token -> invalidated := token :: !invalidated) - ~environment - ~sw - ~support - ~posted - ~invalidations:(ref 0) - () - in - let requested, _ = bootstrap origin in - let runnable = - List.find_map - (function - | Core.Run (Core.Request (_, Core.Fetch_snapshot_metadata _) as runnable) - -> Some runnable - | _ -> None) - requested.effects - |> Option.get - in - Runner.submit t runnable; - wait (Eio.Stdenv.clock environment) (fun () -> completed !posted <> None); - Alcotest.(check bool) - "typed authentication outcome" - true - (match completed !posted, expected_result with - | Some (Ok ()), None -> true - | Some (Error (Core.Effect_failed message)), Some expected -> - String.equal message expected - | _ -> false); - List.iter (fun event -> ignore (Core.step requested.next event)) !posted; - Runner.shutdown t) - in - Alcotest.(check (list string)) - "bounded acquired token attempts" - expected_tokens - (List.rev !acquired); - Alcotest.(check (list string)) - "only first unauthorized token invalidated" - expected_invalidated - (List.rev !invalidated); - check_retired report -;; - -let scenarios = - scenarios - @ [ Alcotest.test_case - "authenticated operation retries one unauthorized response" - `Quick - (submitted_authentication_policy - [ 401; 200 ] - None - [ "token-1"; "token-2" ] - [ "token-1" ]) - ; Alcotest.test_case - "authenticated operation surfaces second unauthorized response" - `Quick - (submitted_authentication_policy - [ 401; 401 ] - (Some "Authentication failed.") - [ "token-1"; "token-2" ] - [ "token-1" ]) - ; Alcotest.test_case - "authenticated operation does not retry forbidden response" - `Quick - (submitted_authentication_policy - [ 403 ] - (Some "Authorization failed.") - [ "token-1" ] - []) - ] -;; - -(* Semaphore ownership belongs to runner I/O: cancellation while a reservation is - incomplete must release the units already acquired, before the first request ends. *) -let cancelled_asset_byte_wait_releases_partial_reservation () = - with_support (fun support -> - Eio_main.run (fun environment -> - Eio.Switch.run (fun sw -> - let entered = ref 0 in - let gate, release = Eio.Promise.create () in - let acquire _ = - incr entered; - Eio.Promise.await gate; - Error "offline" - in - let posted = ref [] in - let t = - runner ~acquire ~environment ~sw ~support ~posted ~invalidations:(ref 0) () - in - let selected, scope = - bootstrap ~mirror_absent:false (Uri.of_string "https://localhost") - in - let version = - Logseq_db_types.Asset_descriptor.version - ~checksum:(String.make 64 'a') - ~file_type:"bin" - |> Result.get_ok - in - let action maximum_plaintext_bytes = - Core.Fetch_asset { asset = scope.graph_id; version; maximum_plaintext_bytes } - in - let first = - asset_submit t selected.next scope "holding-48MiB" (action (24 * 1024 * 1024)) - in - wait (Eio.Stdenv.clock environment) (fun () -> !entered = 1); - let waiting = - asset_submit t first scope "waiting-for-48MiB" (action (24 * 1024 * 1024)) - in - for _ = 1 to 10 do - Eio.Fiber.yield () - done; - let cancelled = - asset_submit - t - waiting - scope - "cancel-waiter" - (Core.Cancel_asset_operation "waiting-for-48MiB") - in - wait (Eio.Stdenv.clock environment) (fun () -> List.length !posted >= 2); - let admitted = - asset_submit t cancelled scope "using-returned-16MiB" (action (8 * 1024 * 1024)) - in - wait (Eio.Stdenv.clock environment) (fun () -> !entered = 2); - Alcotest.(check int) - "cancelled partial reservation immediately admits another request" - 2 - !entered; - Eio.Promise.resolve release (); - wait (Eio.Stdenv.clock environment) (fun () -> List.length !posted = 4); - ignore (asset_collect t admitted !posted); - Runner.shutdown t))) -;; - -let scenarios = - scenarios - @ [ Alcotest.test_case - "asset cancellation releases partial byte reservation" - `Quick - cancelled_asset_byte_wait_releases_partial_reservation - ] -;; - -(* Two legal 40 MiB reservations must not divide the 64 MiB pool and deadlock. *) -let weighted_asset_reservations_make_progress () = - let bytes = "file" in - let version = - Logseq_db_types.Asset_descriptor.version - ~checksum:(Logseq_sync_effect_runner.Asset_codec.checksum bytes) - ~file_type:"bin" - |> Result.get_ok - in - let headers = "Content-Length: 4\r\n" in - let report = - with_peer - [ case (response ~headers bytes); case (response ~headers bytes) ] - (fun ~environment ~sw ~support ~origin -> - let entered = ref 0 in - let gate, release = Eio.Promise.create () in - let acquire _ = - incr entered; - Eio.Promise.await gate; - Ok "fixture-token" - in - let posted = ref [] in - let t = - runner ~acquire ~environment ~sw ~support ~posted ~invalidations:(ref 0) () - in - let selected, scope = bootstrap ~mirror_absent:false origin in - let action = - Core.Fetch_asset - { asset = scope.graph_id - ; version - ; maximum_plaintext_bytes = 20 * 1024 * 1024 - } - in - let first = asset_submit t selected.next scope "weighted-first" action in - wait (Eio.Stdenv.clock environment) (fun () -> !entered = 1); - let second = asset_submit t first scope "weighted-second" action in - for _ = 1 to 10 do - Eio.Fiber.yield () - done; - Alcotest.(check int) - "one entire reservation enters while the second waits" - 1 - !entered; - Eio.Promise.resolve release (); - wait (Eio.Stdenv.clock environment) (fun () -> List.length !posted = 2); - let _, outputs = asset_collect t second !posted in - Alcotest.(check int) - "both weighted TLS downloads complete" - 2 - (List.length outputs); - Alcotest.(check bool) - "weighted reservations release for the next request" - true - (List.for_all - (fun output -> - match output.Core.result with - | Ok (Core.Asset_downloaded _) -> true - | _ -> false) - outputs); - Alcotest.(check int) "second request proceeds after the first" 2 !entered; - Runner.shutdown t) - in - check_retired report -;; - -let scenarios = - scenarios - @ [ Alcotest.test_case - "weighted asset reservations make TLS progress" - `Quick - weighted_asset_reservations_make_progress - ] -;; - -(* Migrated downloaded cache scenarios. *) -module Cache_asset = Logseq_db_types.Asset_descriptor -module Cache_codec = Logseq_sync_effect_runner.Asset_codec - -let cache_version bytes = - Cache_asset.version ~checksum:(Cache_codec.checksum bytes) ~file_type:"png" - |> Result.get_ok -;; - -let cache_response bytes = - case - (response - ~headers:(Printf.sprintf "Content-Length: %d\r\n" (String.length bytes)) - bytes) -;; - -let cache_selection ?(user = "fixture") ?(selected_graph : Core.graph = graph) origin = - let initial = - Core.config ~managed_sync_origin:origin ~limits:(Core_contract.limits ()) - |> Result.get_ok - |> Core.initial - |> Result.get_ok - |> fun core -> (Core.step core (Core.Sign_in_requested {account_generation=0;presentation_generation=0})).next - in - let authenticated = - Core.step initial (Core.Account_authenticated { user_id = Some user }) - in - let catalog = - List.find_map - (function - | Core.Run (Core.Request (ticket, Core.Fetch_catalog _)) -> - Some - (Core.step - authenticated.next - (Core.Runner_completed (Core.Completion (ticket, Ok [ selected_graph ])))) - | _ -> None) - authenticated.effects - |> Option.get - in - let selected = Core.step catalog.next (Core.Graph_selected selected_graph.graph_id) in - selected.next, Core.admitted_graph_scope selected.next |> Option.get -;; - -let with_downloaded_cache ?(budget = 8L) ?(pending_budget = 8L) cases test = - let report = - with_peer cases (fun ~environment ~sw ~support ~origin -> - let posted = ref [] - and invalidations = ref 0 in - let create () = - runner - ~asset_cache_budget_bytes:budget - ~asset_maximum_file_bytes:8 - ~asset_pending_budget_bytes:pending_budget - ~environment - ~sw - ~support - ~posted - ~invalidations - () - in - let active = ref (create ()) in - Fun.protect - ~finally:(fun () -> Runner.shutdown !active) - (fun () -> - let sequence = ref 0 in - let request core scope action = - incr sequence; - asset_run - !active - ~clock:(Eio.Stdenv.clock environment) - ~posted - core - scope - ("cache-" ^ string_of_int !sequence) - action - in - let reopen () = - Runner.shutdown !active; - active := create () - in - test support origin request reopen)) - in - check_retired report -;; - -let cache_download request core (scope : Core.graph_scope) bytes = - match - request - core - scope - (Core.Fetch_asset - { asset = scope.graph_id - ; version = cache_version bytes - ; maximum_plaintext_bytes = 8 - }) - with - | Ok (Core.Asset_downloaded handle) -> handle - | _ -> Alcotest.fail "cache download did not produce a controlled resource" -;; - -let cache_lookup request core (scope : Core.graph_scope) bytes = - match - request core scope (Core.Check_asset_cache (scope.graph_id, cache_version bytes)) - with - | Ok (Core.Asset_cached handle) -> handle - | _ -> Alcotest.fail "cache lookup did not resolve through reducer output" -;; - -let cache_retain request core scope handle = - match request core scope (Core.Retain_asset_file handle) with - | Ok (Core.Asset_retained (Some resource)) -> resource - | _ -> Alcotest.fail "cache resource retention failed" -;; - -let cache_release request core scope handle = - Runner_contract.cache_unit (request core scope (Core.Release_asset_file handle)) -;; - -let test_downloaded_cache_restart () = - with_downloaded_cache - [ cache_response "file" ] - (fun _ origin request reopen -> - let core, scope = cache_selection origin in - let handle = cache_download request core scope "file" in - let lease, path = cache_retain request core scope handle in - Alcotest.(check string) "published bytes" "file" (read_file path); - cache_release request core scope lease; - reopen (); - Alcotest.(check bool) - "restart restores verified file" - true - (Option.is_some (cache_lookup request core scope "file"))) -;; - -let test_downloaded_cache_corruption () = - with_downloaded_cache - [ cache_response "file" ] - (fun _ origin request reopen -> - let core, scope = cache_selection origin in - let handle = cache_download request core scope "file" in - let lease, path = cache_retain request core scope handle in - cache_release request core scope lease; - cache_release request core scope handle; - Runner_contract.cache_unit (request core scope Core.Close_asset_scope); - Out_channel.with_open_bin path (fun output -> output_string output "bad!"); - reopen (); - Alcotest.(check bool) - "corrupt file not ready" - true - (cache_lookup request core scope "file" = None)) -;; - -let test_downloaded_cache_eligibility () = - with_downloaded_cache - [ cache_response "bad!" ] - (fun _ origin request _ -> - let core, scope = cache_selection origin in - Alcotest.(check bool) - "checksum verified before publication" - true - (request - core - scope - (Core.Fetch_asset - { asset = scope.graph_id - ; version = cache_version "file" - ; maximum_plaintext_bytes = 8 - }) - = Error Core.Asset_checksum_mismatch); - Alcotest.(check bool) - "no corrupt record" - true - (cache_lookup request core scope "file" = None)) -;; - -let test_downloaded_cache_leases () = - with_downloaded_cache - ~budget:4L - [ cache_response "file"; cache_response "next"; cache_response "next" ] - (fun support origin request _ -> - let core, scope = cache_selection origin in - let source = Filename.concat support "picker.bin" in - Out_channel.with_open_bin source (fun output -> output_string output "source"); - let staged, _, _ = Runner_contract.cache_stage (request core scope) source in - let stage_lease, staged_path = - request core scope (Core.Retain_staged_file staged) - |> Runner_contract.retained_value - in - cache_release request core scope stage_lease; - let handle = cache_download request core scope "file" in - let lease, path = cache_retain request core scope handle in - cache_release request core scope handle; - Alcotest.(check bool) - "live renderer pins file" - true - (request - core - scope - (Core.Fetch_asset - { asset = scope.graph_id - ; version = cache_version "next" - ; maximum_plaintext_bytes = 8 - }) - = Error Core.Asset_storage_full); - Alcotest.(check bool) "renderer path remains" true (Sys.file_exists path); - cache_release request core scope lease; - ignore (cache_download request core scope "next"); - Alcotest.(check bool) - "unpinned LRU evicted" - true - (cache_lookup request core scope "file" = None); - Alcotest.(check bool) - "LRU does not evict pending import" - true - (Sys.file_exists staged_path)) -;; - -let test_downloaded_cache_filename () = - with_downloaded_cache - [ cache_response "file" ] - (fun _ origin request reopen -> - let core, scope = cache_selection origin in - let handle = cache_download request core scope "file" in - let lease, path = cache_retain request core scope handle in - Alcotest.(check bool) - "download carries its type" - true - (Filename.check_suffix path ".png"); - cache_release request core scope lease; - cache_release request core scope handle; - reopen (); - let handle = cache_lookup request core scope "file" |> Option.get in - let lease, path = cache_retain request core scope handle in - Alcotest.(check bool) - "restart preserves typed filename" - true - (Filename.check_suffix path ".png"); - cache_release request core scope lease; - cache_release request core scope handle) -;; - -let test_downloaded_cache_legacy_cleanup () = - with_downloaded_cache - [ cache_response "file" ] - (fun _ origin request reopen -> - let core, scope = cache_selection origin in - let handle = cache_download request core scope "file" in - let lease, path = cache_retain request core scope handle in - let legacy = Filename.chop_extension path ^ ".bin" in - cache_release request core scope lease; - cache_release request core scope handle; - Sys.rename path legacy; - Runner_contract.cache_unit (request core scope Core.Close_asset_scope); - reopen (); - let restored = cache_lookup request core scope "file" in - Alcotest.(check bool) "untyped data file evicted" false (Sys.file_exists legacy); - Alcotest.(check bool) "manifest without typed data removed" true (restored = None)) -;; - -let test_downloaded_cache_isolation () = - with_downloaded_cache - [ cache_response "file"; cache_response "next" ] - (fun _ origin request _ -> - let first, first_scope = cache_selection origin in - let second, second_scope = cache_selection ~user:"another" origin in - let first_handle=cache_download request first first_scope "file" in - Alcotest.(check bool) - "accounts isolated" - true - (cache_lookup request second second_scope "file" = None); - ignore (cache_download request second second_scope "next"); - cache_release request first first_scope first_handle; - Runner_contract.cache_unit (Runner_contract.retire_cache (request first first_scope)); - Alcotest.(check bool) - "deletion leaves other account" - true - (Option.is_some (cache_lookup request second second_scope "next"))) -;; - -let test_downloaded_cache_account_cleanup () = - with_downloaded_cache - [ cache_response "file"; cache_response "file"; cache_response "file" ] - (fun _ origin request _ -> - let other_report = - with_peer - [ cache_response "file" ] - (fun ~environment:_ ~sw:_ ~support:_ ~origin:other_origin -> - let first = cache_selection origin in - let another_graph = - cache_selection - ~selected_graph:{ graph with graph_id = Core_contract.other_graph_id } - origin - in - let other_user = cache_selection ~user:"other" origin in - let other_origin = cache_selection other_origin in - List.iter - (fun (core, scope) -> - let handle = cache_download request core scope "file" in - cache_release request core scope handle; - Runner_contract.cache_unit (request core scope Core.Close_asset_scope)) - [ first; another_graph; other_user; other_origin ]; - let first_core, first_scope = first in - Runner_contract.cache_unit - (request first_core first_scope Core.Delete_account_assets); - Runner_contract.cache_unit - (request first_core first_scope Core.Delete_account_assets); - List.iter - (fun (core, scope) -> - Alcotest.(check bool) - "all account graphs removed" - true - (cache_lookup request core scope "file" = None)) - [ first; another_graph ]; - List.iter - (fun (core, scope) -> - Alcotest.(check bool) - "other namespace preserved" - true - (Option.is_some (cache_lookup request core scope "file"))) - [ other_user; other_origin ]) - in - check_retired other_report) -;; - -let test_downloaded_cache_graph_cleanup () = - with_downloaded_cache - [ cache_response "file"; cache_response "file" ] - (fun _ origin request _ -> - let first, scope = cache_selection origin in - let another, another_scope = - cache_selection - ~selected_graph:{ graph with graph_id = Core_contract.other_graph_id } - origin - in - let first_handle=cache_download request first scope "file" in - ignore(cache_download request another another_scope "file"); - cache_release request first scope first_handle; - Runner_contract.cache_unit (Runner_contract.retire_cache (request first scope)); - Runner_contract.cache_unit (Runner_contract.retire_cache (request first scope)); - Alcotest.(check bool) - "selected graph removed" - true - (cache_lookup request first scope "file" = None); - Alcotest.(check bool) - "other graph retained" - true - (Option.is_some (cache_lookup request another another_scope "file"))) -;; - -let scenarios = - scenarios - @ List.map - (fun (name, test) -> Alcotest.test_case name `Quick test) - [ "cache restart via submit", test_downloaded_cache_restart - ; "cache corruption via submit", test_downloaded_cache_corruption - ; "cache publication eligibility via submit", test_downloaded_cache_eligibility - ; "cache leases and budget via submit", test_downloaded_cache_leases - ; "cache downloaded filename via submit", test_downloaded_cache_filename - ; "cache legacy bin cleanup via submit", test_downloaded_cache_legacy_cleanup - ; "cache account isolation via submit", test_downloaded_cache_isolation - ; "cache account cleanup via submit", test_downloaded_cache_account_cleanup - ; "cache graph cleanup via submit", test_downloaded_cache_graph_cleanup - ] -;; - -(* These failures execute transport framing/deadlines, not Core policy. There is - no pure reducer boundary that can parse coalesced TLS bytes or own DNS/socket - cancellation. All fixtures enter through public Runner.transport/submit. *) -let test_early_frame_backpressure () = - let message = {|{"type":"changed","t":42}|} in - let wire = String.concat "" (List.init 129 (fun _ -> frame 1 message)) in - let report = - with_peer - [ case ~ws:true wire ] - (fun ~environment ~sw ~support ~origin -> - let posted = ref [] - and invalidations = ref 0 in - let t = runner ~environment ~sw ~support ~posted ~invalidations () in - let start = websocket_start origin in - let scope = - match start with - | Core.Start_websocket r -> r.scope - | _ -> assert false - in - Runner.submit t start; - let count () = - List.length - (List.filter - (function - | Core.Websocket_message _ -> true - | _ -> false) - !posted) - in - wait (Eio.Stdenv.clock environment) (fun () -> - count () = 129 - || List.exists - (function - | Core.Websocket_closed _ -> true - | _ -> false) - !posted); - Alcotest.(check int) - "all small coalesced messages delivered without rejection" - 129 - (count ()); - Runner.submit t (websocket_close scope); - wait (Eio.Stdenv.clock environment) (fun () -> - List.exists - (function - | Core.Websocket_closed _ -> true - | _ -> false) - !posted); - Runner.shutdown t) - in - check_retired report -;; - -let test_segmented_backpressure_control () = - let message = {|{"type":"changed","t":42}|} in - ws_test - ~messages:129 - ~terminal:false - ~segment:26 - (String.concat "" (List.init 129 (fun _ -> frame 1 message))) - () -;; - -let delayed_case ?(tls_delay = 0.) ?(response_delay = 0.) wire = - match case wire with - | `Assoc fields -> - `Assoc - (("tls_delay", `Float tls_delay) - :: ("response_delay", `Float response_delay) - :: fields) - | _ -> assert false -;; - -let test_tls_slow_legal_handshake () = - let report = - with_peer - [ delayed_case ~tls_delay:1.1 (response "x") ] - (fun ~environment ~sw ~support ~origin -> - let posted = ref [] - and invalidations = ref 0 in - let t = runner ~environment ~sw ~support ~posted ~invalidations () in - Runner.submit t (operation ~download:false origin); - wait (Eio.Stdenv.clock environment) (fun () -> - Option.is_some (completed !posted)); - Alcotest.(check bool) - "valid handshake beyond one second succeeds" - true - (Result.is_ok (Option.get (completed !posted))); - Runner.shutdown t) - in - check_retired report -;; - -let test_phased_response_deadline () = - let report = - with_peer - [ delayed_case ~tls_delay:0.3 ~response_delay:0.9 (response "x") ] - (fun ~environment ~sw ~support ~origin -> - let clock = scaled_clock (Eio.Stdenv.clock environment) 20. in - let network = - delayed_network (Eio.Stdenv.net environment) (fun () -> - Eio.Time.sleep clock 10.) - in - let posted = ref [] - and invalidations = ref 0 in - let t = - runner ~network ~clock ~environment ~sw ~support ~posted ~invalidations () - in - let before = Eio.Time.now clock in - Runner.submit t (operation ~download:false origin); - wait (Eio.Stdenv.clock environment) (fun () -> - Option.is_some (completed !posted)); - Alcotest.(check bool) - "response retains its budget after independent dial and TLS" - true - (Result.is_ok (Option.get (completed !posted))); - Alcotest.(check bool) - "total operation legitimately exceeds old outer 30s" - true - (Eio.Time.now clock -. before > 30.); - Runner.shutdown t) - in - check_retired report -;; - -let test_shared_dial_deadline () = - Runner_contract.with_support (fun support -> - Eio_main.run (fun environment -> - Eio.Switch.run (fun sw -> - let clock = scaled_clock (Eio.Stdenv.clock environment) 30. in - let attempts = ref 0 in - let network = - delayed_network (Eio.Stdenv.net environment) (fun () -> - incr attempts; - Eio.Time.sleep clock 100.) - in - let posted = ref [] - and invalidations = ref 0 in - let t = - runner ~network ~clock ~environment ~sw ~support ~posted ~invalidations () - in - let before = Eio.Time.now clock in - Runner.submit t (operation ~download:false (Uri.of_string "https://localhost:1")); - wait (Eio.Stdenv.clock environment) (fun () -> Option.is_some (completed !posted)); - let elapsed = Eio.Time.now clock -. before in - Alcotest.(check bool) - "shared DNS/TCP window reaches 30s" - true - (elapsed >= 29. && elapsed < 33.); - Alcotest.(check bool) - "address fallback uses the shared remaining window" - true - (!attempts >= 2); - Alcotest.(check bool) - "dial deadline returns a transport failure" - true - (Result.is_error (Option.get (completed !posted))); - Runner.shutdown t))) -;; - -let test_tls_independent_deadline () = - let report = - with_peer - [ `Assoc [ "stall_tls", `Bool true ] ] - (fun ~environment ~sw ~support ~origin -> - let clock = scaled_clock (Eio.Stdenv.clock environment) 20. in - let posted = ref [] - and invalidations = ref 0 in - let t = runner ~clock ~environment ~sw ~support ~posted ~invalidations () in - let before = Eio.Time.now clock in - Runner.submit t (operation ~download:false origin); - wait (Eio.Stdenv.clock environment) (fun () -> - Option.is_some (completed !posted)); - let elapsed = Eio.Time.now clock -. before in - Alcotest.(check bool) - "TLS has independent 10s window" - true - (elapsed >= 9. && elapsed < 13.); - Alcotest.(check bool) - "TLS timeout is a transport failure" - true - (Result.is_error (Option.get (completed !posted))); - Runner.shutdown t) - in - check_retired report -;; - -let test_redirect_response_budget_is_shared () = - let report = - with_peer - [ delayed_case - ~response_delay:0.9 - (response ~status:302 ~headers:"Content-Length: 1\r\nLocation: /next\r\n" "x") - ; delayed_case ~response_delay:0.9 (response "x") - ] - (fun ~environment ~sw ~support ~origin -> - let clock = scaled_clock (Eio.Stdenv.clock environment) 20. in - let posted = ref [] - and invalidations = ref 0 in - let t = runner ~clock ~environment ~sw ~support ~posted ~invalidations () in - Runner.submit t (operation ~download:false origin); - wait (Eio.Stdenv.clock environment) (fun () -> - Option.is_some (completed !posted)); - Alcotest.(check bool) - "redirect does not reset response deadline" - true - (Result.is_error (Option.get (completed !posted))); - Runner.shutdown t) - in - check_retired report -;; - -let scenarios = - scenarios - @ [ Alcotest.test_case - "early frames drain bounded backpressure" - `Quick - test_early_frame_backpressure - ; Alcotest.test_case - "segmented early frames control" - `Quick - test_segmented_backpressure_control - ; Alcotest.test_case - "TLS legal slow handshake succeeds" - `Quick - test_tls_slow_legal_handshake - ; Alcotest.test_case - "dial TLS and response deadlines are independent" - `Quick - test_phased_response_deadline - ; Alcotest.test_case - "DNS TCP address attempts share dial deadline" - `Quick - test_shared_dial_deadline - ; Alcotest.test_case - "TLS independent ten second deadline" - `Quick - test_tls_independent_deadline - ; Alcotest.test_case - "redirect response deadline remains shared" - `Quick - test_redirect_response_budget_is_shared - ] -;; - -let test_backpressure_close_unblocks_full_event_sink () = - let message = {|{"type":"changed","t":42}|} in - let wire = String.concat "" (List.init 129 (fun _ -> frame 1 message)) in - let report = - with_peer - [ case ~ws:true wire ] - (fun ~environment ~sw ~support ~origin -> - let sink = Eio.Stream.create 1 in - let entered = ref false in - let posted = ref [] - and invalidations = ref 0 in - let t = - runner - ~environment - ~sw - ~support - ~posted - ~invalidations - ~on_post:(fun event -> - (match event with - | Core.Websocket_message _ -> entered := true - | _ -> ()); - Eio.Stream.add sink event) - () - in - let start = websocket_start origin in - let scope = - match start with - | Core.Start_websocket r -> r.scope - | _ -> assert false - in - Runner.submit t start; - wait (Eio.Stdenv.clock environment) (fun () -> - !entered - || List.exists - (function - | Core.Websocket_closed _ -> true - | _ -> false) - !posted); - Alcotest.(check bool) "consumer reached bounded sink before close" true !entered; - let before = Eio.Time.now (Eio.Stdenv.clock environment) in - Runner.submit t (websocket_close scope); - Alcotest.(check bool) - "close submit never joins on the event consumer" - true - (Eio.Time.now (Eio.Stdenv.clock environment) -. before < 0.1); - Eio.Time.with_timeout_exn (Eio.Stdenv.clock environment) 3. (fun () -> - let rec drain () = - match Eio.Stream.take sink with - | Core.Websocket_retired retired when retired = scope -> () - | _ -> drain () - in - drain ()); - let callbacks = List.length !posted in - Eio.Time.sleep (Eio.Stdenv.clock environment) 0.03; - Alcotest.(check int) - "retired owner cannot emit late callbacks" - callbacks - (List.length !posted); - Runner.shutdown t) - in - check_retired report -;; - -let test_message_callback_exception_retires_owner () = - let report = - with_peer - [ case ~ws:true (frame 1 {|{"type":"changed","t":42}|}) ] - (fun ~environment ~sw ~support ~origin -> - let calls = ref 0 in - let posted = ref [] - and invalidations = ref 0 in - let t = - runner - ~environment - ~sw - ~support - ~posted - ~invalidations - ~on_post:(function - | Core.Websocket_message _ -> - incr calls; - failwith "fixture callback failure" - | _ -> ()) - () - in - Runner.submit t (websocket_start origin); - wait (Eio.Stdenv.clock environment) (fun () -> - List.exists - (function - | Core.Websocket_closed _ -> true - | _ -> false) - !posted); - Alcotest.(check int) "failing callback executes once" 1 !calls; - Runner.shutdown t) - in - check_retired report -;; - -type socket_peaks = - { mutable sockets : int - ; mutable socket_peak : int - ; mutable dials : int - ; mutable dial_peak : int - ; mutable owner_peak : int - ; mutable attempts : int - } - -let metered_network ?(fail_first = false) ?(before_read = fun () -> ()) network peaks = - let module Socket = struct - type t = - { flow : [ `Generic | `Unix ] Eio.Net.stream_socket_ty Eio.Resource.t - ; retire : unit -> unit - } - - type tag = - [ `Generic - | `Unix - ] - - let read_methods = [] - let single_read t buffer = - before_read (); - let count = Eio.Flow.single_read t.flow buffer in - before_read (); - count - let single_write t = Eio.Flow.single_write t.flow - let copy t ~src = Eio.Flow.copy src t.flow - let shutdown t = Eio.Flow.shutdown t.flow - let close t = Fun.protect ~finally:t.retire (fun () -> Eio.Resource.close t.flow) - end - in - let module Network = struct - type t = Eio_unix.Net.t - - type tag = - [ `Generic - | `Unix - ] - - let listen net ~reuse_addr ~reuse_port ~backlog ~sw address = - Eio.Net.listen net ~reuse_addr ~reuse_port ~backlog ~sw address - ;; - - let connect net ~sw address = - peaks.attempts <- peaks.attempts + 1; - peaks.dials <- peaks.dials + 1; - peaks.dial_peak <- max peaks.dial_peak peaks.dials; - peaks.owner_peak <- max peaks.owner_peak (peaks.dials + peaks.sockets); - let flow = - Fun.protect - ~finally:(fun () -> peaks.dials <- peaks.dials - 1) - (fun () -> Eio.Net.connect net ~sw address) - in - peaks.sockets <- peaks.sockets + 1; - peaks.socket_peak <- max peaks.socket_peak peaks.sockets; - peaks.owner_peak <- max peaks.owner_peak (peaks.dials + peaks.sockets); - let released = ref false in - let retire () = - if not !released - then ( - released := true; - peaks.sockets <- peaks.sockets - 1) - in - Eio.Switch.on_release sw retire; - if fail_first && peaks.attempts = 1 then raise End_of_file; - Eio.Resource.T (Socket.{ flow; retire }, Eio.Net.Pi.stream_socket (module Socket)) - ;; - - let datagram_socket net ~reuse_addr ~reuse_port ~sw address = - Eio.Net.datagram_socket net ~reuse_addr ~reuse_port ~sw address - ;; - - let getaddrinfo net ~service host = Eio.Net.getaddrinfo net ~service host - let getnameinfo net address = Eio.Net.getnameinfo net address - end - in - Eio.Resource.T (network, Eio.Net.Pi.network (module Network)) -;; - -let test_runtime_single_owner_and_scope_switch ?(eof = false) ?(reset = false) () = - let peer_case wire = - match case ~ws:true wire with - | `Assoc fields -> - `Assoc (("frame_delay", `Float 0.1) :: ("close_delay", `Float 0.1) :: fields) - | _ -> assert false - in - let first_case = - match peer_case (if eof then "" else frame 8 "\x03\xe8") with - | `Assoc fields -> - `Assoc - (("eof", `Bool eof) :: ("reset", `Bool reset) :: List.remove_assoc "eof" fields) - | _ -> assert false - in - let report = - with_peer - [ first_case; peer_case ""; peer_case "" ] - (fun ~environment ~sw ~support ~origin -> - let peaks = - { sockets = 0 - ; socket_peak = 0 - ; dials = 0 - ; dial_peak = 0 - ; owner_peak = 0 - ; attempts = 0 - } - in - let network = metered_network (Eio.Stdenv.net environment) peaks in - let events = Eio.Stream.create 2 in - let posted = ref [] - and invalidations = ref 0 in - let t = - runner - ~network - ~environment - ~sw - ~support - ~posted - ~invalidations - ~on_post:(Eio.Stream.add events) - () - in - let start = websocket_start origin in - let first = - match start with - | Core.Start_websocket r -> r.scope - | _ -> assert false - in - let core = ref (websocket_core first) in - let retry = ref None - and retirement_seen = ref false - and opened = ref [] - and running = ref true in - let apply event = - let transition = Core.step !core event in - core := transition.next; - (match event with - | Core.Websocket_opened s -> opened := s :: !opened - | _ -> ()); - List.iter - (function - | Core.Run runnable -> - (match runnable with - | Core.Schedule_timer timer when timer.delay_seconds < 30. -> - retry := Some timer - | _ -> ()); - Runner.submit t runnable - | Core.Delegate (Core.Inspect_mirror r) -> - Eio.Fiber.fork ~sw (fun () -> - Eio.Stream.add events (Core.Mirror_inspected (Core.Mirror_available r))) - | Core.Delegate (Core.Attach_graph r) -> - let sync = - Logseq_overlay_db.Types.sync_view - ~token: - (Logseq_overlay_db.Types.sync_token_of_string - "sync-token:v1:singleton" - |> Result.get_ok) - ~checkpoint: - (Logseq_overlay_db.Types.Server_cursor.of_int64 0L |> Result.get_ok) - ~submissions:[] - in - Eio.Fiber.fork ~sw (fun () -> - Eio.Stream.add events (Core.Graph_attached { scope = r.scope; sync })) - | _ -> ()) - transition.effects; - match event with - | Core.Websocket_retired scope when scope = first -> retirement_seen := true - | _ -> () - in - Eio.Fiber.fork ~sw (fun () -> - while !running do - let event = Eio.Stream.take events in - apply event - done); - Runner.submit t start; - wait (Eio.Stdenv.clock environment) (fun () -> - Option.is_some !retry || !retirement_seen); - Alcotest.(check bool) - "real peer Close schedules reducer retry" - true - (Option.is_some !retry); - let timer = Option.get !retry in - Eio.Stream.add events (Core.Timer_elapsed timer.id); - Eio.Stream.add events (Core.Timer_elapsed timer.id); - wait (Eio.Stdenv.clock environment) (fun () -> - List.exists - (fun (s : Core.connection_scope) -> - s.connection_generation > first.connection_generation) - !opened); - Eio.Stream.add - events - (Core.Foreground_changed { foreground = true; lifecycle_generation = 1L }); - Eio.Stream.add - events - (Core.Foreground_changed { foreground = true; lifecycle_generation = 2L }); - Eio.Stream.add events (Core.Graph_selected first.graph.graph_id); - wait (Eio.Stdenv.clock environment) (fun () -> - List.exists - (fun (s : Core.connection_scope) -> - s.graph.graph_generation > first.graph.graph_generation) - !opened); - Eio.Stream.add events (Core.Websocket_opened first); - Eio.Stream.add - events - (Core.Websocket_closed (first, Core.Recoverable_close { message = None; attention = Requires_attention })); - Eio.Stream.add - events - (Core.Websocket_message - ( first - , Logseq_sync_pure_reducer.Sync_protocol.Server.Changed - { t = Core_contract.server_cursor_of_int 42 } )); - Eio.Time.sleep (Eio.Stdenv.clock environment) 0.03; - Runner.shutdown t; - running := false; - Eio.Stream.add events Core.Shutdown; - wait (Eio.Stdenv.clock environment) (fun () -> - peaks.sockets = 0 && peaks.dials = 0); - Alcotest.(check int) "actual live TCP socket peak" 1 peaks.socket_peak; - Alcotest.(check int) "connected plus opening socket peak" 1 peaks.owner_peak; - Alcotest.(check int) "actual simultaneous connect attempt peak" 1 peaks.dial_peak; - Alcotest.(check int) - "exactly initial retry and scope replacement dial" - 3 - peaks.attempts) - in - check_retired report -;; - -let scenarios = - scenarios - @ [ Alcotest.test_case - "close drains full callback sink without join deadlock" - `Quick - test_backpressure_close_unblocks_full_event_sink - ; Alcotest.test_case - "callback exception retires WebSocket owner" - `Quick - test_message_callback_exception_retires_owner - ; Alcotest.test_case - "real Runner Core singleton and scope switch" - `Quick - test_runtime_single_owner_and_scope_switch - ] -;; - -let test_runtime_duplicate_opening_has_one_socket () = - let report = - with_peer - [ case ~ws:true "" ] - (fun ~environment ~sw ~support ~origin -> - let peaks = - { sockets = 0 - ; socket_peak = 0 - ; dials = 0 - ; dial_peak = 0 - ; owner_peak = 0 - ; attempts = 0 - } - in - let network = metered_network (Eio.Stdenv.net environment) peaks in - let posted = ref [] - and invalidations = ref 0 in - let t = runner ~network ~environment ~sw ~support ~posted ~invalidations () in - let start = websocket_start origin in - let scope = - match start with - | Core.Start_websocket r -> r.scope - | _ -> assert false - in - let core = ref (websocket_core scope) in - Runner.submit t start; - List.iter - (fun generation -> - let tr = - Core.step - !core - (Core.Foreground_changed - { foreground = true; lifecycle_generation = generation }) - in - core := tr.next; - List.iter - (function - | Core.Run r -> Runner.submit t r - | _ -> ()) - tr.effects) - [ 1L; 2L ]; - Eio.Time.sleep (Eio.Stdenv.clock environment) 0.15; - Alcotest.(check int) - "repeat foreground during opening creates one actual socket" - 1 - peaks.socket_peak; - Alcotest.(check int) "one actual dial attempt admitted" 1 peaks.attempts; - Runner.shutdown t) - in - check_retired report -;; - -let scenarios = - scenarios - @ [ Alcotest.test_case - "duplicate opening actual socket peak" - `Quick - test_runtime_duplicate_opening_has_one_socket - ] -;; - -let test_upgrade_failure_is_typed_terminal status () = - let cases = - List.init (if status = 401 then 2 else 1) (fun _ -> case (response ~status "x")) - in - let report = - with_peer cases (fun ~environment ~sw ~support ~origin -> - let posted = ref [] - and invalidations = ref 0 in - let t = runner ~environment ~sw ~support ~posted ~invalidations () in - Runner.submit t (websocket_start origin); - wait (Eio.Stdenv.clock environment) (fun () -> - List.exists - (function - | Core.Websocket_closed _ -> true - | _ -> false) - !posted); - Alcotest.(check bool) - "authentication failure cannot enter network retry" - true - (List.exists - (function - | Core.Websocket_closed (_, Core.Authentication_failure _) -> true - | _ -> false) - !posted); - Runner.shutdown t) - in - check_retired report -;; - -let scenarios = - scenarios - @ [ Alcotest.test_case - "real Runner Core recovers peer EOF" - `Quick - (test_runtime_single_owner_and_scope_switch ~eof:true) - ; Alcotest.test_case - "real Runner Core recovers TCP reset" - `Quick - (test_runtime_single_owner_and_scope_switch ~eof:true ~reset:true) - ; Alcotest.test_case - "Upgrade 401 typed terminal failure" - `Quick - (test_upgrade_failure_is_typed_terminal 401) - ; Alcotest.test_case - "Upgrade 403 typed terminal failure" - `Quick - (test_upgrade_failure_is_typed_terminal 403) - ] -;; - -let test_shutdown_does_not_wait_for_stopped_event_consumer ~during_open () = - let config = - match case ~ws:true (if during_open then "" else frame 8 "\x03\xe8") with - | `Assoc fields -> `Assoc (("frame_delay", `Float 0.1) :: fields) - | _ -> assert false - in - let report = - with_peer [ config ] (fun ~environment ~sw:_ ~support ~origin -> - let clock = Eio.Stdenv.clock environment in - let result = - Eio.Time.with_timeout clock 2. (fun () -> - Eio.Switch.run (fun owner -> - let sink = Eio.Stream.create 1 in - if during_open then Eio.Stream.add sink Core.Local_feed_acknowledged; - let entered = ref false in - let posted = ref [] - and invalidations = ref 0 in - let t = - runner - ~environment - ~sw:owner - ~support - ~posted - ~invalidations - ~on_post:(fun event -> - (match event with - | Core.Websocket_opened _ when during_open -> entered := true - | Core.Websocket_closed _ when not during_open -> entered := true - | _ -> ()); - Eio.Stream.add sink event) - () - in - Runner.submit t (websocket_start origin); - wait clock (fun () -> !entered); - Runner.shutdown t); - Ok ()) - in - Alcotest.(check bool) - "shutdown joins without draining the stopped consumer's sink" - true - (Result.is_ok result)) - in - check_retired report -;; - -let scenarios = - scenarios - @ [ Alcotest.test_case - "shutdown cancels blocked open callback" - `Quick - (test_shutdown_does_not_wait_for_stopped_event_consumer ~during_open:true) - ; Alcotest.test_case - "shutdown cancels blocked terminal callback" - `Quick - (test_shutdown_does_not_wait_for_stopped_event_consumer ~during_open:false) - ] -;; - -let test_failed_dial_retires_allocated_socket_before_fallback () = - let report = - with_peer - [ `Assoc [ "stall_tls", `Bool true ]; case (response "x") ] - (fun ~environment ~sw ~support ~origin -> - let peaks = - { sockets = 0 - ; socket_peak = 0 - ; dials = 0 - ; dial_peak = 0 - ; owner_peak = 0 - ; attempts = 0 - } - in - let network = - metered_network ~fail_first:true (Eio.Stdenv.net environment) peaks - in - let posted = ref [] - and invalidations = ref 0 in - let t = runner ~network ~environment ~sw ~support ~posted ~invalidations () in - Runner.submit t (operation ~download:false origin); - wait (Eio.Stdenv.clock environment) (fun () -> - Option.is_some (completed !posted)); - Alcotest.(check bool) - "real fallback response succeeds" - true - (Result.is_ok (Option.get (completed !posted))); - Alcotest.(check int) - "failed backend allocation retires before next dial" - 1 - peaks.owner_peak; - Alcotest.(check int) "all actual allocated sockets released" 0 peaks.sockets; - Runner.shutdown t) - in - check_retired report -;; - -let scenarios = - scenarios - @ [ Alcotest.test_case - "failed dial retires allocated socket before fallback" - `Quick - test_failed_dial_retires_allocated_socket_before_fallback - ] -;; - -(* DNS-SD is executed by the public Runner transport, not Core. RFC7050's - IPv4-only name supplies a real absent-family reply. Only DNS is external: - every resulting TCP candidate is redirected to the independent loopback peer. *) -let test_absent_dns_family_preserves_dial_budget () = - let report = - with_peer - [ case (response "x") ] - (fun ~environment ~sw ~support ~origin -> - let real_clock = Eio.Stdenv.clock environment in - let addresses = - Eio.Time.with_timeout_exn real_clock 5. (fun () -> - Eio.Net.getaddrinfo_stream - ~service:"443" - (Eio.Stdenv.net environment) - "ipv4only.arpa") - in - Alcotest.(check bool) - "fixture requires IPv4 without IPv6; DNS64 would invalidate this regression" - true - (addresses <> [] - && List.for_all - (function - | `Tcp (ip, _) -> - Eio.Net.Ipaddr.fold ~v4:(fun _ -> true) ~v6:(fun _ -> false) ip - | `Unix _ -> false) - addresses); - let clock = scaled_clock real_clock 10. in - let attempts = ref 0 in - let port = Uri.port origin |> Option.get in - let network = - delayed_network - ~destination:(fun _ -> `Tcp (Eio.Net.Ipaddr.V4.loopback, port)) - (Eio.Stdenv.net environment) - (fun () -> incr attempts) - in - let posted = ref [] - and invalidations = ref 0 in - let t = - runner ~clock ~network ~environment ~sw ~support ~posted ~invalidations () - in - Fun.protect - ~finally:(fun () -> Runner.shutdown t) - (fun () -> - Runner.submit - t - (operation ~download:false (Uri.with_host origin (Some "ipv4only.arpa"))); - wait (Eio.Stdenv.clock environment) (fun () -> - Option.is_some (completed !posted)); - Alcotest.(check bool) - "usable family completes HTTPS before shared deadline expires" - true - (Result.is_ok (Option.get (completed !posted))); - Alcotest.(check int) "one loopback connection" 1 !attempts)) - in - check_retired report -;; - -let test_unanswered_dns_obeys_shared_dial_deadline () = - with_support (fun support -> - Eio_main.run (fun environment -> - Eio.Switch.run (fun sw -> - let clock = scaled_clock (Eio.Stdenv.clock environment) 30. in - let attempts = ref 0 in - let network = - delayed_network (Eio.Stdenv.net environment) (fun () -> - incr attempts; - failwith "unanswered DNS must not dial") - in - let posted = ref [] - and invalidations = ref 0 in - let t = - runner ~clock ~network ~environment ~sw ~support ~posted ~invalidations () - in - Fun.protect - ~finally:(fun () -> Runner.shutdown t) - (fun () -> - let host = - Printf.sprintf - "logseq-deadline-%d-%d.local" - (Unix.getpid ()) - (Random.bits ()) - in - let before = Eio.Time.now clock in - Runner.submit - t - (operation ~download:false (Uri.of_string ("https://" ^ host))); - wait (Eio.Stdenv.clock environment) (fun () -> - Option.is_some (completed !posted)); - let elapsed = Eio.Time.now clock -. before in - Alcotest.(check bool) - "unanswered families share the unchanged thirty-second window" - true - (elapsed >= 29. && elapsed < 33.); - Alcotest.(check bool) - "no addresses produce an explicit transport failure" - true - (Result.is_error (Option.get (completed !posted))); - Alcotest.(check int) "DNS timeout opens no socket" 0 !attempts)))) -;; - -let scenarios = - scenarios - @ [ Alcotest.test_case - "absent DNS family preserves shared dial budget" - `Quick - test_absent_dns_family_preserves_dial_budget - ; Alcotest.test_case - "unanswered DNS obeys shared dial deadline" - `Quick - test_unanswered_dns_obeys_shared_dial_deadline - ] -;; - - -(* Failure provenance belongs to actual Runner IO. Pure Core is used only to - issue opaque production tickets; a TLS peer originates HTTP/decode failures. *) -let unlock_classification_effect origin = - let initial = Core.config ~managed_sync_origin:origin ~limits:(Core_contract.limits ()) - |> Result.get_ok |> Core.initial |> Result.get_ok - |> fun core -> (Core.step core (Core.Sign_in_requested {account_generation=0;presentation_generation=0})).next in - let encrypted = { graph with encrypted = true } in - let auth = Core.step initial (Core.Account_authenticated {user_id=Some "LOG_SECRET_USER"}) in - let catalog = List.find_map (function - | Core.Run (Core.Request (ticket, Core.Fetch_catalog _)) -> - Some (Core.step auth.next (Core.Runner_completed (Core.Completion (ticket, Ok [encrypted])))) - | _ -> None) auth.effects |> Option.get in - let selected = Core.step catalog.next (Core.Graph_selected encrypted.graph_id) in - let scope = Core.admitted_graph_scope selected.next |> Option.get in - let absent = Core.step selected.next (Core.Mirror_inspected (Core.Mirror_absent scope)) in - let cached = List.find_map (function - | Core.Run (Core.Request (ticket, Core.Load_and_unlock_graph_key _)) -> - Some (Core.step absent.next (Core.Runner_completed (Core.Completion - (ticket, Error (Core.Effect_failed "synthetic cached key absent"))))) - | _ -> None) absent.effects |> Option.get in - let keys = cached in - let request = List.find_map (function - | Core.Run (Core.Request (_, Core.Fetch_e2ee_graph_key _) as request) -> Some request - | _ -> None) keys.effects |> Option.get in - keys, request -;; - -let capture_unlock_classification run = - let path = Filename.temp_file "journal-unlock-classification-" "" in - let saved = Unix.dup Unix.stderr in - let output = Unix.openfile path [Unix.O_WRONLY;Unix.O_TRUNC] 0o600 in - let previous = Sys.getenv_opt "JOURNAL_UNLOCK_DIAGNOSTICS" in - Unix.putenv "JOURNAL_UNLOCK_DIAGNOSTICS" "1"; - Unix.dup2 output Unix.stderr; - Fun.protect ~finally:(fun () -> - flush stderr; Unix.dup2 saved Unix.stderr; Unix.close saved; Unix.close output; - Unix.putenv "JOURNAL_UNLOCK_DIAGNOSTICS" (Option.value previous ~default:""); Sys.remove path) - (fun () -> run (); flush stderr; read_file path) -;; - -let require_unlock_classification logs category = - let has value = - let n = String.length value in - let rec loop i = i+n <= String.length logs - && (String.sub logs i n = value || loop (i+1)) in loop 0 in - Alcotest.(check bool) ("terminal category=" ^ category) true - (has ("failure_category=" ^ category)); - List.iter (fun secret -> Alcotest.(check bool) "sensitive marker excluded" false (has secret)) - ["LOG_SECRET_USER";"LOG_SECRET_PASSWORD";"LOG_SECRET_TOKEN";"LOG_SECRET_BODY"; - "LOG_SECRET_PACKAGE";"LOG_SECRET_WRAPPED"] -;; - -let test_unlock_fetch_classification ~category ~success cases () = - let logs = capture_unlock_classification (fun () -> - ignore (with_peer cases (fun ~environment ~sw ~support ~origin -> - let posted = ref [] and invalidations = ref 0 in - let t = runner ~acquire:(fun _ -> Ok "LOG_SECRET_TOKEN") - ~environment ~sw ~support ~posted ~invalidations () in - let _, request = unlock_classification_effect origin in - Fun.protect ~finally:(fun () -> Runner.shutdown t) (fun () -> - Runner.submit t request; - wait (Eio.Stdenv.clock environment) (fun () -> !posted <> []); - Alcotest.(check bool) "business completion unchanged" success - (List.exists (function - | Core.Runner_completed (Core.Completion (_, result)) -> Result.is_ok result - | _ -> false) !posted))) : Yojson.Basic.t)) in - require_unlock_classification logs category; - if category = "http" then ( - let status = "http_status=503" in - let rec has i = i+String.length status <= String.length logs - && (String.sub logs i (String.length status) = status || has (i+1)) in - Alcotest.(check bool) "numeric HTTP status preserved" true (has 0)) -;; - -let test_unlock_crypto_classification () = - let logs = capture_unlock_classification (fun () -> - with_support (fun support -> Eio_main.run (fun environment -> Eio.Switch.run (fun sw -> - let posted = ref [] and invalidations = ref 0 in - let t = runner ~environment ~sw ~support ~posted ~invalidations () in - let keys, _ = unlock_classification_effect (Uri.of_string "https://example.invalid") in - let graph_key = List.find_map (function - | Core.Run (Core.Request (ticket, Core.Fetch_e2ee_graph_key _)) -> - Some (Core.step keys.next (Core.Runner_completed (Core.Completion (ticket, Ok "LOG_SECRET_WRAPPED")))) - | _ -> None) keys.effects |> Option.get in - let private_key = List.find_map (function - | Core.Run (Core.Request (ticket, Core.Fetch_e2ee_user_keys _)) -> - Some (Core.step graph_key.next (Core.Runner_completed (Core.Completion (ticket, Ok "LOG_SECRET_PACKAGE")))) - | _ -> None) graph_key.effects |> Option.get in - let unlock = Core.step private_key.next (Core.E2ee_password_submitted - (Core.unlock_challenge (Core.state private_key.next).snapshot.unlock |> Option.get, "LOG_SECRET_PASSWORD")) in - let request = List.find_map (function - | Core.Run (Core.Request (_, Core.Unlock_private_key _) as request) -> Some request - | _ -> None) unlock.effects |> Option.get in - Fun.protect ~finally:(fun () -> Runner.shutdown t) (fun () -> - Runner.submit t request; - wait (Eio.Stdenv.clock environment) (fun () -> !posted <> []); - Alcotest.(check bool) "actual secret adapter failure is posted" true - (List.exists (function Core.Runner_completed (Core.Completion (_,Error _)) -> true | _ -> false) !posted)))))) in - require_unlock_classification logs "crypto" -;; - -let unlock_json_wire ?(status="200 OK") ?(content_type="application/json") body = - Printf.sprintf "HTTP/1.1 %s\r\nContent-Type: %s\r\nContent-Length: %d\r\nConnection: close\r\n\r\n%s" - status content_type (String.length body) body -;; - -let scenarios = scenarios @ - [ Alcotest.test_case "unlock failure transport category" `Quick - (test_unlock_fetch_classification ~category:"transport" ~success:false [case ~eof:true ""]) - ; Alcotest.test_case "unlock failure HTTP status category" `Quick - (test_unlock_fetch_classification ~category:"http" ~success:false - [case (unlock_json_wire ~status:"503 Service Unavailable" "LOG_SECRET_BODY")]) - ; Alcotest.test_case "unlock failure content type category" `Quick - (test_unlock_fetch_classification ~category:"decode" ~success:false - [case (unlock_json_wire ~content_type:"text/plain" "LOG_SECRET_BODY")]) - ; Alcotest.test_case "unlock failure JSON decode category" `Quick - (test_unlock_fetch_classification ~category:"decode" ~success:false - [case (unlock_json_wire "LOG_SECRET_BODY")]) - ; Alcotest.test_case "unlock failure success does not retain error" `Quick - (test_unlock_fetch_classification ~category:"none" ~success:true - [case (unlock_json_wire {|{"encrypted-aes-key":"LOG_SECRET_WRAPPED"}|})]) - ; Alcotest.test_case "unlock failure 401 retry success clears category" `Quick - (test_unlock_fetch_classification ~category:"none" ~success:true - [case (unlock_json_wire ~status:"401 Unauthorized" "LOG_SECRET_BODY"); - case (unlock_json_wire {|{"encrypted-aes-key":"LOG_SECRET_WRAPPED"}|})]) - ; Alcotest.test_case "unlock failure crypto category" `Quick test_unlock_crypto_classification - ] -;; - - -let test_unlock_response_structure ~expected ~success body () = - let logs = capture_unlock_classification (fun () -> - ignore (with_peer [case (unlock_json_wire body)] (fun ~environment ~sw ~support ~origin -> - let posted = ref [] and invalidations = ref 0 in - let t = runner ~acquire:(fun _ -> Ok "LOG_SECRET_TOKEN") - ~environment ~sw ~support ~posted ~invalidations () in - let _, request = unlock_classification_effect origin in - Fun.protect ~finally:(fun () -> Runner.shutdown t) (fun () -> - Runner.submit t request; - wait (Eio.Stdenv.clock environment) (fun () -> !posted <> []); - Alcotest.(check bool) "existing strict decode result unchanged" success - (List.exists (function - | Core.Runner_completed (Core.Completion (_,result)) -> Result.is_ok result - | _ -> false) !posted))) : Yojson.Basic.t)) in - let has value = - let n=String.length value in - let rec loop i=i+n<=String.length logs && (String.sub logs i n=value || loop(i+1)) in loop 0 in - Alcotest.(check bool) "required safe structure observation" true (has expected); - List.iter (fun secret -> Alcotest.(check bool) "field names and values excluded" false (has secret)) - ["LOG_SECRET_USER";"LOG_SECRET_PASSWORD";"LOG_SECRET_TOKEN";"LOG_SECRET_BODY"; - "LOG_SECRET_PACKAGE";"LOG_SECRET_WRAPPED";"LOG_SECRET_UNKNOWN_FIELD_NAME"; - "LOG_SECRET_UNKNOWN_FIELD_NAME2"] -;; - -let scenarios=scenarios @ - [ Alcotest.test_case "unlock shape object extra secret field" `Quick - (test_unlock_response_structure ~success:true - ~expected:"json_kind=object known_fields=1 unknown_fields=1 encrypted_key_present=true" - {|{"encrypted-aes-key":"LOG_SECRET_WRAPPED","LOG_SECRET_UNKNOWN_FIELD_NAME":"LOG_SECRET_BODY"}|}) - ; Alcotest.test_case "unlock shape array" `Quick - (test_unlock_response_structure ~success:false - ~expected:"json_kind=array known_fields=0 unknown_fields=0 encrypted_key_present=false" - {|[{"encrypted-aes-key":"LOG_SECRET_WRAPPED"}]|}) - ; Alcotest.test_case "unlock shape string" `Quick - (test_unlock_response_structure ~success:false - ~expected:"json_kind=string known_fields=0 unknown_fields=0 encrypted_key_present=false" - {|"LOG_SECRET_BODY"|}) - ; Alcotest.test_case "unlock shape missing known field" `Quick - (test_unlock_response_structure ~success:false - ~expected:"json_kind=object known_fields=0 unknown_fields=2 encrypted_key_present=false" - {|{"LOG_SECRET_UNKNOWN_FIELD_NAME":"LOG_SECRET_BODY","LOG_SECRET_UNKNOWN_FIELD_NAME2":"LOG_SECRET_WRAPPED"}|}) - ; Alcotest.test_case "unlock shape valid strict object" `Quick - (test_unlock_response_structure ~success:true - ~expected:"json_kind=object known_fields=1 unknown_fields=0 encrypted_key_present=true" - {|{"encrypted-aes-key":"LOG_SECRET_WRAPPED"}|}) - ; Alcotest.test_case "unlock shape duplicate known field" `Quick - (test_unlock_response_structure ~success:false - ~expected:"json_kind=object known_fields=2 unknown_fields=0 encrypted_key_present=true" - {|{"encrypted-aes-key":"LOG_SECRET_WRAPPED","encrypted-aes-key":"LOG_SECRET_BODY"}|}) - ; Alcotest.test_case "unlock shape malformed JSON" `Quick - (test_unlock_response_structure ~success:false - ~expected:"json_kind=invalid known_fields=unavailable unknown_fields=unavailable encrypted_key_present=unavailable" - "LOG_SECRET_BODY") - ; Alcotest.test_case "unlock shape observation has independent byte cap" `Quick - (test_unlock_response_structure ~success:false - ~expected:"json_kind=unavailable known_fields=unavailable unknown_fields=unavailable encrypted_key_present=unavailable" - ("\""^String.make (64*1024+1) 'x'^"\"")) - ] -;; - -(* The private PPX decoder cannot originate its defect in Core. The public - Runner API is the narrowest accessible boundary; strictness is retained. *) -let scenarios = scenarios @ - [ Alcotest.test_case "E2EE DTO declares base64 string" `Quick - (test_unlock_response_structure ~success:true - ~expected:"base64_field_kind=string" - {|{"encrypted-aes-key":"LOG_SECRET_WRAPPED","encrypted-aes-key-base64":"LOG_SECRET_BODY"}|}) - ; Alcotest.test_case "E2EE DTO declares nullable base64" `Quick - (test_unlock_response_structure ~success:true - ~expected:"base64_field_kind=null" - {|{"encrypted-aes-key":"LOG_SECRET_WRAPPED","encrypted-aes-key-base64":null}|}) - ; Alcotest.test_case "E2EE DTO rejects numeric base64" `Quick - (test_unlock_response_structure ~success:false - ~expected:"base64_field_kind=number" - {|{"encrypted-aes-key":"LOG_SECRET_WRAPPED","encrypted-aes-key-base64":7}|}) - ; Alcotest.test_case "E2EE DTO ignores unrelated external fields" `Quick - (test_unlock_response_structure ~success:true - ~expected:"base64_field_kind=string" - {|{"encrypted-aes-key":"LOG_SECRET_WRAPPED","encrypted-aes-key-base64":"LOG_SECRET_BODY","LOG_SECRET_UNKNOWN_FIELD_NAME":false}|}) - ] -;; - - -let test_external_http_contract ?expected_error kind ~success body () = - ignore - (with_peer - [ case (unlock_json_wire body) ] - (fun ~environment ~sw ~support ~origin -> - let initial = - Core.config ~managed_sync_origin:origin ~limits:(Core_contract.limits ()) - |> Result.get_ok - |> Core.initial - |> Result.get_ok - |> fun core -> (Core.step core (Core.Sign_in_requested {account_generation=0;presentation_generation=0})).next - in - let request = - match kind with - | `Catalog -> - let auth = - Core.step - initial - (Core.Account_authenticated { user_id = Some "synthetic-user" }) - in - List.find_map - (function - | Core.Run (Core.Request (_, Core.Fetch_catalog _) as request) -> - Some request - | _ -> None) - auth.effects - |> Option.get - | `Graph_key -> snd (unlock_classification_effect origin) - | `User_keys -> - let keys, _ = unlock_classification_effect origin in - let private_keys = - List.find_map - (function - | Core.Run (Core.Request (ticket, Core.Fetch_e2ee_graph_key _)) -> - Some - (Core.step - keys.next - (Core.Runner_completed - (Core.Completion (ticket, Ok "synthetic wrapped key")))) - | _ -> None) - keys.effects - |> Option.get - in - List.find_map - (function - | Core.Run (Core.Request (_, Core.Fetch_e2ee_user_keys _) as request) -> - Some request - | _ -> None) - private_keys.effects - |> Option.get - in - let posted = ref [] - and invalidations = ref 0 in - let t = runner ~environment ~sw ~support ~posted ~invalidations () in - Fun.protect - ~finally:(fun () -> Runner.shutdown t) - (fun () -> - Runner.submit t request; - wait (Eio.Stdenv.clock environment) (fun () -> !posted <> []); - Alcotest.(check bool) - "actual HTTP JSON decoder completion" - success - (List.exists - (function - | Core.Runner_completed (Core.Completion (_, result)) -> - Result.is_ok result - | _ -> false) - !posted); - Option.iter (fun expected -> - let actual = List.find_map (function - | Core.Runner_completed (Core.Completion (_, Error (Core.Effect_failed message | Core.Crypto_failed (_, message)))) -> Some message - | _ -> None) !posted in - Alcotest.(check (option string)) "safe decoder field cause reaches caller" - (Some expected) actual) expected_error)) - : Yojson.Basic.t) -;; - -let scenarios = - scenarios - @ [ Alcotest.test_case - "HTTP graph key ignores external extensions" - `Quick - (test_external_http_contract - `Graph_key - ~success:true - {|{"encrypted-aes-key":"synthetic","future":{"nested":null}}|}) - ; Alcotest.test_case - "HTTP user keys ignore external extensions" - `Quick - (test_external_http_contract - `User_keys - ~success:true - {|{"public-key":"synthetic","encrypted-private-key":"synthetic","future":{"nested":[1,null]}}|}) - ; Alcotest.test_case - "HTTP catalog ignores nested external extensions" - `Quick - (test_external_http_contract - `Catalog - ~success:true - {|{"graphs":[{"graph-id":"11111111-1111-4111-8111-111111111111","graph-name":"synthetic","graph-ready-for-use?":true,"schema-version":null,"future":{"nested":true}}],"future":[1,null]}|}) - ; Alcotest.test_case - "HTTP graph key validates declared fields with extras" - `Quick - (test_external_http_contract - `Graph_key - ~success:false - {|{"encrypted-aes-key":null,"future":true}|}) - ; Alcotest.test_case - "HTTP graph key still requires declared fields" - `Quick - (test_external_http_contract `Graph_key ~success:false {|{"future":true}|}) - ; Alcotest.test_case - "HTTP user keys validate declared fields with extras" - `Quick - (test_external_http_contract - `User_keys - ~success:false - {|{"public-key":"synthetic","encrypted-private-key":false,"future":true}|}) - ; Alcotest.test_case - "HTTP catalog validates nested declared fields" - `Quick - (test_external_http_contract - `Catalog - ~success:false - {|{"graphs":[{"graph-id":"11111111-1111-4111-8111-111111111111","graph-name":null,"graph-ready-for-use?":true,"future":true}]}|}) - ] -;; - -let test_default_connection_sequence () = - let wire = - response ~headers:"Content-Length: 3\r\nConnection: keep-alive\r\n" "abc" - in - let report = - with_peer [ case wire; case wire ] (fun ~environment ~sw ~support ~origin -> - let posted = ref [] and invalidations = ref 0 in - let t = runner ~environment ~sw ~support ~posted ~invalidations () in - Fun.protect ~finally:(fun () -> Runner.shutdown t) (fun () -> - List.iter (fun download -> - posted := []; - Runner.submit t (operation ~download origin); - wait (Eio.Stdenv.clock environment) (fun () -> - Option.is_some (completed !posted)); - Alcotest.(check bool) "response completes without waiting for peer EOF" true - (Result.is_ok (Option.get (completed !posted)))) - [ false; true ])) - in - Alcotest.(check int) "same Runner opens two separate connections" 2 - (List.length (Yojson.Basic.Util.to_list report)); - check_retired report; - check_default_connection report -;; - -let scenarios = - scenarios - @ [ Alcotest.test_case "HTTP default connection retires between requests" - `Quick test_default_connection_sequence ] -;; - - -(* Core sees opaque transport completion text; its public reducer cannot execute - DNS validation, socket setup, TLS authentication, or Upgrade parsing. These - regressions enter through the real public Runner, with no fabricated result. *) -let transport_failure_message ~websocket posted = - if websocket - then - List.find_map - (function - | Core.Websocket_closed (_, Core.Authentication_failure message) -> - Some (`Authentication, message) - | Core.Websocket_closed (_, Core.Recoverable_close { message = Some message; _ }) -> - Some (`Recoverable, message) - | _ -> None) - posted - else ( - match completed posted with - | Some (Error (Core.Effect_failed message)) -> Some (`Recoverable, message) - | _ -> None) -;; - -let require_transport_failure ~websocket ~authentication ~expected posted invalidations = - match transport_failure_message ~websocket posted with - | None -> Alcotest.fail "transport did not return its existing typed failure" - | Some (kind, message) -> - Alcotest.(check string) "fixed safe transport reason" expected message; - Alcotest.(check bool) - "existing authentication/retry classification retained" - (websocket && authentication) - (kind = `Authentication); - Alcotest.(check int) - "TLS or non-401 Upgrade failure does not refresh tokens" - 0 - invalidations -;; - -let test_tls_setup_reason ~websocket reason () = - with_support (fun support -> - Eio_main.run (fun environment -> - Eio.Switch.run (fun sw -> - let real_clock = Eio.Stdenv.clock environment in - let clock = - if reason = `No_address then scaled_clock real_clock 30. else real_clock - in - let attempts = ref 0 in - let network = - delayed_network (Eio.Stdenv.net environment) (fun () -> - incr attempts; - if reason = `Dial_refused then - raise (Unix.Unix_error (Unix.ECONNREFUSED, "LOG_SECRET_SOCKET", "LOG_SECRET_PATH")); - if reason = `Dial_timeout then raise Eio.Time.Timeout; - failwith - "LOG_SECRET_SOCKET https://LOG_SECRET_HOST/LOG_SECRET_PATH \ - token=LOG_SECRET_TOKEN") - in - let host, detail = - match reason with - | `Invalid_host -> - String.make 64 'x' ^ ".LOG_SECRET_HOST.invalid", "invalid DNS host" - | `No_address -> - ( Printf.sprintf "log-secret-dns-%d-%d.local" (Unix.getpid ()) (Random.bits ()) - , "no network address (DNS resolution timed out)" ) - | `Dial_failed -> "localhost", "all resolved addresses failed: operation raised Failure" - | `Dial_refused -> "localhost", "all resolved addresses failed: connection refused" - | `Dial_timeout -> "localhost", "all resolved addresses failed: dial timed out" - in - let origin = Uri.of_string ("https://" ^ host ^ ":1") in - let posted = ref [] - and invalidations = ref 0 in - let t = - runner - ~clock - ~network - ~acquire:(fun _ -> Ok "LOG_SECRET_TOKEN") - ~environment - ~sw - ~support - ~posted - ~invalidations - () - in - Fun.protect - ~finally:(fun () -> Runner.shutdown t) - (fun () -> - Runner.submit - t - (if websocket - then websocket_start origin - else operation ~download:false origin); - wait real_clock (fun () -> - Option.is_some (transport_failure_message ~websocket !posted)); - let prefix = - if websocket then "WebSocket TLS setup failed: " else "TLS setup failed: " - in - require_transport_failure - ~websocket - ~authentication:false - ~expected:(prefix ^ detail) - !posted - !invalidations; - if websocket then ( - let attention = List.find_map (function - | Core.Websocket_closed (_, Core.Recoverable_close { attention; _ }) -> Some attention - | _ -> None) !posted in - let expected = match reason with - | `No_address | `Dial_refused | `Dial_timeout -> Core.Transient_connection - | `Invalid_host | `Dial_failed -> Core.Requires_attention in - Alcotest.(check bool) "typed transport presentation preserves transient vs unknown" true - (attention = Some expected)); - if List.mem reason [`Dial_failed; `Dial_refused; `Dial_timeout] - then - Alcotest.(check bool) - "real address attempts executed the failing network capability" - true - (!attempts > 0) - else - Alcotest.(check int) "invalid or unresolved host never dials" 0 !attempts)))) -;; - -let test_tls_authentication_reason ~websocket () = - let report = - with_peer - [ case (response "x") ] - (fun ~environment ~sw ~support ~origin -> - let posted = ref [] - and invalidations = ref 0 in - let t = - runner - ~tls_authenticator:(Runner.system_tls_authenticator () |> Result.get_ok) - ~acquire:(fun _ -> Ok "LOG_SECRET_TOKEN") - ~environment - ~sw - ~support - ~posted - ~invalidations - () - in - Fun.protect - ~finally:(fun () -> Runner.shutdown t) - (fun () -> - Runner.submit - t - (if websocket - then websocket_start origin - else operation ~download:false origin); - wait (Eio.Stdenv.clock environment) (fun () -> - Option.is_some (transport_failure_message ~websocket !posted)); - require_transport_failure - ~websocket - ~authentication:true - ~expected: - (if websocket - then "WebSocket TLS authentication failed" - else "TLS authentication failed") - !posted - !invalidations)) - in - check_retired report -;; - -let test_upgrade_rejection_reason () = - let report = - with_peer - [ case (response ~status:503 "LOG_SECRET_RESPONSE_BODY") ] - (fun ~environment ~sw ~support ~origin -> - let posted = ref [] - and invalidations = ref 0 in - let t = - runner - ~acquire:(fun _ -> Ok "LOG_SECRET_TOKEN") - ~environment - ~sw - ~support - ~posted - ~invalidations - () - in - Fun.protect - ~finally:(fun () -> Runner.shutdown t) - (fun () -> - Runner.submit t (websocket_start origin); - wait (Eio.Stdenv.clock environment) (fun () -> - Option.is_some (transport_failure_message ~websocket:true !posted)); - require_transport_failure - ~websocket:true - ~authentication:false - ~expected:"WebSocket Upgrade rejected (HTTP 503)" - !posted - !invalidations)) - in - check_retired report -;; - -let scenarios = - scenarios - @ List.concat_map - (fun websocket -> - let prefix = if websocket then "WebSocket " else "HTTP " in - [ Alcotest.test_case - (prefix ^ "TLS diagnostic invalid host") - `Quick - (test_tls_setup_reason ~websocket `Invalid_host) - ; Alcotest.test_case - (prefix ^ "TLS diagnostic no address") - `Quick - (test_tls_setup_reason ~websocket `No_address) - ; Alcotest.test_case - (prefix ^ "TLS diagnostic failed dial") - `Quick - (test_tls_setup_reason ~websocket `Dial_failed) - ; Alcotest.test_case - (prefix ^ "TLS diagnostic refused errno") - `Quick - (test_tls_setup_reason ~websocket `Dial_refused) - ; Alcotest.test_case - (prefix ^ "TLS diagnostic dial timeout") - `Quick - (test_tls_setup_reason ~websocket `Dial_timeout) - ; Alcotest.test_case - (prefix ^ "TLS diagnostic authentication") - `Quick - (test_tls_authentication_reason ~websocket) - ]) - [ false; true ] - @ [ Alcotest.test_case - "WebSocket Upgrade diagnostic status" - `Quick - test_upgrade_rejection_reason - ] -;; - -(* A real transport table can collide even when pure Core emits correctly scoped - instructions. This peer exercises simultaneous TLS channels and their traffic; - it does not stand in for an authoritative DB transaction or GUI acceptance. *) -module Resident_network_fixture = struct -module C = Core -module R = Runner -module O = Logseq_overlay_db.Types -module U = Logseq_db_types.Graph_types.Uuid -let peer = {resident_peer| -import base64, hashlib, json, os, pathlib, socket, ssl, struct, select, subprocess, sys, threading, time -root = pathlib.Path(sys.argv[1]) -key, cert = root/'ephemeral-key.pem', root/'ephemeral-cert.pem' -subprocess.run(['openssl','req','-x509','-newkey','ec','-pkeyopt','ec_paramgen_curve:P-256', - '-nodes','-keyout',str(key),'-out',str(cert),'-days','1','-subj','/CN=localhost', - '-addext','subjectAltName=DNS:localhost'],check=True,stdout=subprocess.DEVNULL,stderr=subprocess.DEVNULL) -context=ssl.SSLContext(ssl.PROTOCOL_TLS_SERVER);context.load_cert_chain(cert,key) -listener=socket.socket(socket.AF_INET6);listener.setsockopt(socket.IPPROTO_IPV6,socket.IPV6_V6ONLY,1);listener.bind(('::1',0));listener.listen(2) -listener4=socket.socket();listener4.bind(('127.0.0.1',listener.getsockname()[1]));listener4.listen(2) -(root/'ready').write_text(str(listener.getsockname()[1])) -lock=threading.Lock();barrier=threading.Barrier(2);events=[];active=set();peak=0;threads=[] -def note(kind,graph,**values): - with lock: - events.append(dict(kind=kind,graph=graph,**values)) - (root/'peer-progress.json').write_text(json.dumps({'peak_simultaneous_sockets':peak,'events':events},indent=2)+'\n') -def send(conn,value): - data=json.dumps(value,separators=(',',':')).encode();length=len(data) - conn.sendall(bytes([0x81,length]) + data if length<126 else bytes([0x81,126])+struct.pack('!H',length)+data) -def receive(conn): - def exact(n): - data=b'' - while len(data)()) -let require label condition=if not condition then failwith label -let cursor n=Logseq_db_types.Server_cursor.of_int64 (Int64.of_int n) |> Result.get_ok -let graphs : C.graph list=List.map(fun(id,name)->({graph_id=U.of_string id |> Result.get_ok;name; - schema={major=65;minor=33;exact=true};encrypted=false}:C.graph)) - ["11111111-1111-4111-8111-111111111111","Synthetic A";"99999999-9999-4999-8999-999999999999","Synthetic B"] -let startup core : C.startup_scope=let f=(C.state core).snapshot.startup in - {account_generation=f.account_generation;presentation_generation=f.presentation_generation} -let run ()=with_support(fun root-> - let path file=Filename.concat root file in - Eio_main.run(fun env->Eio.Switch.run(fun sw-> - let clock=Eio.Stdenv.clock env in - let wait label predicate=Eio.Time.with_timeout_exn clock 10. (fun()-> - while not(predicate()) do Eio.Time.sleep clock 0.01 done);Printf.printf "PASS %s\n%!" label in - let process=Eio.Process.spawn ~sw (Eio.Stdenv.process_mgr env) ["python3";"-c";peer;root] in - wait "loopback TLS peer ready" (fun()->Sys.file_exists(path "ready")); - let port=int_of_string(read(path "ready")) in - let origin=Uri.of_string(Printf.sprintf "https://localhost:%d" port) in - let limits=C.limits ~maximum_response_bytes:Logseq_db_types.Limits.maximum_response_bytes ~maximum_artifact_bytes:1048576 ~submission_batch_size:32 |> Result.get_ok in - let initial=C.config ~managed_sync_origin:origin ~limits |> Result.get_ok |> C.initial |> Result.get_ok in - let signed=C.step initial (C.Sign_in_requested(startup initial)) in - let auth=C.step signed.next (C.Account_authenticated {user_id=Some "synthetic-local-user"}) in - let catalog=List.find_map(function C.Run(C.Request(ticket,C.Fetch_catalog _))-> - Some(C.step auth.next (C.Runner_completed(C.Completion(ticket,Ok graphs))))|_->None) auth.effects |> Option.get in - let selected=C.step catalog.next (C.Graph_selected(List.hd graphs).graph_id) in - let configured=C.step selected.next (C.Resident_graphs_changed(List.map(fun(g:C.graph)->g.graph_id) graphs)) in - let inspections=selected.effects @ configured.effects in - let current,starts=List.fold_left(fun(core,starts)(graph:C.graph)-> - let scope=C.scope_for_graph core ~graph_id:graph.graph_id |> Option.get in - let request=List.find_map(function C.Delegate(C.Inspect_mirror request) when request.scope=scope ->Some request|_->None) inspections |> Option.get in - let inspected=C.step core (C.Mirror_inspected(C.Mirror_available request)) in - let token=O.sync_token_of_string("sync-token:v1:" ^ U.to_string graph.graph_id) |> Result.get_ok in - let sync=O.sync_view ~token ~checkpoint:(cursor 0) ~submissions:[] in - let attached=C.step inspected.next (C.Graph_attached {scope;sync}) in - attached.next,starts @ List.filter_map(function C.Run(C.Start_websocket _ as runnable)->Some runnable|_->None) attached.effects) - (configured.next,[]) graphs in - require "two real graph connect instructions" (List.length starts=2); - let core=ref current and events=Eio.Stream.create 128 and opened=ref [] and messages=ref [] - and retired=ref [] and io_fences=ref [] and expected_retirement=ref None in - let cert=X509.Certificate.decode_pem(read(path "ephemeral-cert.pem")) |> Result.get_ok in - let fingerprint=X509.Certificate.fingerprint `SHA256 cert in - let authenticator=X509.Authenticator.cert_fingerprint ~time:(fun()->Some(Ptime_clock.now())) ~hash:`SHA256 ~fingerprint in - let secrets=R.secrets ~unlock_private_key:(fun ~managed_sync_origin:_ ~user_id:_ ~password:_ ~private_key_package:_->Error "Unused") - ~unlock_graph_key:(fun ~managed_sync_origin:_ ~user_id:_ ~encrypted_graph_key:_->Error "Unused") - ~load_wrapped_graph_key:(fun ~managed_sync_origin:_ ~user_id:_ ~graph_id:_->Error(R.Wrapped_graph_key_unavailable "Unused")) - ~verify_and_save_wrapped_graph_key:(fun ~managed_sync_origin:_ ~user_id:_ ~graph_id:_ ~encrypted_graph_key:_->Error "Unused") - ~delete_account_secrets:(fun ~managed_sync_origin:_ ~user_id:_->Ok()) |> Result.get_ok in - let crypto=R.crypto ~encrypt_aes_gcm:(fun ~key:_ ~plaintext:_->Error "Unused") - ~decrypt_aes_gcm:(fun ~key:_ ~iv:_ ~ciphertext:_->Error "Unused") |> Result.get_ok in - let deps=R.dependencies ~runtime:(R.runtime ~fork:Eio.Fiber.fork ~sleep:(Eio.Time.sleep clock) |> Result.get_ok) - ~transport:(R.transport ~websocket_liveness:R.Disabled ~tls_authenticator:(R.tls_authenticator authenticator) - ~network:(Eio.Stdenv.net env) ~clock |> Result.get_ok) - ~local_store:(R.local_store ~application_support_directory:root () |> Result.get_ok) - ~artifact_store:(R.artifact_store ~staging_directory:(path "staging") |> Result.get_ok) - ~secrets ~crypto ~id_token_provider:(R.id_token_provider ~acquire:(fun account-> - require "synthetic token stays on loopback" (account.C.managed_sync_origin=origin);Ok "synthetic-local-fixture") - ~invalidate:(fun _ ~token:_->())) |> Result.get_ok in - let runner=R.create ~sw deps ~post:(fun event-> - (match event with C.Websocket_closed(_,C.Recoverable_close{message;_})->Printf.eprintf "CLOSED %s\n%!" (Option.value message ~default:"none") - | Websocket_closed(_,(Authentication_failure message|Protocol_failure message))->Printf.eprintf "CLOSED %s\n%!" message - | _->());Eio.Stream.add events event) |> Result.get_ok in - let run=function - | C.Await_io _ | Start_websocket _ | Send_websocket _ | Close_websocket _ | Cancel_effects _ | Cancel_network_effects _ | Schedule_timer _ as op->R.submit runner op - | Request _ | Asset_io _ | Protected_io _ -> () in - let running=ref true in - Eio.Fiber.fork ~sw (fun()->while !running do - let event=Eio.Stream.take events in - (match event with C.Websocket_opened scope->opened:=scope::!opened - | Websocket_retired scope->retired:=scope::!retired - | IO_fence_completed(id,scope,result)-> - let retired_at_ack=Option.exists(fun expected->List.mem expected !retired)!expected_retirement in - io_fences:=(id,(scope,result,retired_at_ack))::!io_fences - | Websocket_message(scope,message)->messages:=(scope,message)::!messages|_->()); - let next=C.step !core event in core:=next.next; - List.iter(function C.Run instruction->run instruction|_->()) next.effects - done); - List.iter run starts; - wait "two concurrent authenticated TLS WebSockets" (fun()->List.length !opened=2); - wait "baseline pull responses received for both graphs" (fun()->List.filter_map(function (scope:C.connection_scope),Logseq_sync_pure_reducer.Sync_protocol.Server.Pull_ok _->Some scope.graph.graph_id|_->None) !messages |> List.sort_uniq compare |> List.length = 2); - let a=(List.nth graphs 0).graph_id and b=(List.nth graphs 1).graph_id in - require "A remains visible before remote injection" ((C.state !core).snapshot.selected_graph=Some a); - touch(path "inject"); - wait "hidden B receives real server Changed at cursor 1" (fun()->List.exists(function (scope:C.connection_scope),Logseq_sync_pure_reducer.Sync_protocol.Server.Changed {t}->scope.C.graph.graph_id=b && t=cursor 1|_->false) !messages); - require "remote B event does not focus B" ((C.state !core).snapshot.selected_graph=Some a); - Eio.Time.sleep clock 0.05; - let removal=C.step !core (C.Resident_graphs_changed [b]) in core:=removal.next; - List.iter(function C.Run instruction->run instruction|_->()) removal.effects; - let a_scope=List.find(fun(scope:C.connection_scope)->scope.graph.graph_id=a)!opened in - expected_retirement:=Some a_scope; - let io_scope={(C.effect_scope_of_graph a_scope.graph) with - presentation_generation=None;lifecycle_generation=None} in - let fence=C.step !core(C.IO_fence_requested(9001L,io_scope)) in core:=fence.next; - List.iter(function C.Run instruction->run instruction|_->()) fence.effects; - wait "A scope I/O fence finishes while B remains active" (fun()->List.mem_assoc 9001L !io_fences); - require "I/O fence receives exact successful payload only after actual retirement callback" - (List.assoc 9001L !io_fences=(io_scope,Ok(),true)); - wait "closing A retires only its real socket" (fun()->Sys.file_exists(path "retired-A")); - let b_responses ()=List.length(List.filter(function (scope:C.connection_scope),Logseq_sync_pure_reducer.Sync_protocol.Server.Pull_ok _->scope.graph.graph_id=b|_->false) !messages) in - let before=b_responses () in touch(path "after-A"); - wait "B completes a real retry pull/response after A retires" (fun()-> - b_responses ()>before && (try - let progress=path "peer-progress.json" in - let open Yojson.Safe.Util in Yojson.Safe.from_file progress |> member "events" |> to_list - |> List.exists(fun event->member "kind" event=`String "reply-sent" && member "graph" event=`String "B" - && member "phase" event=`String "after-A") with _->false)); - require "B remains open after successful independent roundtrip" (not(Sys.file_exists(path "retired-B"))); - R.shutdown runner;touch(path "stop");running:=false;Eio.Stream.add events C.Shutdown; - Eio.Time.with_timeout_exn clock 10. (fun()->Eio.Process.await_exn process); - let report=Yojson.Safe.from_file(path "peer-report.json") in - let open Yojson.Safe.Util in - require "server observed real overlap" (member "peak_simultaneous_sockets" report |> to_int=2); - require "hidden B sent its own pull after notification" (member "events" report |> to_list |> List.exists(fun event-> - member "graph" event=`String "B" && member "phase" event=`String "after-injection" - && (event |> member "message" |> member "type") = `String "pull")); - print_endline "PASS hidden graph real TLS notification, scoped pull, and independent retirement"; - print_endline "LIMIT transport/core acceptance only; no DB transaction or GUI rendering claimed"))) -;; -end -let scenarios=scenarios @ [Alcotest.test_case - "resident real TLS channels keep hidden pull and retirement independent" - `Quick Resident_network_fixture.run] -;; - -(* The peer close status is owned by frame transport; a reducer receives only - the already-decoded termination and cannot reproduce loss of this status. *) -let test_websocket_close_retains_status () = - List.iter (fun code -> - let payload = String.init 2 (function - | 0 -> Char.chr (code lsr 8) | _ -> Char.chr (code land 255)) - ^ "PRIVATE_GRAPH_CONTENT" in - let report = with_peer [case ~ws:true (frame 8 payload)] - (fun ~environment ~sw ~support ~origin -> - let posted = ref [] and invalidations = ref 0 in - let t = runner ~environment ~sw ~support ~posted ~invalidations () in - Runner.submit t (websocket_start origin); - let closed () = List.find_map (function - | Core.Websocket_closed (_, reason) -> Some reason | _ -> None) !posted in - wait (Eio.Stdenv.clock environment) (fun () -> Option.is_some (closed ())); - let message = match Option.get (closed ()) with - | Core.Protocol_failure message -> Some message - | Core.Recoverable_close {message; _} -> message - | _ -> None in - Alcotest.(check (option string)) "close code survives without peer reason text" - (Some (Printf.sprintf "WebSocket peer closed with status %d" code)) message; - Runner.shutdown t) in - check_retired report) [1000;1008;1011] -;; -let scenarios = scenarios @ [Alcotest.test_case - "WebSocket peer close retains status" `Quick test_websocket_close_retains_status] -;; - -(* The fault occurs after TLS setup in the real transport owner. A reducer only - receives its completed error and cannot reproduce loss of errno/TLS status. *) -let test_transport_runtime_cause ~websocket error expected () = - let armed = ref false and injected = ref false in - let peaks = {sockets=0;socket_peak=0;dials=0;dial_peak=0;owner_peak=0;attempts=0} in - let wire = if websocket then frame 9 "probe" - else response ~headers:"Content-Length: 8192\r\n" (String.make 8192 'x') in - let config = case ~ws:websocket ~segment:16 wire in - let config = if websocket then match config with - | `Assoc fields -> `Assoc (("frame_delay", `Float 0.05) :: fields) | _ -> config - else config in - let report = with_peer [config] - (fun ~environment ~sw ~support ~origin -> - let network = metered_network (Eio.Stdenv.net environment) peaks - ~before_read:(fun () -> if !armed then (injected := true; raise error)) in - let posted = ref [] and invalidations = ref 0 in - let t = runner ~network ~environment ~sw ~support ~posted ~invalidations - ~on_post:(function - | Core.Websocket_opened _ when websocket -> armed := true - | Core.Snapshot_download_progress {received_bytes; _} - when not websocket && received_bytes > 0L -> armed := true - | _ -> ()) () in - Runner.submit t (if websocket then websocket_start origin else operation ~download:true origin); - let failure () = List.find_map (function - | Core.Websocket_closed (_, Core.Recoverable_close {message; _}) -> message - | Core.Runner_completed (Core.Completion (_, Error (Core.Effect_failed message))) -> Some message - | _ -> None) !posted in - wait (Eio.Stdenv.clock environment) (fun () -> Option.is_some (failure ())); - Alcotest.(check bool) "failure occurred after transport setup" true !injected; - let module D = Logseq_db_types.Failure_diagnostic in - let source = Option.get (failure ()) in - let actual = Option.map D.message (D.decode source) in - let operation = if websocket then "websocketTransport" else "httpTransport" in - let expected = D.of_source ~operation ~reason:"networkFailed" expected in - Alcotest.(check (option string)) "transport retains bounded concrete cause" - (Some (D.message expected)) actual; - Runner.shutdown t) in - check_retired report -;; -let scenarios = scenarios @ List.concat_map (fun websocket -> - let prefix = if websocket then "WebSocket" else "HTTP" in - [ Alcotest.test_case (prefix ^ " runtime preserves errno") `Quick - (test_transport_runtime_cause ~websocket - (Unix.Unix_error (Unix.EACCES, "PRIVATE_OPERATION", "PRIVATE_PATH")) - "Unix.Unix_error(Unix.EACCES,PRIVATE_OPERATION,PRIVATE_PATH)") - ]) [false;true] -;; - -let test_transport_tls_cause websocket () = - let config = match case ~ws:websocket "" with - | `Assoc fields -> `Assoc (("corrupt_tls", `Bool true) :: fields) - | _ -> assert false in - let report = with_peer [config] (fun ~environment ~sw ~support ~origin -> - let posted = ref [] and invalidations = ref 0 in - let t = runner ~environment ~sw ~support ~posted ~invalidations () in - Runner.submit t (if websocket then websocket_start origin else operation ~download:false origin); - let failure () = List.find_map (function - | Core.Websocket_closed (_, Core.Recoverable_close {message; _}) -> message - | Core.Runner_completed (Core.Completion (_, Error (Core.Effect_failed message))) -> Some message - | _ -> None) !posted in - wait (Eio.Stdenv.clock environment) (fun () -> Option.is_some (failure ())); - let module D = Logseq_db_types.Failure_diagnostic in - let actual = Option.map D.message (D.decode (Option.get (failure ()))) in - let operation = if websocket then "websocketTransport" else "httpTransport" in - let expected = D.of_source ~operation ~reason:"networkFailed" "TLS failure: bad record mac" in - Alcotest.(check (option string)) "TLS record authentication cause survives transport" - (Some (D.message expected)) actual; - Runner.shutdown t) in - check_retired report -;; -let scenarios = scenarios @ [ - Alcotest.test_case "HTTP runtime preserves TLS record cause" `Quick (test_transport_tls_cause false); - Alcotest.test_case "WebSocket runtime preserves TLS record cause" `Quick (test_transport_tls_cause true)] -;; - -let scenarios = scenarios @ [ - Alcotest.test_case "graph key decoder preserves field cause" `Quick - (test_external_http_contract `Graph_key ~success:false - ~expected_error:"E2EE graph key response: field encrypted-aes-key is missing or has an invalid type" - {|{"encrypted-aes-key":false,"PRIVATE_FIELD":"PRIVATE_CONTENT"}|}); - Alcotest.test_case "user keys decoder preserves field cause" `Quick - (test_external_http_contract `User_keys ~success:false - ~expected_error:"E2EE user keys response: field encrypted-private-key is missing or has an invalid type" - {|{"public-key":"PRIVATE_KEY","encrypted-private-key":false}|}); - Alcotest.test_case "catalog decoder preserves nested field cause" `Quick - (test_external_http_contract `Catalog ~success:false - ~expected_error:"graph catalog: field graph-name is missing or has an invalid type" - {|{"graphs":[{"graph-id":"11111111-1111-4111-8111-111111111111","graph-name":false,"graph-ready-for-use?":true}]}|})] -;; diff --git a/swift/JournalApplicationPlatform.swift b/swift/JournalApplicationPlatform.swift index af983437..fd54cc32 100644 --- a/swift/JournalApplicationPlatform.swift +++ b/swift/JournalApplicationPlatform.swift @@ -70,6 +70,7 @@ import AppKit /// each connect/disconnect transition, in delivery order on the main actor. @ObservationIgnored var requestObserver: ((JournalPlatformWire.Request) -> Void)? @ObservationIgnored var connectionObserver: ((Bool) -> Void)? + @ObservationIgnored var authenticationNotificationCompletionObserver: (() -> Void)? init(services: JournalPlatformServices) { self.services = services } @@ -222,8 +223,10 @@ import AppKit } /// Called after the first local timeline frame or successful native sign-in. - func refreshAuthentication() { - guard !pending.terminating, !accountAuthenticationPresented, services.onlineAllowed else { return } + @discardableResult + func refreshAuthentication() -> Task? { + guard !pending.terminating, !accountAuthenticationPresented, + services.onlineAllowed, !services.signingOut else { return nil } refresh?.cancel() let identity = connection, scope=startupScope refresh = Task { [weak self] in @@ -248,6 +251,7 @@ import AppKit } } } + return refresh } func setAccountAuthenticationPresented(_ presented: Bool) { @@ -362,7 +366,9 @@ import AppKit forName: JournalCognitoSession.changed, object: nil, queue: .main) { @Sendable [weak self] _ in Task { @MainActor [weak self] in guard let self, connection == identity, connected else { return } - refreshAuthentication() + let refresh = refreshAuthentication() + await refresh?.value + authenticationNotificationCompletionObserver?() } } } diff --git a/swift/JournalAssetImport.swift b/swift/JournalAssetImport.swift index ded5e986..777f94e5 100644 --- a/swift/JournalAssetImport.swift +++ b/swift/JournalAssetImport.swift @@ -156,6 +156,11 @@ import UIKit _ = emit(Event(action: "unavailable", request: request, reason: reason)) } + private func emitFailure(_ reason: String, request: Request) { + handled = true + _ = emit(Event(action: "picked-batch", items: [], request: request, error: reason)) + } + private func pick(path: String, title: String, type: String, identity: String) -> Pick { Pick(operation: UUID().uuidString.lowercased(), asset: UUID().uuidString.lowercased(), localMutation: UUID().uuidString.lowercased(), metadataMutation: UUID().uuidString.lowercased(), @@ -272,7 +277,7 @@ import UIKit case .failure(let failure): selection.release() if (failure as NSError).code == NSUserCancelledError { emitDismissed() } - else if isCurrent(request, epoch: epoch) { emitUnavailable("Unable to access the selected files. Please try again. (\(JournalAssetDiagnostic.classify(failure, stage: .fileSelection)))", request: request) } + else if isCurrent(request, epoch: epoch) { emitFailure("Unable to access the selected files. Please try again. (\(JournalAssetDiagnostic.classify(failure, stage: .fileSelection)))", request: request) } } } .photosPicker(isPresented: $photosPresented, selection: $photoItems, @@ -337,7 +342,7 @@ import UIKit #else emitUnavailable("Camera capture is not available on this platform.", request: request) #endif - default: emitUnavailable("Unknown attachment source.", request: request) + default: emitFailure("Unknown attachment source.", request: request) } } .onChange(of: properties?.completion) { _, operation in diff --git a/swift/JournalE2EECrypto.swift b/swift/JournalE2EECrypto.swift index b2666944..f0de7c14 100644 --- a/swift/JournalE2EECrypto.swift +++ b/swift/JournalE2EECrypto.swift @@ -112,6 +112,7 @@ enum JournalE2EECryptoError: Error { case keyStoreFailure(OSStatus) case keyStoreInvalidType case invalidPrivateKey + case unsupportedPrivateKey case rsaDecryptionFailed case invalidGraphKeyLength case invalidAESInput @@ -142,7 +143,7 @@ private struct JournalDERReader { } offset += 1 let length = try readLength() - guard length >= 0, offset + length <= bytes.count else { + guard length >= 0, length <= bytes.count - offset else { throw JournalE2EECryptoError.operationFailed } let result = Data(bytes[offset..<(offset + length)]) @@ -159,11 +160,16 @@ private struct JournalDERReader { guard count > 0, count <= MemoryLayout.size, offset + count <= bytes.count else { throw JournalE2EECryptoError.operationFailed } + guard bytes[offset] != 0 else { throw JournalE2EECryptoError.operationFailed } var result = 0 for _ in 0..= 128 else { throw JournalE2EECryptoError.operationFailed } return result } } @@ -402,13 +408,112 @@ enum JournalE2EECrypto { return try AES.GCM.open(box, using: SymmetricKey(data: key)) } - private static func unwrapPKCS8(_ data: Data) throws -> Data { - var outer = JournalDERReader(bytes: [UInt8](data)) - let sequence = try outer.element(tag: 0x30) - var contents = JournalDERReader(bytes: [UInt8](sequence)) - _ = try contents.element(tag: 0x02) - _ = try contents.element(tag: 0x30) - return try contents.element(tag: 0x04) + private static func rsaPKCS1(_ data: Data) throws -> Data { + do { + var outer = JournalDERReader(bytes: [UInt8](data)) + let sequence = try outer.element(tag: 0x30) + guard outer.offset == outer.bytes.count else { throw JournalE2EECryptoError.invalidPrivateKey } + var contents = JournalDERReader(bytes: [UInt8](sequence)) + let version = try contents.element(tag: 0x02) + guard contents.offset < contents.bytes.count else { throw JournalE2EECryptoError.invalidPrivateKey } + if contents.bytes[contents.offset] == 0x30 { + guard version == Data([0]) else { throw JournalE2EECryptoError.unsupportedPrivateKey } + var algorithm = JournalDERReader(bytes: [UInt8](try contents.element(tag: 0x30))) + let oid = try algorithm.element(tag: 0x06) + guard oid == Data([0x2a, 0x86, 0x48, 0x86, 0xf7, 0x0d, 0x01, 0x01, 0x01]) else { + throw JournalE2EECryptoError.unsupportedPrivateKey + } + if algorithm.offset < algorithm.bytes.count { + guard try algorithm.element(tag: 0x05).isEmpty else { throw JournalE2EECryptoError.invalidPrivateKey } + } + guard algorithm.offset == algorithm.bytes.count else { throw JournalE2EECryptoError.invalidPrivateKey } + let key = try contents.element(tag: 0x04) + if contents.offset < contents.bytes.count { + try validatePrivateKeyAttributes(try contents.element(tag: 0xa0)) + } + guard contents.offset == contents.bytes.count else { throw JournalE2EECryptoError.invalidPrivateKey } + var inner = JournalDERReader(bytes: [UInt8](key)) + let innerSequence = try inner.element(tag: 0x30) + guard inner.offset == inner.bytes.count else { throw JournalE2EECryptoError.invalidPrivateKey } + var integers = JournalDERReader(bytes: [UInt8](innerSequence)) + try validateRSAIntegers(&integers) + return key + } + contents.offset = 0 + try validateRSAIntegers(&contents) + return data + } catch JournalE2EECryptoError.unsupportedPrivateKey { + throw JournalE2EECryptoError.unsupportedPrivateKey + } catch { + throw JournalE2EECryptoError.invalidPrivateKey + } + } + + private static func validatePrivateKeyAttributes(_ data: Data) throws { + guard data.count <= 65_536 else { throw JournalE2EECryptoError.invalidPrivateKey } + func validateOID(_ oid: Data) throws { + guard !oid.isEmpty, oid.last! & 0x80 == 0 else { + throw JournalE2EECryptoError.invalidPrivateKey + } + var atStart = true + for byte in oid { + guard !atStart || byte != 0x80 else { throw JournalE2EECryptoError.invalidPrivateKey } + atStart = byte & 0x80 == 0 + } + } + func validateValues(_ data: Data, depth: Int) throws { + guard depth <= 16 else { throw JournalE2EECryptoError.invalidPrivateKey } + var values = JournalDERReader(bytes: [UInt8](data)) + while values.offset < values.bytes.count { + let tag = values.bytes[values.offset] + guard tag != 0 else { throw JournalE2EECryptoError.invalidPrivateKey } + if tag & 0x1f == 0x1f { + values.offset += 1 + let start = values.offset + var tagNumber = 0 + repeat { + guard values.offset < values.bytes.count, + values.offset - start < 4, + values.offset != start || values.bytes[values.offset] != 0x80 + else { throw JournalE2EECryptoError.invalidPrivateKey } + let byte = values.bytes[values.offset] + tagNumber = tagNumber * 128 + Int(byte & 0x7f) + if byte & 0x80 == 0 { break } + values.offset += 1 + } while true + guard tagNumber >= 31 else { throw JournalE2EECryptoError.invalidPrivateKey } + } + let value = try values.element(tag: values.bytes[values.offset]) + if tag & 0x20 != 0 { + try validateValues(value, depth: depth + 1) + } else if tag == 0x06 { + try validateOID(value) + } + } + } + var attributes = JournalDERReader(bytes: [UInt8](data)) + while attributes.offset < attributes.bytes.count { + var attribute = JournalDERReader(bytes: [UInt8](try attributes.element(tag: 0x30))) + try validateOID(try attribute.element(tag: 0x06)) + let values = try attribute.element(tag: 0x31) + guard attribute.offset == attribute.bytes.count else { + throw JournalE2EECryptoError.invalidPrivateKey + } + try validateValues(values, depth: 0) + } + } + + private static func validateRSAIntegers(_ contents: inout JournalDERReader) throws { + let version = try contents.element(tag: 0x02) + guard version == Data([0]) else { throw JournalE2EECryptoError.unsupportedPrivateKey } + for _ in 0..<8 { + let integer = try contents.element(tag: 0x02) + guard let first = integer.first, first < 0x80, + integer.contains(where: { $0 != 0 }), + integer.count == 1 || first != 0 || integer[integer.startIndex + 1] >= 0x80 + else { throw JournalE2EECryptoError.invalidPrivateKey } + } + guard contents.offset == contents.bytes.count else { throw JournalE2EECryptoError.invalidPrivateKey } } private static func rsaPrivateKey(_ privateKeyData: Data) throws -> SecKey { @@ -417,16 +522,10 @@ enum JournalE2EECrypto { kSecAttrKeyClass: kSecAttrKeyClassPrivate, kSecAttrKeySizeInBits: 4096, ] + let pkcs1 = try rsaPKCS1(privateKeyData) var error: Unmanaged? - var key = SecKeyCreateWithData(privateKeyData as CFData, attributes as CFDictionary, &error) - if key == nil { - error = nil - let pkcs1: Data - do { pkcs1 = try unwrapPKCS8(privateKeyData) } - catch { throw JournalE2EECryptoError.invalidPrivateKey } - key = SecKeyCreateWithData(pkcs1 as CFData, attributes as CFDictionary, &error) - } - guard let key else { throw JournalE2EECryptoError.invalidPrivateKey } + guard let key = SecKeyCreateWithData(pkcs1 as CFData, attributes as CFDictionary, &error) + else { throw JournalE2EECryptoError.invalidPrivateKey } return key } @@ -1143,6 +1242,7 @@ func journalCryptoFailureCode(_ error: any Error) -> String { switch error as? JournalE2EECryptoError { case .invalidRequest: return "cryptoInvalidRequest" case .invalidPrivateKey: return "cryptoInvalidPrivateKey" + case .unsupportedPrivateKey: return "cryptoUnsupportedPrivateKey" case .rsaDecryptionFailed: return "cryptoRSADecryptionFailed" case .invalidGraphKeyLength: return "cryptoInvalidGraphKeyLength" case .invalidAESInput: return "cryptoInvalidAESInput" diff --git a/swift/JournalPlatformServices.swift b/swift/JournalPlatformServices.swift index ccc94f8c..d88ff3d7 100644 --- a/swift/JournalPlatformServices.swift +++ b/swift/JournalPlatformServices.swift @@ -236,7 +236,7 @@ struct JournalLocalAccount: Equatable, Sendable { private(set) var timelinePresented = false private(set) var onlineAllowed = false private var authenticationGeneration: UInt64 = 0 - private var signingOut = false + private(set) var signingOut = false private(set) var accountIssue: JournalAccountIssue? private var preparedUserID: String? diff --git a/test/account_lifecycle/account_test.ml b/test/account_lifecycle/account_test.ml new file mode 100644 index 00000000..00a79edd --- /dev/null +++ b/test/account_lifecycle/account_test.ml @@ -0,0 +1,371 @@ +module Account = Journal_sync_lifecycle.Account +module Dsl = Sync_dsl.Core +module Graph = Logseq_db_types.Managed_graph +module Uuid = Logseq_db_types.Graph_types.Uuid + +let ok = function + | Ok value -> value + | Error error -> failwith (Dsl.error_message error) +;; + +let check label condition = if not condition then failwith label + +let rejected label = function + | Error _ -> () + | Ok _ -> failwith label +;; + +let graph suffix = + { Graph.graph_id = + Uuid.of_string ("00000000-0000-0000-0000-00000000000" ^ suffix) |> Result.get_ok + ; name = suffix + ; schema = { major = 1; minor = 0; exact = true } + ; encrypted = false + } +;; + +let first = graph "1" +let second = graph "2" + +let with_root run = + let root = Filename.temp_file "account-lifecycle" "" in + Sys.remove root; + Unix.mkdir root 0o700; + Fun.protect + ~finally:(fun () -> + Array.iter (fun name -> Sys.remove (Filename.concat root name)) (Sys.readdir root); + Unix.rmdir root) + (fun () -> run { Account.origin = "https://example.org"; account = "user"; root }) +;; + +let fixture ?(maximum_services = 1) ?(fetch = fun () -> Ok [ first; second ]) run = + with_root (fun account -> + let events = ref [] in + let fail_close = ref false in + let on_start = ref (fun () -> ()) in + let add value = events := !events @ [ value ] in + let controller = + Account.create + ~maximum_services + ~catalog:(fun ~account:_ ~check_current ~gate:_ -> + let result = fetch () in + match check_current () with + | Ok () -> result + | Error _ as error -> error) + ~start: + (fun + ~account:_ + ~account_generation:_ + ~graph_generation:_ + ~graph + ~upload_store:_ + ~network:_ + ~on_change:_ -> + add ("start:" ^ graph.Graph.name); + !on_start (); + Ok + (Account.service + ~set_network:(fun gate -> + add + (match gate with + | Dsl.Closed -> "closed" + | Dsl.Open _ -> "open"); + Ok ()) + ~close:(fun () -> + add "close"; + if !fail_close then Error (Dsl.Storage_failure "held") else Ok ()))) + ~delete_local:(fun ~account:_ ~graph:_ -> + add "delete"; + Ok ()) + ~on_change:(fun _ _ -> add "change") + |> ok + in + ok (Account.replace controller account); + run controller account events fail_close on_start; + fail_close := false; + ok (Account.logout controller)) +;; + +let selection () = + fixture (fun controller _ events _ _ -> + ignore (ok (Account.refresh controller)); + ok (Account.select controller first.graph_id); + ok (Account.select controller first.graph_id); + check "one start" (!events = [ "start:1" ]); + ok (Account.deselect controller); + check "deselected" (Account.selected controller = None); + check "bounded warm registry" (Account.service_count controller = 1); + ok (Account.set_network controller (Dsl.Open { authorization_epoch = 1L })); + ok (Account.delete_local controller first.graph_id); + check "close precedes deletion" (!events = [ "start:1"; "open"; "close"; "delete" ]); + check "removed" (Account.service_count controller = 0); + rejected "deleted graph cannot restart" (Account.select controller first.graph_id)) +;; + +let local_restoration () = + fixture ~maximum_services:2 ~fetch:(fun () -> failwith "offline restore fetched catalog") + (fun controller _ events fail_close _ -> + ok (Account.restore_catalog controller [first;second]); + ok (Account.select controller first.graph_id); + ok (Account.select controller second.graph_id); + check "local residents start without network" (!events = ["start:1";"start:2"]); + fail_close := true; + rejected "failed retirement retains resident" (Account.release_graph controller first.graph_id); + check "failed retirement retains ownership" (Account.service_count controller = 2); + fail_close := false; + ok (Account.release_graph controller first.graph_id); + check "host close retains catalog" (List.length (Account.catalog controller) = 2); + check "other resident remains selected" (Account.selected controller = Some second.graph_id); + check "host close releases service" (Account.service_count controller = 1); + ok (Account.select controller first.graph_id); + check "closed graph can be reopened" (Account.service_count controller = 2)) +;; + +let retirement_failure () = + fixture (fun controller account events fail_close _ -> + ignore (ok (Account.refresh controller)); + ok (Account.select controller first.graph_id); + fail_close := true; + rejected "delete blocked" (Account.delete_local controller first.graph_id); + check "no delete" (not (List.mem "delete" !events)); + rejected "logout blocked" (Account.logout controller); + rejected "replacement blocked" (Account.replace controller account); + rejected "revoked catalog" (Account.refresh controller); + check "retained registry" (Account.service_count controller = 1); + check + "store retained" + (Sys.file_exists (Filename.concat account.root "asset-upload-intents.sqlite")); + fail_close := false; + ok (Account.logout controller); + ok (Account.replace controller account)) +;; + +let stale_refresh () = + let during_fetch = ref (fun () -> ()) in + fixture + ~fetch:(fun () -> + !during_fetch (); + Ok [ first; second ]) + (fun controller account _ _ _ -> + (during_fetch + := fun () -> + ok (Account.logout controller); + ok (Account.replace controller account)); + rejected "old account result" (Account.refresh controller); + check "no stale catalog" (Account.catalog controller = [])) +;; + +let refresh_order () = + let during_fetch = ref (fun () -> ()) in + fixture + ~fetch:(fun () -> + !during_fetch (); + Ok [ first ]) + (fun controller _ _ _ _ -> + (during_fetch + := fun () -> + (during_fetch := fun () -> ()); + ignore (ok (Account.refresh controller))); + rejected "older request" (Account.refresh controller); + check "new catalog retained" (List.length (Account.catalog controller) = 1)) +;; + +let reentry () = + fixture (fun controller _ _ _ on_start -> + ignore (ok (Account.refresh controller)); + (on_start := fun () -> rejected "start reentry" (Account.logout controller)); + ok (Account.select controller first.graph_id)) +;; + +let network_refresh () = + let during_fetch = ref (fun () -> ()) in + fixture + ~fetch:(fun () -> + !during_fetch (); + Ok [ first ]) + (fun controller _ _ _ _ -> + ok (Account.set_network controller (Dsl.Open { authorization_epoch = 1L })); + (during_fetch := fun () -> ok (Account.set_network controller Dsl.Closed)); + rejected "network revocation" (Account.refresh controller)) +;; + +let multiple_services () = + fixture ~maximum_services:2 (fun controller account events _ _ -> + ignore (ok (Account.refresh controller)); + ok (Account.select controller first.graph_id); + ok (Account.select controller second.graph_id); + check "two resident services" (Account.service_count controller = 2); + ok (Account.replace controller account); + check + "both services physically closed" + (!events = [ "start:1"; "start:2"; "close"; "close" ]); + check "replacement registry empty" (Account.service_count controller = 0); + check "replacement selection empty" (Account.selected controller = None)) +;; + +let bounded_replacement () = + fixture (fun controller _ events fail_close _ -> + ignore (ok (Account.refresh controller)); + ok (Account.select controller first.graph_id); + fail_close := true; + rejected + "replacement waits for retirement" + (Account.select controller second.graph_id); + check "no overlapping start" (!events = [ "start:1"; "close" ]); + check "failed retirement counted" (Account.service_count controller = 1); + fail_close := false; + ok (Account.select controller second.graph_id); + check + "close precedes replacement start" + (!events = [ "start:1"; "close"; "close"; "start:2" ]); + check "bounded after replacement" (Account.service_count controller = 1)) +;; + +let account_store () = + with_root (fun account -> + let stores = ref [] in + let closed = ref 0 in + let controller = + Account.create + ~maximum_services:2 + ~catalog:(fun ~account:_ ~check_current:_ ~gate:_ -> Ok [ first; second ]) + ~start: + (fun + ~account:_ + ~account_generation:_ + ~graph_generation:_ + ~graph:_ + ~upload_store + ~network:_ + ~on_change:_ -> + stores := upload_store :: !stores; + Ok + (Account.service + ~set_network:(fun _ -> Ok ()) + ~close:(fun () -> + incr closed; + Ok ()))) + ~delete_local:(fun ~account:_ ~graph:_ -> Ok ()) + ~on_change:(fun _ _ -> ()) + |> ok + in + Fun.protect + ~finally:(fun () -> ok (Account.logout controller)) + (fun () -> + ok (Account.replace controller account); + ignore (ok (Account.refresh controller)); + ok (Account.select controller first.graph_id); + ok (Account.select controller second.graph_id); + let previous = List.hd !stores in + check "one store per account" (previous == List.nth !stores 1); + ok (Account.replace controller account); + check "old services retired before reopening store" (!closed = 2); + ignore (ok (Account.refresh controller)); + ok (Account.select controller first.graph_id); + check "replacement store has a fresh owner" (previous != List.hd !stores))) +;; + +let duplicate_catalog () = + fixture + ~fetch:(fun () -> Ok [ first; first ]) + (fun controller _ _ _ _ -> + rejected "duplicate catalog" (Account.refresh controller); + check "atomic catalog publication" (Account.catalog controller = [])) +;; + +let http_catalog () = + with_root (fun account -> + let module Client = Logseq_sync_transport.Http_client in + let module Native = Logseq_sync_native_transport.Http_eio in + let calls = ref 0 in + let refreshes = ref [] in + let current = ref true in + let gate = ref (Dsl.Open { authorization_epoch = 7L }) in + let status = ref 200 in + let body = + ref + {|{"graphs":[{"graph-ready-for-use?":false},{"graph-ready-for-use?":true,"graph-id":"00000000-0000-0000-0000-000000000001","graph-name":"One","schema-version":"2.3"}]}|} + in + let content_type = ref "application/json" in + let after_request = ref (fun () -> ()) in + let backend = + { Client.perform = + (fun request -> + incr calls; + check + "catalog endpoint" + (Uri.path request.Logseq_sync_native_transport.Http.uri = "/graphs"); + check + "authorization" + (List.mem_assoc "Authorization" request.headers + || List.mem_assoc "authorization" request.headers); + !after_request (); + Ok + { Native.status = (if !calls = 1 then 401 else !status) + ; headers = [ "content-type", !content_type ] + ; body = !body + }) + ; download = + (fun ~request:_ ~destination:_ ~maximum_bytes:_ -> failwith "not a download") + } + in + let fetch = + Account.http_catalog ~backend ~token:(fun ~account:_ ~epoch ~refresh -> + check "authorization epoch" (epoch = 7L); + refreshes := !refreshes @ [ refresh ]; + Ok "token") + in + let run () = + fetch + ~account + ~check_current:(fun () -> if !current then Ok () else Error Dsl.Stale_context) + ~gate:(fun () -> !gate) + in + let graphs = ok (run ()) in + check "one auth refresh" (!refreshes = [ false; true ]); + check "ready graphs only" (List.length graphs = 1); + let decoded = List.hd graphs in + check + "native catalog decoding" + (decoded.encrypted + && decoded.schema.major = 2 + && decoded.schema.minor = 3 + && decoded.schema.exact); + status := 404; + rejected "404 is not empty catalog" (run ()); + status := 200; + body := "not JSON"; + rejected "malformed JSON" (run ()); + body := "{\"graphs\":[]}"; + content_type := "text/html"; + rejected "content type" (run ()); + content_type := "application/json"; + (after_request := fun () -> current := false); + rejected "stale HTTP result" (run ()); + current := true; + (after_request := fun () -> gate := Dsl.Closed); + rejected "network closed during HTTP" (run ()); + let before = !calls in + rejected "closed network admission" (run ()); + check "no request while closed" (!calls = before)) +;; + +let () = + List.iter + (fun (name, run) -> + run (); + Printf.printf "PASS %s\n%!" name) + [ "selection and checked deletion", selection + ; "offline restore and host retirement", local_restoration + ; "retirement failure", retirement_failure + ; "stale account catalog", stale_refresh + ; "catalog request ordering", refresh_order + ; "mutation reentry", reentry + ; "network revocation", network_refresh + ; "multiple service retirement", multiple_services + ; "bounded replacement", bounded_replacement + ; "account-owned upload store", account_store + ; "duplicate catalog", duplicate_catalog + ; "native HTTP catalog adapter", http_catalog + ] +;; diff --git a/test/account_lifecycle/dune b/test/account_lifecycle/dune new file mode 100644 index 00000000..fc4dc519 --- /dev/null +++ b/test/account_lifecycle/dune @@ -0,0 +1,4 @@ +(tests + (names account_test worker_migration_fixture) + (modules account_test worker_migration_fixture) + (libraries datascript_ocaml.types datascript-ocaml-native logseq_journal.sync_lifecycle journal_sync_lifecycle_impl sync_dsl logseq_db_types logseq_sync.transport logseq_sync.native_transport uri unix)) diff --git a/test/account_lifecycle/worker_migration_fixture.ml b/test/account_lifecycle/worker_migration_fixture.ml new file mode 100644 index 00000000..833b2f69 --- /dev/null +++ b/test/account_lifecycle/worker_migration_fixture.ml @@ -0,0 +1,178 @@ +module Account = Journal_sync_lifecycle.Account +module Dsl = Sync_dsl.Core +module Graph = Logseq_db_types.Managed_graph +module Uuid = Logseq_db_types.Graph_types.Uuid + +let check label condition = if not condition then failwith label +let ok = function Ok value -> value | Error error -> failwith (Dsl.error_message error) + +let graph index : Graph.t = + { graph_id = Uuid.of_string (Printf.sprintf "71000000-0000-4000-8000-%012d" index) |> Result.get_ok + ; name = string_of_int index + ; schema = { major = 65; minor = 33; exact = true } + ; encrypted = false + } +;; + +type fixture = + { controller : Account.t + ; account : Account.account + ; starts : (Uuid.t * int64 * int64) list ref + ; store_reused : bool list ref + ; closed : Uuid.t list ref + ; deleted : Uuid.t list ref + ; start_error : Dsl.error option ref + ; close_error : Dsl.error option ref + ; fetches : int ref + } + +let fixture run = + let root = Filename.temp_file "worker-account-migration" "" in + Sys.remove root; + Unix.mkdir root 0o700; + Fun.protect + ~finally:(fun () -> + Array.iter (fun name -> Sys.remove (Filename.concat root name)) (Sys.readdir root); + Unix.rmdir root) + (fun () -> + let account : Account.account = { origin="https://example.invalid"; account="synthetic"; root } in + let starts=ref [] and closed=ref [] and deleted=ref [] in + let store_reused=ref [] and previous_store=ref None in + let start_error=ref None and close_error=ref None and fetches=ref 0 in + let controller=Account.create ~maximum_services:3 + ~catalog:(fun ~account:_ ~check_current ~gate:_ -> + incr fetches; + Result.map (fun () -> List.init 3 (fun index -> graph (index+1))) (check_current ())) + ~start:(fun ~account:_ ~account_generation ~graph_generation ~graph ~upload_store ~network:_ ~on_change:_ -> + starts := !starts @ [graph.Graph.graph_id,account_generation,graph_generation]; + store_reused := !store_reused @ [Option.fold ~none:true ~some:((==) upload_store) !previous_store]; + previous_store := Some upload_store; + match !start_error with + | Some error -> Error error + | None -> Ok (Account.service ~set_network:(fun _ -> Ok ()) ~close:(fun () -> + closed := !closed @ [graph.graph_id]; + match !close_error with Some error -> Error error | None -> Ok ()))) + ~delete_local:(fun ~account:_ ~graph -> deleted := !deleted @ [graph.Graph.graph_id]; Ok ()) + ~on_change:(fun _ _ -> ()) |> ok in + ok (Account.replace controller account); + Fun.protect + ~finally:(fun () -> close_error:=None; ok (Account.logout controller)) + (fun () -> run {controller;account;starts;store_reused;closed;deleted;start_error;close_error;fetches})) +;; + +let startup_failure () = + List.iter (fun error -> fixture (fun fixture -> + ok (Account.restore_catalog fixture.controller [graph 1]); + fixture.start_error := Some error; + check "service startup preserves its concrete cause" + (Account.select fixture.controller (graph 1).graph_id = Error error); + check "failed bootstrap never publishes a service" + (Account.service_count fixture.controller=0 && Account.selected fixture.controller=None); + check "non-network failure does not trigger an automatic retry" (List.length !(fixture.starts)=1); + fixture.start_error := None; + ok (Account.select fixture.controller (graph 1).graph_id); + let _,_,failed_generation=List.hd !(fixture.starts) in + let _,_,opened_generation=List.nth !(fixture.starts) 1 in + check "corrected startup uses a new execution generation" (opened_generation>failed_generation))) + [Dsl.Storage_failure "local mirror is closed"; + Dsl.Protocol_failure "Graph schema 65.29 is not supported"; + Dsl.Crypto_failure "invalid graph key"; + Dsl.Activation_failure (Snapshot_parse_error "unsupported synthetic snapshot schema"); + Dsl.Authoritative_failure Authoritative_database_closed; + Dsl.Authoritative_failure Authoritative_sync_token_conflict; + Dsl.Authoritative_failure Authoritative_cursor_discontinuous; + Dsl.Authoritative_failure (Authoritative_decode_failed "unsupported synthetic transaction")] +;; + +let local_residents () = + List.iter (fun count -> fixture (fun fixture -> + let graphs=List.init count (fun index -> graph (index+1)) in + ok (Account.restore_catalog fixture.controller graphs); + List.iter (fun graph -> ok (Account.select fixture.controller graph.Graph.graph_id)) graphs; + check "offline restoration does not fetch catalog" (!(fixture.fetches)=0); + check "all cached residents started" (Account.service_count fixture.controller=count); + check "last restored graph remains selected" + (Account.selected fixture.controller=Some (List.nth graphs (count-1)).graph_id); + let before= !(fixture.starts) in + List.iter (fun graph -> ok (Account.select fixture.controller graph.Graph.graph_id)) graphs; + check "focus never restarts a warm resident" (!(fixture.starts)=before); + check "all residents borrow the same account store" (List.for_all Fun.id !(fixture.store_reused)))) + [1;2;3] +;; + +let retirement_and_replacement () = + fixture (fun fixture -> + ok (Account.restore_catalog fixture.controller [graph 1;graph 2]); + List.iter (fun graph -> ok (Account.select fixture.controller graph.Graph.graph_id)) [graph 1;graph 2]; + let _,account_generation,_=List.hd !(fixture.starts) in + ok (Account.replace fixture.controller fixture.account); + check "replacement physically closes every resident" + (List.sort_uniq compare !(fixture.closed)=[(graph 1).graph_id;(graph 2).graph_id]); + check "replacement clears selection and services" + (Account.selected fixture.controller=None && Account.service_count fixture.controller=0); + ok (Account.restore_catalog fixture.controller [graph 1]); + ok (Account.select fixture.controller (graph 1).graph_id); + let _,next_account,_=List.nth !(fixture.starts) 2 in + check "replacement does not borrow the retired store" (next_account>account_generation && not(List.nth !(fixture.store_reused) 2)); + ok (Account.logout fixture.controller); + check "logout closes the replacement resident" (List.length !(fixture.closed)=3)) +;; + +let deletion_waits_for_physical_close () = + fixture (fun fixture -> + ok (Account.restore_catalog fixture.controller [graph 1;graph 2]); + List.iter (fun graph -> ok (Account.select fixture.controller graph.Graph.graph_id)) [graph 1;graph 2]; + let error=Dsl.Storage_failure "reader still owns the database" in + fixture.close_error:=Some error; + check "deletion exposes physical close failure" + (Account.delete_local fixture.controller (graph 1).graph_id=Error error); + check "no private deletion before physical close" + (!(fixture.deleted)=[] && Account.service_count fixture.controller=2); + check "unrelated selected graph stays live" (Account.selected fixture.controller=Some (graph 2).graph_id); + fixture.close_error:=None; + ok (Account.delete_local fixture.controller (graph 1).graph_id); + check "exact target is deleted after close" (!(fixture.deleted)=[(graph 1).graph_id]); + check "private deletion does not retire unrelated service" + (Account.service_count fixture.controller=1 && Account.selected fixture.controller=Some (graph 2).graph_id)) +;; + +let revoked_account_blocks_producers () = + fixture (fun fixture -> + ok (Account.restore_catalog fixture.controller [graph 1;graph 2]); + List.iter (fun graph -> ok (Account.select fixture.controller graph.Graph.graph_id)) [graph 1;graph 2]; + let error=Dsl.Storage_failure "hidden resource has not joined" in + fixture.close_error:=Some error; + check "logout reports unsettled cleanup" (Account.logout fixture.controller=Error error); + let starts=List.length !(fixture.starts) and fetches= !(fixture.fetches) in + check "revoked account cannot refresh" (Account.refresh fixture.controller=Error Dsl.Stale_context); + check "revoked account cannot allocate a graph" + (Account.select fixture.controller (graph 3).graph_id=Error Dsl.Stale_context); + check "replacement cannot steal an unsettled owner" (Account.replace fixture.controller fixture.account=Error error); + check "no catalog or graph producer after revocation" + (List.length !(fixture.starts)=starts && !(fixture.fetches)=fetches); + check "unsettled services retain original store ownership" (Account.service_count fixture.controller=2)) +;; + +let target_retirement_preserves_other_execution () = + fixture (fun fixture -> + ok (Account.restore_catalog fixture.controller [graph 1;graph 2]); + List.iter (fun graph -> ok (Account.select fixture.controller graph.Graph.graph_id)) [graph 1;graph 2]; + let before=List.nth !(fixture.starts) 1 in + ok (Account.release_graph fixture.controller (graph 1).graph_id); + check "only target physically closes" (!(fixture.closed)=[(graph 1).graph_id]); + ok (Account.select fixture.controller (graph 2).graph_id); + check "unrelated execution is preserved" (List.length !(fixture.starts)=2 && List.nth !(fixture.starts) 1=before); + ok (Account.select fixture.controller (graph 1).graph_id); + let _,_,old_generation=List.hd !(fixture.starts) in + let _,_,new_generation=List.nth !(fixture.starts) 2 in + check "reopened graph has a fresh physical generation" (new_generation>old_generation)) +;; + +let () = + List.iter (fun (name,run) -> run (); Printf.printf "PASS migrated %s\n%!" name) + ["bootstrap and local restore failures",startup_failure; + "cached residents and warm focus",local_residents; + "all-resident retirement and replacement",retirement_and_replacement; + "private deletion waits for close",deletion_waits_for_physical_close; + "revoked account producer gate",revoked_account_blocks_producers; + "target retirement preserves other execution",target_retirement_preserves_other_execution] diff --git a/test/application_view_test.ml b/test/application_view_test.ml index ca299f8b..755de0a9 100644 --- a/test/application_view_test.ml +++ b/test/application_view_test.ml @@ -101,14 +101,24 @@ let fixture_with_scope (manager:Service.state) = {manager with snapshot={manager.snapshot with resident_scopes=scopes}} ;; +let fixture_account_state : Logseq_db_types.Sync_account.state = + { snapshot = + { sync_phase = Offline; catalog = []; selected_graph = None; resident_graphs = [] + ; applied_server_t = None; timeline_presentation_pending = false; unlock = Unlock_inactive + ; startup = + { authenticated = false; local_access = Local_pending; network_phase = Local_only + ; catalog_loading = false; awaiting_selection = false; restoring_local = false + ; bootstrapping = false; awaiting_e2ee_password = false; failure = None + ; account_generation = 0; graph_generation = 0; presentation_generation = 0 } + ; last_error = None; last_error_attention = None; local_deletion = None } + ; diagnostics = { groups = [] } } +;; + let fixture_coordination ?account ?(version=1L) ?(phase=Logseq_db_worker.Graph_open) (manager:Service.state) registration = - let module C=Logseq_sync_pure_reducer.Core in + let module C=Logseq_db_types.Sync_account in let manager=fixture_with_scope manager in let scopes=manager.snapshot.resident_scopes in - let limits=C.limits ~maximum_response_bytes:Worker_pure_data.maximum_response_bytes - ~maximum_artifact_bytes:1048576 ~submission_batch_size:1 |> Result.get_ok in - let config=C.config ~managed_sync_origin:(Uri.of_string "https://example.invalid") ~limits |> Result.get_ok in - let initial=C.initial config |> Result.get_ok |> C.state in + let initial=fixture_account_state in let residents=List.map(fun(scope:Service.asset_scope)-> let state={initial with snapshot={initial.snapshot with selected_graph=Some scope.graph_id;resident_graphs=List.map(fun(s:Service.asset_scope)->s.graph_id) scopes; @@ -298,7 +308,7 @@ let test_diagnostic_phase_rows_retain_current_error () = let message = "Graph schema 65.29 is not supported. Journal requires schema 65.33 or later." in let snapshot : Service.snapshot = { sync_phase = Failed; catalog = []; resident_graphs=[];resident_scopes=[]; selected_graph = None; applied_server_t = None - ; unlock = Logseq_sync_pure_reducer.Core.Unlock_inactive + ; unlock = Logseq_sync_domain.Core.Unlock_inactive ; timeline_presentation_pending = false; local_deletion = None; last_error_attention = None; last_error = Some message ; startup = { authenticated = true; local_access=Service.Local_available; network_phase=Service.Online_by_user; catalog_loading = false; awaiting_selection = false ; restoring_local = false; bootstrapping = true; awaiting_e2ee_password = false @@ -321,7 +331,7 @@ let test_diagnostic_identity_comes_from_worker () = let actual = uuid "60000000-0000-4000-8000-000000000002" in let snapshot : Service.snapshot = { sync_phase = Offline; catalog = []; resident_graphs=[];resident_scopes=[]; selected_graph = Some selected - ; applied_server_t = None; unlock = Logseq_sync_pure_reducer.Core.Unlock_inactive + ; applied_server_t = None; unlock = Logseq_sync_domain.Core.Unlock_inactive ; timeline_presentation_pending = false ; local_deletion = None; last_error_attention = None; last_error = None ; startup = { authenticated = true; catalog_loading = false @@ -960,7 +970,7 @@ let run_favorites_native_visibility ; resident_graphs=[];resident_scopes=[] ; selected_graph = Some (uuid 900) ; applied_server_t = Some (server_cursor_of_int 0) - ; unlock = Logseq_sync_pure_reducer.Core.Unlock_inactive + ; unlock = Logseq_sync_domain.Core.Unlock_inactive ; timeline_presentation_pending = false ; startup = { authenticated = true; local_access=Service.Local_available; network_phase=Service.Online_by_user @@ -1006,14 +1016,11 @@ let run_favorites_native_visibility startup={manager.snapshot.startup with graph_generation=selected.graph_generation; presentation_generation=Atomic.get status_presentation}}}in let coordination_baseline registration = - let module C=Logseq_sync_pure_reducer.Core in + let module C=Logseq_db_types.Sync_account in let generation=Atomic.get graph_generation in let presentation=max generation(Atomic.get status_presentation) in let scope={scope with graph_generation=generation} in - let limits=C.limits ~maximum_response_bytes:Worker_pure_data.maximum_response_bytes - ~maximum_artifact_bytes:1048576 ~submission_batch_size:1 |> Result.get_ok in - let config=C.config ~managed_sync_origin:scope.account.managed_sync_origin ~limits |> Result.get_ok in - let state=C.initial config |> Result.get_ok |> C.state in + let state=fixture_account_state in let state={state with snapshot={state.snapshot with selected_graph=Some scope.graph_id; resident_graphs=[scope.graph_id];startup={state.snapshot.startup with account_generation=1;graph_generation=generation;presentation_generation=presentation}}} in @@ -1738,7 +1745,7 @@ let run_favorites_native_visibility Atomic.set durable_paths (durable_path :: Atomic.get durable_paths); let asset = A.create ~uuid:source.asset ~source:(Managed None) - ~current_checksum:(Some (Logseq_sync_effect_runner.Asset_codec.checksum bytes)) + ~current_checksum:(Some (Logseq_sync_native_crypto.Asset_codec.checksum bytes)) ~size:(Some (Int64.of_int (String.length bytes))) ~dimensions:None |> Result.get_ok in @@ -1808,14 +1815,12 @@ let run_favorites_native_visibility let store=Drafts.create ~directory:support_directory ~binding:{user_id="fixture";managed_sync_origin="https://example.invalid"} in let recovered=Result.get_ok(Drafts.load store) in - let locked=List.exists(fun(draft:Drafts.draft)-> - draft.graph=G.Uuid.to_string scope.graph_id - && Journal_capture.retry_identity_locked draft.capture - && not(Journal_capture.can_save draft.capture))recovered in + let retained=List.exists(fun(draft:Drafts.draft)-> + draft.graph=G.Uuid.to_string scope.graph_id)recovered in List.iter(fun(draft:Drafts.draft)-> List.iter Journal_asset_import.discard_staged_file (Journal_capture.pending_attachments draft.capture))recovered; - Alcotest.(check bool) "physical Append sees a durable non-editable lock" true locked; + Alcotest.(check bool) "physical Append sees no submitted restart draft" false retained; Atomic.incr capture_append_writes; Alcotest.(check int) "the full attachment-only capture has one root" 0(Rrbvec.length root.children); let captured={page with uuid=uuid 94321;name=string_of_int journal_day; @@ -5213,7 +5218,7 @@ let test_reference_outer_terminals cancel = ; resident_graphs=[];resident_scopes=[] ; selected_graph = Some (uuid 900) ; applied_server_t = Some (server_cursor_of_int 0) - ; unlock = Logseq_sync_pure_reducer.Core.Unlock_inactive + ; unlock = Logseq_sync_domain.Core.Unlock_inactive ; timeline_presentation_pending = false ; startup = { authenticated = true; local_access=Service.Local_available; network_phase=Service.Online_by_user @@ -5789,7 +5794,7 @@ let test_favorite_page_uses_shared_reference_lookup () = let manager : Service.state = { snapshot = { sync_phase = Current; catalog = []; resident_graphs=[];resident_scopes=[]; selected_graph = Some (uuid 9) - ; applied_server_t = Some (server_cursor_of_int 0); unlock = Logseq_sync_pure_reducer.Core.Unlock_inactive + ; applied_server_t = Some (server_cursor_of_int 0); unlock = Logseq_sync_domain.Core.Unlock_inactive ; timeline_presentation_pending = false ; startup = { authenticated = true; local_access=Service.Local_available; network_phase=Service.Online_by_user; catalog_loading = false; awaiting_selection = false ; restoring_local = false; bootstrapping = false; awaiting_e2ee_password = false @@ -6280,41 +6285,6 @@ let test_fixture_exception_cleanup exception_ = check_fixture_worker_idle () ;; -(* Build opaque unlock evidence through the production reducer, including its - actual key-package and attachment owners. No boolean-to-receipt fixture. *) -let synthetic_unlock_prompt graph_id = - let module C = Logseq_sync_pure_reducer.Core in - let limits = C.limits ~maximum_response_bytes:Logseq_db_types.Limits.maximum_response_bytes - ~maximum_artifact_bytes:1048576 ~submission_batch_size:32 |> Result.get_ok in - let initial = C.config ~managed_sync_origin:(Uri.of_string "https://example.invalid") ~limits - |> Result.get_ok |> C.initial |> Result.get_ok in - let permitted = C.step initial (C.Sign_in_requested - { account_generation = 0; presentation_generation = 0 }) in - let auth = C.step permitted.next (C.Account_authenticated { user_id = Some "synthetic-user" }) in - let graph : C.graph = { graph_id; name = "Private Journal"; encrypted = true - ; schema = { major = 65; minor = 33; exact = true } } in - let catalog = List.find_map (function - | C.Run (C.Request (ticket, C.Fetch_catalog _)) -> Some (C.step auth.next - (C.Runner_completed (C.Completion (ticket, Ok [graph])))) - | _ -> None) auth.effects |> Option.get in - let selected = C.step catalog.next (C.Graph_selected graph_id) in - let mirror = List.find_map (function C.Delegate (C.Inspect_mirror request) -> Some request - | _ -> None) selected.effects |> Option.get in - let local = C.step selected.next (C.Mirror_inspected (C.Mirror_available mirror)) in - let prepare = List.find_map (function - | C.Run (C.Request (ticket, C.Load_and_unlock_graph_key _)) -> Some (C.step local.next - (C.Runner_completed (C.Completion (ticket, Error (C.Effect_failed "synthetic missing key"))))) - | _ -> None) local.effects |> Option.get in - let wrapped = List.find_map (function - | C.Run (C.Request (ticket, C.Fetch_e2ee_graph_key _)) -> Some (C.step prepare.next - (C.Runner_completed (C.Completion (ticket, Ok "synthetic-wrapped")))) - | _ -> None) prepare.effects |> Option.get in - List.find_map (function - | C.Run (C.Request (ticket, C.Fetch_e2ee_user_keys _)) -> Some (C.step wrapped.next - (C.Runner_completed (C.Completion (ticket, Ok "synthetic-package")))) - | _ -> None) wrapped.effects |> Option.get -;; - (* Application owns password admission and UI. No public pure root event exists for E2EE; this fixture exercises mounted Application through its public hooks. *) let unlock_fixture ?(worker_failure = false) ?(queue_full = false) ?(fail_init = false) ?(cold_start = false) @@ -6329,9 +6299,7 @@ let unlock_fixture ?(worker_failure = false) ?(queue_full = false) ?(fail_init = Logseq_db_types.Graph_types.Uuid.of_string "00000000-0000-0000-0000-000000000901" |> Result.get_ok in - let module C = Logseq_sync_pure_reducer.Core in - let prompt = synthetic_unlock_prompt graph_id in - let core = ref prompt.next in + let module C = Logseq_db_types.Sync_account in let manager : Service.state = { snapshot = { sync_phase = Offline @@ -6345,7 +6313,7 @@ let unlock_fixture ?(worker_failure = false) ?(queue_full = false) ?(fail_init = ] ; selected_graph = Some graph_id ; applied_server_t = None - ; unlock = (C.state !core).snapshot.unlock + ; unlock = C.Unlock_inactive ; timeline_presentation_pending = false ; startup = { authenticated = true; local_access=Service.Local_available; network_phase=Service.Online_by_user @@ -6357,7 +6325,7 @@ let unlock_fixture ?(worker_failure = false) ?(queue_full = false) ?(fail_init = ; failure = None ; account_generation = 1 ; graph_generation = 1 - ; presentation_generation = (C.state !core).snapshot.startup.presentation_generation + ; presentation_generation = 2 } ; last_error_attention = None; last_error = None ; local_deletion = None @@ -6426,21 +6394,21 @@ let unlock_fixture ?(worker_failure = false) ?(queue_full = false) ?(fail_init = ; phase = Atomic.get graph_phase ; error = None }) - | Client_command (Submit_e2ee_password _) when worker_failure -> + | Client_command (Submit_native_password _) when worker_failure -> Error "Synthetic unlock worker failure" - | Client_command (Submit_e2ee_password (challenge, password)) -> - let unlocking = C.step !core (C.E2ee_password_submitted (challenge, password)) in - core := unlocking.next; + | Client_command (Submit_native_password (scope, password)) -> + Alcotest.(check bool) "native password uses selected resident owner" true + (List.mem scope (Atomic.get current).snapshot.resident_scopes); Atomic.set submissions (password :: Atomic.get submissions); let old = Atomic.get current in publish { old with snapshot = { old.snapshot with - unlock = (C.state !core).snapshot.unlock + unlock = C.Unlock_inactive ; startup = { old.snapshot.startup with - awaiting_e2ee_password = false + awaiting_e2ee_password = true ; failure = None } ; last_error_attention = None; last_error = None @@ -6454,26 +6422,9 @@ let unlock_fixture ?(worker_failure = false) ?(queue_full = false) ?(fail_init = Eio.Time.Mono.sleep (W.Request_context.clock context) 0.01 done; if attach_mirror then ( - let private_key = List.find_map (function - | C.Run (C.Request (ticket, C.Unlock_private_key _)) -> Some (C.step !core - (C.Runner_completed (C.Completion (ticket, Ok ())))) - | _ -> None) unlocking.effects |> Option.get in - let graph_key = List.find_map (function - | C.Run (C.Request (ticket, C.Fetch_and_unlock_graph_key request)) -> - Some (C.step private_key.next (C.Runner_completed (C.Completion - (ticket, Ok (C.graph_key_handle ~id:"synthetic-key" ~scope:request.scope))))) - | _ -> None) private_key.effects |> Option.get in - let attach = List.find_map (function C.Delegate (C.Attach_graph request) -> Some request - | _ -> None) graph_key.effects |> Option.get in - let token = Logseq_overlay_db.Types.sync_token_of_string "sync-token:v1:synthetic-unlock" - |> Result.get_ok in - let sync = Logseq_overlay_db.Types.sync_view ~token - ~checkpoint:(server_cursor_of_int 51) ~submissions:[] in - let completed = C.step graph_key.next (C.Graph_attached { scope = attach.scope; sync }) in - core := completed.next; let attached = { old with snapshot = { old.snapshot with - unlock = (C.state !core).snapshot.unlock + unlock = C.Unlock_inactive ; applied_server_t = Some (server_cursor_of_int 51) ; timeline_presentation_pending = true ; startup = { old.snapshot.startup with @@ -6496,17 +6447,11 @@ let unlock_fixture ?(worker_failure = false) ?(queue_full = false) ?(fail_init = (if graph_first then manager_changed else graph_changed) (); Atomic.set attachment_stage 2) else ( - let failed = List.find_map (function - | C.Run (C.Request (ticket, C.Unlock_private_key _)) -> Some (C.step !core - (C.Runner_completed (C.Completion (ticket, - Error (C.Effect_failed "Could not unlock the graph. Check your encryption password and try again."))))) - | _ -> None) unlocking.effects |> Option.get in - core := failed.next; publish { manager with snapshot = { manager.snapshot with - unlock = (C.state !core).snapshot.unlock + unlock = C.Unlock_inactive ; startup = { manager.snapshot.startup with failure = Some During_e2ee } ; last_error = Some @@ -7167,7 +7112,6 @@ let test_application_token_control ~saturated ~failure () = let module W = Logseq_db_worker_lui.Journal_worker in let module R = Logseq_db_worker_lui.Journal_worker_runtime in let module S = Logseq_db_worker_lui.Logseq_db_worker_lui_service in - let module SyncR = Logseq_sync_effect_runner.Effect_runner in let support = Filename.temp_file "journal-token-control-" "" in Sys.remove support; Unix.mkdir support 0o700; @@ -7185,29 +7129,6 @@ let test_application_token_control ~saturated ~failure () = done; Alcotest.(check bool) label true (f ()) in - let crypto = - SyncR.crypto - ~encrypt_aes_gcm:(fun ~key:_ ~plaintext:_ -> Error "disabled") - ~decrypt_aes_gcm:(fun ~key:_ ~iv:_ ~ciphertext:_ -> Error "disabled") - |> Result.get_ok - in - let secrets = - SyncR.secrets - ~unlock_private_key: - (fun - ~managed_sync_origin:_ ~user_id:_ ~password:_ ~private_key_package:_ -> - Error "disabled") - ~unlock_graph_key:(fun ~managed_sync_origin:_ ~user_id:_ ~encrypted_graph_key:_ -> - Error "disabled") - ~load_wrapped_graph_key:(fun ~managed_sync_origin:_ ~user_id:_ ~graph_id:_ -> - Error (SyncR.Wrapped_graph_key_unavailable "disabled")) - ~verify_and_save_wrapped_graph_key: - (fun - ~managed_sync_origin:_ ~user_id:_ ~graph_id:_ ~encrypted_graph_key:_ -> - Error "disabled") - ~delete_account_secrets:(fun ~managed_sync_origin:_ ~user_id:_ -> Ok ()) - |> Result.get_ok - in let overlay = Logseq_overlay_db.Database.dependencies ~epoch_ms:(fun () -> 1700000000000L) @@ -7224,13 +7145,7 @@ let test_application_token_control ~saturated ~failure () = } |> Result.get_ok in - let deps = - S.dependencies - ~overlay - ~secrets - ~crypto - ~tls_authenticator:(SyncR.system_tls_authenticator () |> Result.get_ok) - in + let deps = S.dependencies ~overlay in let config = Logseq_db_worker.Config.create ~application_support_directory:support @@ -7472,7 +7387,7 @@ let test_graph_picker_controls scenario () = catalog = [graph first "Current synthetic graph"; graph second long_name] @ (if resident_scenario then [graph (uuid 3) "Third synthetic graph";graph (uuid 4) "Fourth synthetic graph"] else []) ; selected_graph = Some first; applied_server_t = Some (server_cursor_of_int 0) - ; unlock = Logseq_sync_pure_reducer.Core.Unlock_inactive + ; unlock = Logseq_sync_domain.Core.Unlock_inactive ; timeline_presentation_pending = false ; startup = { authenticated = scenario <> `Offline; local_access=(if scenario=`Retry_local then Service.Local_unavailable Local_restore_failed else Service.Local_available); network_phase=(if scenario = `Offline then Service.Local_only else Service.Online_by_user); catalog_loading = false; awaiting_selection = false ; restoring_local = false; bootstrapping = false; awaiting_e2ee_password = false @@ -7697,7 +7612,8 @@ let test_graph_picker_controls scenario () = publish{previous with snapshot={previous.snapshot with resident_graphs=ids; resident_scopes=List.mapi(fun index id->resident_scope id (index+1))ids}} | Select_graph id | Focus_resident_graph id -> - let generation = if held_close then + let generation = if scenario=`Select then previous.snapshot.startup.graph_generation + else if held_close then (List.find(fun(s:Service.asset_scope)->s.graph_id=id)previous.snapshot.resident_scopes).graph_generation else previous.snapshot.startup.graph_generation + 1 in publish { previous with snapshot = { previous.snapshot with selected_graph = Some id @@ -8130,13 +8046,30 @@ let test_graph_picker_controls scenario () = wait "focused graph presents even when its initial push preceded membership" (fun()-> (Atomic.get current).snapshot.selected_graph=Some second && Atomic.get graph_opens>before) | `Select -> + consume (hooks.extension_event (picker_list ()) "event" + {|{"id":1,"payload":"{\"type\":\"visible_range\",\"first\":0,\"last\":7}"}|}); + wait "initial presence checks complete" (fun () -> Atomic.get inspections >= 2); + let before_selection = Atomic.get inspections in + let before_open = Atomic.get graph_opens in press (node ("graph-picker:" ^ Logseq_db_types.Graph_types.Uuid.to_string second)); wait "name selection reaches worker" (fun () -> List.exists (function | Service.Focus_resident_graph id -> id = second | _ -> false) - (Atomic.get commands)) + (Atomic.get commands)); + wait "selected graph is projected and opens through Application" (fun () -> + (Atomic.get current).snapshot.selected_graph = Some second + && Atomic.get graph_opens > before_open); + wait "selected graph toolbar mounts" (fun () -> + Option.is_some (find "text" "Switch graph")); + press (Option.get (find "text" "Switch graph")); + wait "picker remounts after selection" (fun () -> + Option.is_some (find "accessibility-identifier" "graph-picker-list")); + consume (hooks.extension_event (picker_list ()) "event" + {|{"id":2,"payload":"{\"type\":\"visible_range\",\"first\":0,\"last\":7}"}|}); + wait "selection with reused generations rechecks local presence" (fun () -> + Atomic.get inspections > before_selection) | `Late_graph_cancel -> press (node ("graph-picker:" ^ Logseq_db_types.Graph_types.Uuid.to_string second)); wait "new graph is visible while old cancellation cleanup is held" (fun () -> @@ -8384,6 +8317,29 @@ let test_detail_reconciliation_diagnostics mode = (List.length (complete()).responses) ;; +let test_camera_capability_feedback () = + let module A = Root_state in + let reason = "The camera is not available on this device." in + let draft = A.step (A.create ~graph_generation:7) (Capture_edited "Keep this draft") in + let unarmed = A.step draft (Capture_picker_unavailable reason) in + Alcotest.(check bool) "unarmed feedback is ignored" true (unarmed == draft); + let armed = A.step draft (Capture_picker_requested Journal_asset_import.Camera) in + let unavailable = A.step armed (Capture_picker_unavailable reason) in + Alcotest.(check (list (pair string string))) "capability is not a native bug" [] + (A.diagnostic_rows unavailable); + Alcotest.(check bool) "capability preserves draft" true + (A.capture unavailable = A.capture draft); + Alcotest.(check bool) "capability disarms picker" true + (A.step unavailable (Capture_picker_unavailable reason) == unavailable); + List.iter (fun reason -> + let armed = A.step draft (Capture_picker_requested Journal_asset_import.Camera) in + let failed = A.step armed (Capture_assets_picked ([], Some 1, Some reason)) in + Alcotest.(check bool) (reason ^ " remains a failure") true + (A.diagnostic_rows failed <> [])) + [ "camera-encode.data-unavailable"; "stage-write.no-space" + ; "file-selection.permission-denied"; "Unknown attachment source." ] +;; + (* Root_navigation owns acceptance of feed completions and its issue ledger. Generate the failure through the real graph runtime, then retire its owner. *) let test_integrity_attention_survives_unrelated_read_success () = @@ -8503,6 +8459,7 @@ let () = Alcotest.test_case "exclusive channel response routing" `Quick test_worker_response_routing]) ; ( "diagnostic identity", [ Alcotest.test_case "actual worker identity" `Quick test_diagnostic_identity_comes_from_worker ]) ; ( "diagnostic failure", [ + Alcotest.test_case "camera capability feedback is not a native failure" `Quick test_camera_capability_feedback; Alcotest.test_case "integrity attention survives read success and bounded history" `Quick test_integrity_attention_survives_unrelated_read_success; Alcotest.test_case "retired feed Worker failure is ignored" `Quick (fun () -> test_retired_feed_failure_does_not_publish_diagnostics false); @@ -8776,7 +8733,7 @@ let () = ; "status-rejected" ] ) ; ( "Capture attachment imports" - , [ Alcotest.test_case "durable draft lock failure prevents dispatch" `Quick + , [ Alcotest.test_case "durable draft removal failure prevents dispatch" `Quick (fun () -> let support=Filename.temp_file "journal-draft-dispatch-" "" in Unix.unlink support;Unix.mkdir support 0o700; diff --git a/test/coordination_test_fixture.ml b/test/coordination_test_fixture.ml index 48f7625e..a09a6741 100644 --- a/test/coordination_test_fixture.ml +++ b/test/coordination_test_fixture.ml @@ -2,7 +2,7 @@ module W = Journal_worker_coordination.Worker_api module Read = Worker_pure_read module S = Logseq_db_worker_lui.Logseq_db_worker_lui_service module Id = Journal_worker_ids -module Sync = Logseq_sync_pure_reducer.Core +module Sync = Logseq_sync_domain.Core let check condition message = if not condition then failwith message let uuid s = W.Uuid.of_string s |> Result.get_ok @@ -52,7 +52,20 @@ let fixture ?(graph = graph) ?(graph_generation = 7) ?(limits = limits) epoch = |> fun limits -> Sync.config ~managed_sync_origin:account.managed_sync_origin ~limits |> Result.get_ok in - let state = Sync.initial cfg |> Result.get_ok |> Sync.state in + let module Pure = Logseq_db_worker_pure_reducer.Core in + let worker = + Logseq_db_worker.Config.create + ~application_support_directory:(Filename.get_temp_dir_name ()) + ~target:(Managed_sync { base_url = Uri.to_string account.managed_sync_origin }) + ~compatibility_profile:Logseq_65_33_or_newer + ~response_budget_bytes:limits.response_budget_bytes + ~default_page_size:32 + |> Result.get_ok + in + let state = + Pure.config ~worker ~sync:cfg |> Pure.initial |> Result.get_ok |> Pure.view + |> fun view -> view.sync + in let state = { state with snapshot = diff --git a/test/dune b/test/dune index fb4f9d6e..043ba664 100644 --- a/test/dune +++ b/test/dune @@ -1,11 +1,11 @@ (library (name coordination_test_fixture) (modules coordination_test_fixture) - (libraries logseq_db_worker.typed_read logseq_db_worker.identity datascript_ocaml journal_worker_coordination_impl logseq_journal.worker_coordination uri + (libraries logseq_db_worker.pure_reducer logseq_db_worker.typed_read logseq_db_worker.identity datascript_ocaml journal_worker_coordination_impl logseq_journal.worker_coordination uri logseq_db_types logseq_db_worker logseq_db_worker.lui - logseq_sync.pure_reducer + logseq_sync.domain logseq_sync.asset_transfer logseq_db_worker.typed_data worker_effect_authority)) @@ -16,7 +16,7 @@ logseq_db_types logseq_db_worker logseq_db_worker.lui - logseq_sync.pure_reducer + logseq_sync.domain logseq_sync.asset_transfer logseq_db_worker.typed_data)) (test @@ -24,7 +24,7 @@ (modules journal_platform_test) (libraries app logseq_db_worker.lui - logseq_sync.pure_reducer + logseq_sync.domain logseq_sync.asset_transfer yojson journal_app_business)) @@ -123,7 +123,7 @@ (test (name application_view_test) (modules application_view_test) - (libraries datascript-ocaml-native logseq_db_worker.typed_read logseq_db_worker.identity coordination_test_fixture alcotest app + (libraries datascript_ocaml.types datascript-ocaml-native logseq_db_worker.typed_read logseq_db_worker.identity coordination_test_fixture alcotest app logseq_journal.worker_coordination worker_effect_authority eio logseq_db_types @@ -132,8 +132,8 @@ logseq_db_worker.lui logseq_db_worker.pure_reducer logseq_overlay_db - logseq_sync.effect_runner - logseq_sync.pure_reducer + logseq_sync.native_crypto logseq_sync.assets + logseq_sync.domain logseq_sync.asset_transfer lui mtime.clock rrbvec @@ -310,7 +310,7 @@ (modules worker_coordination_runtime_test) (libraries datascript-ocaml-native logseq_db_worker.typed_read logseq_db_worker.identity logseq_db_storage app journal_worker_coordination_impl logseq_journal.worker_coordination logseq_overlay_db_test_support logseq_db_worker_test_support - logseq_sync.effect_runner logseq_sync.pure_reducer digestif + logseq_sync.native_crypto logseq_sync.assets logseq_sync.domain logseq_sync.asset_transfer digestif datascript_ocaml eio eio_main @@ -332,7 +332,7 @@ (name worker_coordination_pure_test) (modules worker_coordination_pure_test) (libraries datascript-ocaml-native logseq_db_worker.typed_read logseq_db_worker.identity coordination_test_fixture datascript_ocaml journal_worker_coordination_impl logseq_journal.worker_coordination - logseq_db_worker logseq_sync.pure_reducer uri + logseq_db_worker logseq_sync.domain logseq_sync.asset_transfer uri logseq_db_types logseq_db_worker.contract logseq_db_worker.lui diff --git a/test/journal_platform_test.ml b/test/journal_platform_test.ml index 72d18a43..ae7ad73b 100644 --- a/test/journal_platform_test.ml +++ b/test/journal_platform_test.ml @@ -60,7 +60,7 @@ let test_scoped_authentication_preserves_identity () = require (Journal_platform.decode_authenticated_user packet = Ok (Some "synthetic-user")) "native scoped authentication loses its verified user"; require (Journal_platform.decode_authenticated_scope packet = - Ok {Logseq_sync_pure_reducer.Core.account_generation=1; presentation_generation=2}) + Ok {Logseq_sync_domain.Core.account_generation=1; presentation_generation=2}) "native scoped authentication loses its startup identity" ;; diff --git a/test/journal_routes_test.ml b/test/journal_routes_test.ml index 7aece598..20760e1a 100644 --- a/test/journal_routes_test.ml +++ b/test/journal_routes_test.ml @@ -5730,14 +5730,62 @@ let test_restart_draft_store () = require(List.length(Result.get_ok(S.load store))=1) "failed write destroyed last good snapshot"; let interrupted=Journal_capture.begin_submission capture in Result.get_ok(S.save store [{item with parent=Some graph;capture=interrupted}]); - let restored=List.hd(Result.get_ok(S.load store)) in - require(restored.parent=Some graph) "append parent not retained"; - require(not(Journal_capture.can_save restored.capture)) "interrupted save became a new mutation"; - let edited=Journal_capture.update_source restored.capture ~source:"must not become editable" in - require(Journal_capture.source edited="unsaved 中文") "interrupted recovery allowed editing an unknown submission"; - let _,retry=Journal_capture.retry edited in - require(retry=None) "interrupted recovery admitted another mutation"; - require(Journal_capture.attachment_imports restored.capture=None) "execution authority survived restart"; + require(Result.get_ok(S.load store)=[]) "accepted Send survived as a restart draft"; + require(Journal_capture.source interrupted="unsaved 中文") "draft deletion destroyed submission text"; + List.iter (fun parent -> + let sent,rejected=match parent with + |None->let sent=Journal_capture.begin_submission capture in + sent,Journal_capture.release_rejected_attempt sent + |Some _-> + let detail=Journal_detail.update_child_source(child_identity_detail()) "unsaved 中文" in + let saving,request=child_identity_append detail in + let command=child_command(Option.get request) in + let sent=Option.get(Journal_detail.child_capture saving) in + let rejected=Journal_detail.release_rejected_child saving ~command ~message:"Known rejected" in + require(Journal_detail.child_error rejected=Some "Known rejected") "Append lost rejection feedback"; + let retained=Option.get(Journal_detail.retain_composer saving) in + let retained=Journal_detail.release_rejected_retained retained ~command ~message:"Known rejected" in + Result.get_ok(S.save store [{item with parent;capture=Journal_detail.retained_capture retained}]); + require(Result.get_ok(S.load(S.create ~directory:root ~binding))=[]) + "retained Append checkpoint resurrected a rejected Send"; + sent,Option.get(Journal_detail.child_capture rejected) in + Result.get_ok(S.save store [{item with parent;capture=sent}]); + require(Journal_capture.can_save rejected) "known rejection lost the in-memory retry"; + require(Journal_capture.source rejected="unsaved 中文") "known rejection lost execution text"; + let checkpoint capture=Result.get_ok(S.save store [{item with parent;capture}]) in + let restart ()=Result.get_ok(S.load(S.create ~directory:root ~binding)) in + checkpoint rejected; + require(restart()=[]) "ordinary checkpoint resurrected a rejected Send"; + checkpoint(Journal_capture.rebind rejected ~session_number:20L); + require(restart()=[]) "remount resurrected a rejected Send"; + checkpoint(Journal_capture.update_source rejected ~source:"unsaved 中文"); + require(restart()=[]) "unchanged editor echo resurrected a rejected Send"; + let native text=edit ~session_id:(Journal_capture.session_id rejected) + ~local_revision:1L ~base_document_revision:0L ~text + ~selection_start:0 ~selection_end:0 () in + checkpoint(Journal_capture.apply_text_edit rejected (native "unsaved 中文")); + require(restart()=[]) "selection-only native edit resurrected a rejected Send"; + checkpoint(Journal_capture.apply_text_edit rejected (native "new native draft")); + require(Journal_capture.source(List.hd(restart())).capture="new native draft") + "new native content did not become a recoverable draft"; + checkpoint(Journal_capture.update_source rejected ~source:"new user draft"); + let restored=List.hd(restart()) in + require(restored.parent=parent && Journal_capture.source restored.capture="new user draft") + "explicit new edit did not recover under its Capture/Append owner") [None;Some graph]; + Result.get_ok(S.save store [item]); + let owned=Filename.concat root "JournalDrafts-v1" in + let account=Filename.concat owned (Array.get(Sys.readdir owned)0) in + let manifest=Filename.concat account "state.json" in + let json=Yojson.Safe.from_file manifest in + let rec mark_interrupted = function + | `Assoc fields -> `Assoc(List.map(fun(key,value)-> + key,if key="interrupted" then `Bool true else mark_interrupted value)fields) + | `List values -> `List(List.map mark_interrupted values) + | value -> value in + Yojson.Safe.to_file manifest (mark_interrupted json); + require(Result.get_ok(S.load store)=[]) "legacy interrupted draft was restored"; + let migrated=Yojson.Safe.Util.member "drafts" (Yojson.Safe.from_file manifest) in + require(migrated=`List []) "legacy interrupted source remained on disk"; let media=Filename.temp_file "journal-import-test-" ".bin" in let channel=open_out_bin media in output_string channel "synthetic-media";close_out channel; let selection=`Assoc["operation",`String graph;"asset",`String graph; @@ -5747,6 +5795,13 @@ let test_restart_draft_store () = (`Assoc["action",`String "picked";"selection",selection])) with |Ok(Picked(staged,_))->staged |_->failwith "invalid synthetic pick" in let with_media=Journal_capture.add_attachment capture staged in + List.iter(fun parent-> + let rejected=Journal_capture.begin_submission with_media |> Journal_capture.release_rejected_attempt in + Result.get_ok(S.save store [{item with parent;capture=rejected}]); + require(Result.get_ok(S.load(S.create ~directory:root ~binding))=[]) + "attachment checkpoint resurrected a rejected Send"; + require(Journal_capture.pending_attachments rejected=[staged] && Sys.file_exists media) + "Send cleanup released the in-memory attachment lease") [None;Some graph]; Result.get_ok(S.save store [{item with capture=with_media}]); Unix.unlink media; let recovered=List.hd(Result.get_ok(S.load store)) in @@ -5780,8 +5835,6 @@ let test_restart_draft_store () = let retained=Journal_routes.recover_captures [graph,capture] in let parent,roundtrip=List.hd(Journal_routes.recovery_captures retained) in require(parent=graph && Journal_capture.source roundtrip="unsaved 中文") "append restart adapter lost owner or text"; - let retained=Journal_detail.recover_composer ~parent:graph restored.capture in - require(Journal_detail.retained_pending_submission retained=None) "append restart restored execution authority"; let old_media=Array.to_list(Sys.readdir account) |>List.filter(fun name->String.ends_with ~suffix:".media" name) in let uncertain=S.For_testing.with_directory_sync store diff --git a/test/logseq_overlay_db_boundary_test.ml b/test/logseq_overlay_db_boundary_test.ml index 31a6c8c0..ffaad872 100644 --- a/test/logseq_overlay_db_boundary_test.ml +++ b/test/logseq_overlay_db_boundary_test.ml @@ -106,10 +106,18 @@ let test_package_metadata root = ;; let test_durable_intent_owner root = - require_text root "logseq_overlay_db/spec/database.mli" - ["type local_intent";"type intent_resolution";"val persist_local_intent"; - "val commit_local_intent";"val inspect_local_intent";"val pending_local_intents"; - "val acknowledge_local_intent";"val acknowledge_rejected_local_intent"] + require_text + root + "logseq_overlay_db/spec/database.mli" + [ "type local_intent" + ; "type intent_resolution" + ; "val persist_local_intent" + ; "val commit_local_intent" + ; "val inspect_local_intent" + ; "val pending_local_intents" + ; "val acknowledge_local_intent" + ; "val acknowledge_rejected_local_intent" + ] ;; let test_canonical_modules root = @@ -368,6 +376,37 @@ let test_persistence_codec_ownership root = (files_with_suffixes root "logseq_overlay_db/spec" [ ".mli"; "dune" ]) ;; +let test_checkpoint_checksum_reads_only_memory root = + let source = read_file (path root "logseq_overlay_db/lib/database.ml") in + let lines = String.split_on_char '\n' source in + let rec find = function + | [] -> fail "checkpoint checksum implementation is missing" + | line :: rest -> + if String.starts_with ~prefix:"let check_checkpoint_checksum " line + then check rest + else find rest + and check = function + | [] -> fail "checkpoint checksum implementation is unterminated" + | ";;" :: _ -> () + | line :: rest -> + List.iter + (fun forbidden -> + if contains line forbidden + then fail "checkpoint checksum must not read the graph: %s" forbidden) + [ "Datascript." + ; "Storage." + ; "Session." + ; "authoritative_database" + ; "authoritative_connection" + ; "Authoritative_checksum.initialize" + ; "Authoritative_checksum.recompute" + ; "Authoritative_checksum.advance" + ]; + check rest + in + find lines +;; + let test_install_manifest root install_manifest = let filename = if (not (Filename.is_relative install_manifest)) || Sys.file_exists install_manifest @@ -400,6 +439,7 @@ let () = then failwith "usage: logseq_overlay_db_boundary_test REPOSITORY_ROOT INSTALL_MANIFEST"; let root = Sys.argv.(1) in test_package_metadata root; + test_checkpoint_checksum_reads_only_memory root; test_durable_intent_owner root; test_canonical_modules root; test_public_boundary root; diff --git a/test/source_boundary_test.ml b/test/source_boundary_test.ml index 965e361e..056a1ab9 100644 --- a/test/source_boundary_test.ml +++ b/test/source_boundary_test.ml @@ -117,8 +117,10 @@ let require_documentation_only_guide root relative = ]; require_file root relative; let candidate = path root relative in - if Sys.file_exists candidate && not (Sys.is_directory candidate) - && not (is_single_documentation_comment (read_file candidate)) + if + Sys.file_exists candidate + && (not (Sys.is_directory candidate)) + && not (is_single_documentation_comment (read_file candidate)) then fail "execution guide must contain only one odoc comment: %s" relative ;; @@ -206,9 +208,12 @@ let test_no_jane_street_core_or_base_dependency root = require_text root "app/dune" [ "\n mtime.clock\n" ]; forbid_text root "dune-project" [ "(core ("; "(base (" ]; require_text root "dune-project" [ "(mtime (= 2.1.0))" ]; - List.iter (fun relative -> forbid_text root relative [ "\"core\" {"; "\"base\" {" ]) - [ "logseq_journal.opam"; "logseq_journal.opam.locked" - ; "logseq_db_worker.opam.locked" ]; + List.iter + (fun relative -> forbid_text root relative [ "\"core\" {"; "\"base\" {" ]) + [ "logseq_journal.opam" + ; "logseq_journal.opam.locked" + ; "logseq_db_worker.opam.locked" + ]; require_text root "logseq_journal.opam" [ "\"mtime\" {= \"2.1.0\"}" ] ;; @@ -258,15 +263,7 @@ let test_public_api_only_test_boundaries root = in List.iter (fun relative -> - (* The external negative compiler test intentionally names inaccessible - paths in rejected source strings, using only public spec CMIs. *) - if relative = "logseq_sync/test/test_asset_cache.ml" - then - forbid_text - root - relative - [ "logseq_sync/lib/effect_runner/."; "logseq_sync/lib/pure_reducer/." ] - else if contains_dune_internal_module_path (read_file (path root relative)) + if contains_dune_internal_module_path (read_file (path root relative)) then fail "test source names an unexposed Dune module: %s" relative) ocaml_files; List.concat_map @@ -274,8 +271,6 @@ let test_public_api_only_test_boundaries root = test_roots |> List.iter (fun relative -> forbid_text root relative [ ".objs/byte"; ".objs/native" ]); - require_file root "logseq_sync/test/core_contract.ml"; - require_file root "logseq_sync/test/runner_contract.ml"; require_file root "logseq_sync/test/sync_protocol_contract.ml"; forbid_path root "logseq_sync/test/test_support.ml" ;; @@ -297,10 +292,8 @@ let test_logseq_sync_suite_boundary root = ; "logseq_db_worker" ]) sync_test_files; - require_text root "logseq_sync/test/test_sync.ml" [ "Alcotest.run" ]; + require_text root "logseq_sync/test/test_wire_protocol.ml" [ "Alcotest.run" ]; require_text root "logseq_sync/test/dune" [ "alcotest" ]; - forbid_text root "logseq_sync/lib/pure_reducer/dune" [ "alcotest" ]; - forbid_text root "logseq_sync/lib/effect_runner/dune" [ "alcotest" ]; require_text root "dune-project" [ "(alcotest (and :with-test (= 1.7.0)))" ]; require_text root "logseq_sync.opam" [ "\"alcotest\" {with-test & = \"1.7.0\"}" ]; require_text root "logseq_sync.opam.locked" [ "\"alcotest\" {= \"1.7.0\" & with-test}" ]; @@ -334,17 +327,8 @@ let test_logseq_sync_package_boundary root = ; "logseq_db_storage/lib/dune" ; "logseq_db_storage/lib/sync_checkpoint_store.ml" ; "logseq_db_storage/lib/sync_checkpoint_store.mli" - ; "logseq_sync/lib/pure_reducer/dune" - ; "logseq_sync/lib/pure_reducer/core.ml" - ; "logseq_sync/lib/pure_reducer/sync_protocol.ml" - ; "logseq_sync/lib/effect_runner/dune" - ; "logseq_sync/lib/effect_runner/effect_runner.ml" - ; "logseq_sync/lib/effect_runner/platform/platform_crypto.ml" - ; "logseq_sync/spec/pure_reducer/dune" - ; "logseq_sync/spec/pure_reducer/core.mli" - ; "logseq_sync/spec/pure_reducer/sync_protocol.mli" - ; "logseq_sync/spec/effect_runner/dune" - ; "logseq_sync/spec/effect_runner/effect_runner.mli" + ; "logseq_sync/lib/protocol/sync_protocol.ml" + ; "logseq_sync/spec/protocol/sync_protocol.mli" ]; List.iter (forbid_path root) @@ -394,22 +378,10 @@ let test_logseq_sync_package_boundary root = ; "(package\n (name logseq_db_worker)" ]; require_occurrences root "dune-project" "(logseq_overlay_db (= 0.1.0))" 2; - require_text - root - "logseq_sync/spec/pure_reducer/dune" - [ "(name logseq_sync_pure_reducer)"; "(public_name logseq_sync.pure_reducer)" ]; - require_text - root - "logseq_sync/spec/effect_runner/dune" - [ "(name logseq_sync_effect_runner)" - ; "(public_name logseq_sync.effect_runner)" - ; "logseq_sync.pure_reducer" - ]; - require_text root "logseq_sync/lib/effect_runner/dune" [ "(private_modules" ]; require_text root "logseq_db_worker/lib/dune" - [ "logseq_overlay_db"; "logseq_sync.pure_reducer" ]; + [ "logseq_overlay_db"; "execute"; "sync_dsl"; "logseq_sync.domain" ]; forbid_text root "logseq_db_worker/lib/dune" [ "logseq_db_storage"; "datascript" ]; require_text root "logseq_db_storage.opam" [ "\"logseq_db_types\" {= \"0.1.0\"}" ]; require_text root "logseq_sync.opam" [ "\"logseq_overlay_db\" {= \"0.1.0\"}" ]; @@ -467,6 +439,9 @@ let test_logseq_sync_package_boundary root = ; "Logseq_db_worker__" ; "logseq_db_worker/lib" ; "logseq_db_worker.sync_spec" + ; "Journal_sync_lifecycle" + ; "journal_sync_lifecycle" + ; "logseq_journal." ]) (files_with_suffixes root "logseq_sync/lib" [ ".ml"; ".mli"; ".c"; "dune" ] @ files_with_suffixes root "logseq_sync/spec" [ ".ml"; ".mli"; "dune" ]); @@ -486,21 +461,30 @@ let test_logseq_sync_package_boundary root = ] ;; +let test_removed_sync_api_boundary root = + List.iter (forbid_path root) + [ "logseq_sync/lib/pure_reducer"; "logseq_sync/spec/pure_reducer" + ; "logseq_sync/lib/effect_runner"; "logseq_sync/spec/effect_runner" + ; "logseq_sync/test/core_contract.ml"; "logseq_sync/test/runner_contract.ml" + ; "logseq_sync/test/test_sync.ml"; "logseq_sync/test/pure_reducer_bad_case_support.ml" + ]; + List.iter + (fun directory -> + files_with_suffixes root directory [ ".ml"; ".mli"; "dune" ] + |> List.filter (fun relative -> relative <> "test/source_boundary_test.ml") + |> List.iter (fun relative -> + forbid_text root relative + [ "Logseq_sync_pure_reducer"; "Logseq_sync_effect_runner" + ; "logseq_sync.pure_reducer"; "logseq_sync.effect_runner" + ; "logseq_sync_pure_reducer_impl"; "logseq_sync_effect_runner_impl" ])) + [ "app"; "logseq_db_worker"; "logseq_sync"; "test" ] +;; + let test_injected_logseq_sync_api_boundary root = List.iter (require_file root) - [ "logseq_sync/spec/pure_reducer/dune" - ; "logseq_sync/spec/pure_reducer/core.mli" - ; "logseq_sync/spec/pure_reducer/sync_protocol.mli" - ; "logseq_sync/spec/effect_runner/dune" - ; "logseq_sync/spec/effect_runner/effect_runner.mli" - ; "logseq_sync/lib/pure_reducer/dune" - ; "logseq_sync/lib/pure_reducer/sync_protocol.ml" - ; "logseq_sync/lib/pure_reducer/core.ml" - ; "logseq_sync/lib/effect_runner/dune" - ; "logseq_sync/lib/effect_runner/effect_runner.ml" - ; "logseq_sync/test/core_contract.ml" - ; "logseq_sync/test/runner_contract.ml" + [ "logseq_sync/spec/protocol/sync_protocol.mli" + ; "logseq_sync/lib/protocol/sync_protocol.ml" ]; List.iter (forbid_path root) @@ -518,64 +502,12 @@ let test_injected_logseq_sync_api_boundary root = ; "logseq_sync/lib/sync_protocol_core.ml" ; "logseq_sync/test/api_contract.ml" ]; - require_text - root - "logseq_sync/spec/pure_reducer/dune" - [ "(name logseq_sync_pure_reducer)" - ; "(public_name logseq_sync.pure_reducer)" - ; "(modules core sync_protocol asset_transfer)" - ; "(virtual_modules core sync_protocol asset_transfer)" - ; "(default_implementation logseq_sync_pure_reducer_impl)" - ]; - require_text - root - "logseq_sync/spec/effect_runner/dune" - [ "(name logseq_sync_effect_runner)" - ; "(public_name logseq_sync.effect_runner)" - ; "(modules effect_runner asset_codec asset_cache)" - ; "(virtual_modules effect_runner asset_codec asset_cache)" - ; "(default_implementation logseq_sync_effect_runner_impl)" - ; "logseq_sync.pure_reducer" - ]; - require_text - root - "logseq_sync/spec/pure_reducer/core.mli" - [ "type t" - ; "type event =" - ; "type runner_effect" - ; "type worker_effect" - ; "type instruction =" - ; "val initial : config -> (t, create_error) result" - ; "val step : t -> event -> transition" - ; "| Run of runner_effect" - ; "| Delegate of worker_effect" - ; "| Publish of output" - ; "Sync_protocol.Client.message" - ; "Sync_protocol.Server.message" - ]; - forbid_text - root - "logseq_sync/spec/pure_reducer/core.mli" - [ "type effect ="; "val effect :" ]; - require_text - root - "logseq_sync/spec/effect_runner/effect_runner.mli" - [ "type t" - ; "val create" - ; "post:(Logseq_sync_pure_reducer.Core.event -> unit)" - ; "val submit : t -> Logseq_sync_pure_reducer.Core.runner_effect -> unit" - ; "val shutdown : t -> unit" - ]; - forbid_text - root - "logseq_sync/lib/pure_reducer/core.ml" - [ "mutable"; " := "; "Effect.perform"; "Eio"; "Unix"; "Sys."; "Logseq_db_worker" ]; require_text root "logseq_db_worker/lui/logseq_db_worker_lui_service.ml" [ "module Pure = Logseq_db_worker_pure_reducer.Core" ; "module Worker_runner = Logseq_db_worker_effect_runner.Effect_runner" - ; "module Sync_runner = Logseq_sync_effect_runner.Effect_runner" + ; "module Sync = Logseq_sync_domain.Core" ; "Db.create" ; "Db.post" ; "Db.read_for_scope" @@ -599,7 +531,7 @@ let test_injected_logseq_sync_api_boundary root = let test_standalone_sync_protocol_boundary root = require_text root - "logseq_sync/spec/pure_reducer/sync_protocol.mli" + "logseq_sync/spec/protocol/sync_protocol.mli" [ "type cursor = Logseq_db_types.Server_cursor.t" ; "type checksum = string" ; "module Client : sig" @@ -613,25 +545,27 @@ let test_standalone_sync_protocol_boundary root = ]; forbid_text root - "logseq_sync/spec/pure_reducer/sync_protocol.mli" + "logseq_sync/spec/protocol/sync_protocol.mli" [ "Logseq_sync_pure_core"; "Sync_protocol_core" ]; require_text root - "logseq_sync/lib/pure_reducer/dune" - [ "(name logseq_sync_pure_reducer_impl)" - ; "(public_name logseq_sync.pure_reducer.impl)" - ; "(implements logseq_sync_pure_reducer)" - ; "(modules core sync_protocol asset_transfer)" + "logseq_sync/spec/protocol/dune" + [ "(name logseq_sync_protocol)" + ; "(public_name logseq_sync.protocol)" + ; "(virtual_modules sync_protocol)" + ; "(default_implementation logseq_sync_protocol_impl)" ]; - List.iter - (forbid_path root) - [ "logseq_sync/lib/pure_reducer/checksum.ml" - ; "logseq_sync/lib/pure_reducer/checksum.mli" - ; "logseq_sync/lib/pure_reducer/pure_tx.ml" + require_text + root + "logseq_sync/lib/protocol/dune" + [ "(name logseq_sync_protocol_impl)" + ; "(public_name logseq_sync.protocol.impl)" + ; "(implements logseq_sync_protocol)" + ; "(modules sync_protocol)" ]; require_text root - "logseq_sync/lib/pure_reducer/sync_protocol.ml" + "logseq_sync/lib/protocol/sync_protocol.ml" [ "type cursor = Logseq_db_types.Server_cursor.t" ; "type checksum = string" ; "module Client = struct" @@ -641,48 +575,12 @@ let test_standalone_sync_protocol_boundary root = ]; forbid_text root - "logseq_sync/lib/pure_reducer/sync_protocol.ml" + "logseq_sync/lib/protocol/sync_protocol.ml" [ "Sync_protocol_core"; "include Logseq_sync_pure_core" ]; - require_text - root - "logseq_sync/lib/pure_reducer/core.ml" - [ "Sync_protocol.Client.message" - ; "Sync_protocol.Server.message" - ; "Sync_protocol.codec_error" - ]; - forbid_text - root - "logseq_sync/lib/pure_reducer/core.ml" - [ "type authoritative_message =" - ; "let parse_authoritative_message" - ; "let pull_payload" - ; "let submission_payload" - ; "pending_payload" - ; "Websocket_frame" - ; "non-authoritative WebSocket message" - ; "Sync_protocol_core" - ; "Core_protocol" - ; "Protocol_adapter" - ; "export_client_message" - ; "import_server_message" - ; "import_rejection_reason" - ]; - require_text - root - "logseq_sync/lib/effect_runner/effect_runner.ml" - [ "Sync_protocol.decode_server_message" - ; "Sync_protocol.encode_client_message" - ; "Core.Websocket_message" - ; "Core.Websocket_protocol_error" - ]; - forbid_text - root - "logseq_sync/lib/effect_runner/effect_runner.ml" - [ "Core.Websocket_frame" ]; require_text root "logseq_sync/test/sync_protocol_contract.ml" - [ "module Protocol = Logseq_sync_pure_reducer.Sync_protocol" + [ "module Protocol = Logseq_sync_protocol.Sync_protocol" ; "all client messages round trip" ; "all server messages round trip" ; "owned client fields are strict and safe" @@ -692,21 +590,14 @@ let test_standalone_sync_protocol_boundary root = ;; let test_worker_dune_closure_names root = - require_text root "logseq_sync/lib/effect_runner/dune" [ "logseq_sync.pure_reducer" ]; - require_text root "logseq_db_worker/lib/dune" [ "logseq_sync.pure_reducer" ]; - let pure_dune = "logseq_sync/lib/pure_reducer/dune" in - forbid_text - root - pure_dune - [ "eio"; "unix"; "x509"; "tls"; "httpun"; "logseq_db_storage"; "platform_crypto" ]; + require_text root "logseq_db_worker/lib/dune" + [ "logseq_overlay_db"; "execute"; "sync_dsl"; "logseq_sync.domain" ]; List.iter - (forbid_path root) - [ "logseq_sync/lib/pure_core.ml" - ; "logseq_sync/lib/sync_protocol_core.ml" - ; "logseq_sync/lib/core.ml" - ; "logseq_sync/lib/sync_protocol.ml" - ; "logseq_sync/lib/effect_runner.ml" - ] + (fun directory -> + require_file root (directory ^ "/dune"); + forbid_text root (directory ^ "/dune") + [ "eio"; "unix"; "x509"; "tls"; "httpun"; "logseq_db_storage"; "platform_crypto" ]) + [ "logseq_sync/lib/domain"; "logseq_sync/lib/asset_transfer" ] ;; let forbid_worker_storage_access root relative = @@ -721,6 +612,10 @@ let forbid_worker_storage_access root relative = ;; let test_worker_owned_overlay_orchestration root = + List.iter + (fun relative -> forbid_text root relative [ "pure_reducer"; "Logseq_db_worker"; "logseq_db_worker" ]) + (files_with_suffixes root "logseq_sync/lib/worker_owner" [ ".ml"; ".mli"; "dune" ] + @ files_with_suffixes root "logseq_sync/test/asset_owner_lane" [ ".ml"; ".mli"; "dune" ]); let forbidden_raw_data_plane = [ "Datascript.db" ; "Datascript.tx_op" @@ -733,9 +628,33 @@ let test_worker_owned_overlay_orchestration root = ] in List.iter - (fun relative -> forbid_text root relative forbidden_raw_data_plane) + (fun relative -> + if relative = "logseq_sync/lib/worker_owner/core.ml" then ( + forbid_text root relative (List.filter (fun text -> text <> "Logseq_db_storage") forbidden_raw_data_plane); + let contents = read_file (path root relative) in + if count_occurrences contents "Logseq_db_storage" <> 1 + || count_occurrences contents "module Store = Logseq_db_storage.Asset_upload_store" <> 1 + then fail "asset owner accesses storage outside its upload intent store: %s" relative) + else forbid_text root relative forbidden_raw_data_plane) (files_with_suffixes root "logseq_sync/spec" [ ".mli" ] @ files_with_suffixes root "logseq_sync/lib" [ ".ml"; ".mli" ]); + List.iter + (fun relative -> + forbid_text root relative + [ "Datascript." + ; "datascript" + ; "logseq_db_storage" + ; "Logseq_db_storage" + ; "Logseq_db_worker" + ; "logseq_db_worker" + ; "Logseq_sync_pure_reducer" + ; "logseq_sync.pure_reducer" + ; "Logseq_sync_effect_runner" + ; "logseq_sync.effect_runner" + ]) + (List.concat_map + (fun directory -> files_with_suffixes root directory [ ".ml"; ".mli"; "dune" ]) + [ "logseq_sync/lib/native_transport"; "logseq_sync/lib/transport" ]); let sync_files = files_with_suffixes root "logseq_sync" [ ".ml"; ".mli" ] in List.iter (fun relative -> @@ -744,10 +663,6 @@ let test_worker_owned_overlay_orchestration root = relative [ "Engine.t"; "Engine.open_"; "Engine.close"; "Logseq_db_worker" ]) sync_files; - require_text - root - "logseq_sync/spec/pure_reducer/core.mli" - [ "Logseq_overlay_db.Types"; "type worker_effect =" ]; require_text root "logseq_db_worker/lib/effect_runner/effect_runner.ml" @@ -762,7 +677,7 @@ let test_worker_owned_overlay_orchestration root = "logseq_db_worker/lui/logseq_db_worker_lui_service.ml" [ "Logseq_db_worker_pure_reducer.Core" ; "Logseq_db_worker_effect_runner.Effect_runner" - ; "Sync_runner.create" + ; "Db.create_lifecycle" ]; forbid_text root @@ -777,38 +692,30 @@ let test_worker_owned_overlay_orchestration root = let test_logseq_sync_install_manifest filename = let contents = read_file filename in - if - not - (contains contents {|/lib/logseq_sync/pure_reducer/logseq_sync_pure_reducer.cmi"|}) - then fail "installed logseq_sync package is missing its pure-reducer interface"; List.iter - (fun module_name -> - if - not - (contains - contents - (Printf.sprintf - {|/lib/logseq_sync/pure_reducer/logseq_sync_pure_reducer__%s.cmi"|} - module_name)) - then fail "installed pure-reducer library is missing public %s" module_name) - [ "Core"; "Sync_protocol" ]; - if - not - (contains - contents - {|/lib/logseq_sync/effect_runner/logseq_sync_effect_runner.cmi"|}) - then fail "installed logseq_sync package is missing its effect-runner interface"; - if - not - (contains - contents - {|/lib/logseq_sync/effect_runner/logseq_sync_effect_runner__Effect_runner.cmi"|}) - then fail "installed effect-runner library is missing public Effect_runner"; + (fun (library, module_name) -> + let interface = Printf.sprintf "/lib/logseq_sync/%s/logseq_sync_%s.cmi\"" library library in + let public_module = Printf.sprintf "/lib/logseq_sync/%s/logseq_sync_%s__%s.cmi\"" library library module_name in + List.iter + (fun expected -> + if not (contains contents expected) + then fail "installed logseq_sync package is missing %s" expected) + [ interface; public_module ]) + [ "protocol", "Sync_protocol" + ; "domain", "Core" + ; "asset_transfer", "Core" + ; "assets", "Core" + ; "crypto", "Core" + ; "native_crypto", "Asset_codec" + ; "native_transport", "Http" + ]; List.iter (fun obsolete -> if contains contents obsolete - then fail "obsolete combined Logseq_sync interface remains installed: %s" obsolete) - [ {|/lib/logseq_sync/logseq_sync.cmi"|} + then fail "obsolete Sync interface remains installed: %s" obsolete) + [ "/lib/logseq_sync/pure_reducer/" + ; "/lib/logseq_sync/effect_runner/" + ; {|/lib/logseq_sync/logseq_sync.cmi"|} ; "/logseq_sync__Core.cmi" ; "/logseq_sync__Effect_runner.cmi" ; "/logseq_sync__Sync_protocol.cmi" @@ -877,9 +784,6 @@ let test_final_overlay_data_plane_boundary root = ; "logseq_db_worker/test/fixtures/performance/reference-apple-silicon-v1.json" ; "logseq_db_worker/tool/performance_benchmark.ml" ; "logseq_db_worker/tool/test_performance.sh" - ; "logseq_sync/lib/pure_reducer/pure_tx.ml" - ; "logseq_sync/lib/pure_reducer/checksum.ml" - ; "logseq_sync/lib/pure_reducer/checksum.mli" ; "logseq_db_types/test/test_mutation_identity.ml" ; "logseq_db_types/test/dune" ]; @@ -909,7 +813,22 @@ let test_final_overlay_data_plane_boundary root = require_documentation_only_guide root guide; List.iter (fun relative -> forbid_text root relative [ "Logseq_overlay_db"; "Logseq_sync" ]) - (List.filter (fun relative -> relative <> guide) (ocaml_product_files root)); + (List.filter + (fun relative -> + relative <> guide + && relative <> "app/impl/sync_lifecycle/core.ml" + && relative <> "app/spec/sync_lifecycle/core.mli" + && relative <> "app/impl/sync_lifecycle/account.ml" + && relative <> "app/spec/sync_lifecycle/account.mli") + (ocaml_product_files root)); + forbid_path root "logseq_sync/lib/service_coordinator"; + require_text root "app/spec/sync_lifecycle/dune" + [ "(public_name logseq_journal.sync_lifecycle)" + ; "(virtual_modules core account)" + ; "(default_implementation journal_sync_lifecycle_impl)" + ]; + require_text root "app/impl/sync_lifecycle/dune" + [ "(implements journal_sync_lifecycle)" ]; forbid_text root "app/dune" [ "logseq_sync"; "logseq_overlay_db" ]; List.iter (fun relative -> @@ -973,48 +892,13 @@ let test_sync_transport_is_websocket_only root = in List.iter (fun relative -> forbid_text root relative obsolete_symbols) - [ "logseq_db_worker/lui/logseq_db_worker_lui_service.ml"; "app/journal_platform.ml" ]; - forbid_text - root - "logseq_sync/lib/effect_runner/eio/http.ml" - [ "\"since\", string_of_int"; "graph_path graph_id \"tx/batch\"" ]; - forbid_text - root - "logseq_sync/lib/effect_runner/eio/http.mli" - [ "val pull"; "val transaction_batch" ] -;; - -let has_exact_dependency contents ~package ~version = - contains contents (Printf.sprintf "\"%s\" {= \"%s\"}" package version) + [ "logseq_db_worker/lui/logseq_db_worker_lui_service.ml"; "app/journal_platform.ml" ] ;; let test_startup_phase_ownership root = - require_text - root - "logseq_sync/spec/pure_reducer/core.mli" - [ "type sync_phase =" - ; "| Offline" - ; "| Connecting" - ; "| Pulling" - ; "| Submitting" - ; "| Current" - ; "| Paused" - ; "sync_phase : sync_phase" - ]; - forbid_text - root - "logseq_sync/spec/pure_reducer/core.mli" - [ "type phase =" - ; "| Signed_out" - ; "| Loading_catalog" - ; "| Awaiting_selection" - ; "| Restoring_local" - ; "| Bootstrapping" - ; "| Awaiting_e2ee_password" - ; "| Opening_graph" - ; "| Graph_open" - ; "| Sync_paused" - ]; + require_text root "logseq_db_types/lib/sync_account.mli" + [ "type sync_phase ="; "| Offline"; "| Connecting"; "| Pulling" + ; "| Submitting"; "| Current"; "| Paused"; "sync_phase : sync_phase" ]; require_text root "logseq_db_worker/lib/logseq_db_worker.mli" @@ -1049,6 +933,10 @@ let test_startup_phase_ownership root = [ "Logseq_sync.Core.phase"; "snapshot.Logseq_sync.Core.phase" ] ;; +let has_exact_dependency contents ~package ~version = + contains contents (Printf.sprintf "\"%s\" {= \"%s\"}" package version) +;; + let require_exact_dependency root relative ~package ~version = let candidate = path root relative in if not (Sys.file_exists candidate && not (Sys.is_directory candidate)) @@ -1159,6 +1047,7 @@ let () = test_public_api_only_test_boundaries root; test_logseq_sync_suite_boundary root; test_logseq_sync_package_boundary root; + test_removed_sync_api_boundary root; test_injected_logseq_sync_api_boundary root; test_standalone_sync_protocol_boundary root; test_worker_dune_closure_names root; @@ -1290,12 +1179,6 @@ let () = ; "app/journal_time.mli" ; "app/journal_timeline_state.ml" ; "app/journal_timeline_state.mli" - ; "logseq_sync/lib/effect_runner/storage/bootstrap.ml" - ; "logseq_sync/lib/effect_runner/storage/bootstrap.mli" - ; "logseq_sync/lib/effect_runner/protocol/catalog.ml" - ; "logseq_sync/lib/effect_runner/protocol/catalog.mli" - ; "logseq_sync/lib/effect_runner/eio/http.ml" - ; "logseq_sync/lib/effect_runner/eio/http.mli" ; "swift/App.swift" ; "swift/JournalApplicationPlatform.swift" ; "swift/JournalNativeServices.swift" @@ -1419,10 +1302,21 @@ let () = "app/application.ml" [ "Ui.Theme.create"; "~mode:System"; "Journal_composer_view.content" ]; require_text root "app/journal_composer_view.ml" [ "V.composer" ]; - require_text root "app/journal_detail_view.ml" [ "Detail_list.view"; "journal-detail-route" ]; - require_text root "app/journal_manager_view.ml" - [ "Journal_header.startup"; "V.secure_field"; "graph_picker_list"; "graph-picker-list" - ; "e2ee-password-cancel"; "Refresh the authorized graph catalog"; "journal-startup-diagnostics" ]; + require_text + root + "app/journal_detail_view.ml" + [ "Detail_list.view"; "journal-detail-route" ]; + require_text + root + "app/journal_manager_view.ml" + [ "Journal_header.startup" + ; "V.secure_field" + ; "graph_picker_list" + ; "graph-picker-list" + ; "e2ee-password-cancel" + ; "Refresh the authorized graph catalog" + ; "journal-startup-diagnostics" + ]; forbid_text root "app/journal_header.ml" [ "V.divider" ]; forbid_text root @@ -1559,10 +1453,7 @@ let () = root "app/journal_routes.ml" [ "Capture_view"; "open_capture"; "update_capture" ]; - require_text - root - "app/application.ml" - [ "V.Sheet.create"; "Set status" ]; + require_text root "app/application.ml" [ "V.Sheet.create"; "Set status" ]; require_text root "app/journal_status_view.ml" @@ -1660,24 +1551,15 @@ let () = ; "ProgressView()" ]; forbid_text root "swift/App.swift" [ "Darwin.exit" ]; - require_text - root - "logseq_sync/lib/effect_runner/eio/http_eio.ml" - [ "Httpun_eio.Client.create_connection"; "Httpun_eio.Client.request" ]; - forbid_text - root - "logseq_sync/lib/effect_runner/eio/http_eio.ml" - [ "let start_connection"; "HTTP parser did not consume network input" ]; - require_text root "logseq_sync/lib/effect_runner/dune" [ "httpun-eio" ]; List.iter (require_file root) - [ "logseq_sync/lib/effect_runner/eio/tls_client_eio.ml" - ; "logseq_sync/lib/effect_runner/eio/tls_client_eio.mli" + [ "logseq_sync/lib/native_transport/tls_client_eio.ml" + ; "logseq_sync/lib/native_transport/tls_client_eio.mli" ]; - require_occurrences root "logseq_sync/lib/effect_runner/dune" "tls_client_eio" 2; + require_occurrences root "logseq_sync/lib/native_transport/dune" "tls_client_eio" 1; require_text root - "logseq_sync/lib/effect_runner/eio/tls_client_eio.ml" + "logseq_sync/lib/native_transport/tls_client_eio.ml" [ "Mirage_crypto_rng_unix.use_default" ; "Domain_name.of_string" ; "Domain_name.host" @@ -1704,8 +1586,8 @@ let () = ; "Eio.Net.connect" ; "Tls_eio.client_of_flow" ]) - [ "logseq_sync/lib/effect_runner/eio/http_eio.ml" - ; "logseq_sync/lib/effect_runner/eio/websocket_eio.ml" + [ "logseq_sync/lib/native_transport/http_eio.ml" + ; "logseq_sync/lib/native_transport/websocket_eio.ml" ]; require_text root @@ -1788,10 +1670,6 @@ let () = root "logseq_db_worker/lui/logseq_db_worker_lui_service.ml" [ "interpret_local_action"; "interpret_network_action"; "Local_completion" ]; - require_text - root - "logseq_sync/lib/pure_reducer/core.ml" - [ "type diagnostics"; "type state ="; "let state core = core.public_state" ]; require_text root "logseq_db_worker/lui/logseq_db_worker_lui_service.mli" @@ -1914,9 +1792,7 @@ let () = List.iter (fun relative -> forbid_text root relative [ "history : string list"; "append_diagnostic_history" ]) - [ "logseq_sync/spec/pure_reducer/core.mli" - ; "logseq_sync/lib/pure_reducer/core.ml" - ; "logseq_db_worker/lui/logseq_db_worker_lui_service.mli" + [ "logseq_db_worker/lui/logseq_db_worker_lui_service.mli" ; "logseq_db_worker/lui/logseq_db_worker_lui_service.ml" ]; forbid_text root "app/application.ml" [ "Recent sync transitions" ]; @@ -2025,26 +1901,6 @@ let () = root "app/application.ml" [ "Journal_calendar.Sampler.sample"; "Journal_calendar.present_journal_day" ]; - require_text - root - "logseq_sync/spec/pure_reducer/core.mli" - [ "type runner_effect = private"; "Asset_requested"; "Asset_finished" ]; - forbid_text - root - "logseq_sync/spec/effect_runner/effect_runner.mli" - [ "val submit_asset" - ; "val run_scoped_asset" - ; "val upload_asset" - ; "val stage_asset" - ; "val retain_asset_file" - ; "val release_asset_file" - ; "val close_asset_scope" - ; "val delete_graph_assets" - ; "val staged_asset_path" - ; "val release_staged_asset" - ; "val prune_staged_assets" - ; "val retain_staged_file" - ]; match List.rev !failures with | [] -> print_endline "source boundary is clean" | failures -> diff --git a/test/startup_test.ml b/test/startup_test.ml index 357d02ad..df126c2e 100644 --- a/test/startup_test.ml +++ b/test/startup_test.ml @@ -94,10 +94,11 @@ let startup_snapshot Graph_service. { sync_phase ; catalog = [] - ; resident_graphs=[];resident_scopes=[] + ; resident_graphs = [] + ; resident_scopes = [] ; selected_graph = Some graph_id ; applied_server_t = Some (server_cursor_of_int 11) - ; unlock = Logseq_sync_pure_reducer.Core.Unlock_inactive + ; unlock = Logseq_sync_domain.Core.Unlock_inactive ; timeline_presentation_pending ; startup = { authenticated @@ -220,185 +221,493 @@ let test_stale_graph_generation_and_structured_failures () = | None | Some _ -> fail "graph failure did not expose structured recovery" ;; +let test_bootstrap_crypto_failure_is_terminal () = + let cause = "Capacity exceeded: Crypto items" in + let snapshot = startup_snapshot ~sync_phase:Failed ~bootstrapping:true () in + let snapshot = { snapshot with last_error = Some cause } in + let state = + require_startup_phase + Journal_startup.Failed + snapshot + (graph_state ~phase:Graph_closed ()) + "crypto failure left startup preparing" + in + require + (Option.map (fun error -> error.Journal_startup.message) state.error = Some cause) + "bootstrap failure lost its diagnostic"; + let structured = + { snapshot with + startup = { snapshot.startup with failure = Some Graph_service.During_e2ee } + } + in + let state = + require_startup_phase + Journal_startup.Failed + structured + (graph_state ~phase:Graph_closed ()) + "structured failure lost priority" + in + require + (Option.map (fun error -> error.Journal_startup.owner) state.error + = Some Journal_startup.E2ee) + "bootstrap fallback replaced structured failure owner"; + require + (Option.map (fun error -> error.Journal_startup.message) state.error = Some cause) + "structured failure lost original diagnostic"; + let state = + require_startup_phase + Journal_startup.Failed + snapshot + (graph_state ~phase:Graph_failed ~error:(worker_error "engine unavailable") ()) + "graph failure lost priority" + in + require + (Option.map (fun error -> error.Journal_startup.message) state.error + = Some "engine unavailable") + "bootstrap fallback replaced graph diagnostic"; + ignore + (require_startup_phase + Journal_startup.Ready + snapshot + (graph_state ()) + "background sync failure displaced ready graph") +;; + let test_noncurrent_deletion_keeps_active_graph_ready () = - List.iter (fun local_deletion -> - let snapshot = { (startup_snapshot ()) with local_deletion = Some local_deletion } in - let derived = Journal_startup.derive ~snapshot ~graph:(graph_state ()) in - require (derived.phase = Journal_startup.Ready) - "non-current deletion displaced the active graph") - [ Graph_service.Deletion_in_progress Deleting_mirror; Deletion_failed Deleting_mirror ] + List.iter + (fun local_deletion -> + let snapshot = + { (startup_snapshot ()) with local_deletion = Some local_deletion } + in + let derived = Journal_startup.derive ~snapshot ~graph:(graph_state ()) in + require + (derived.phase = Journal_startup.Ready) + "non-current deletion displaced the active graph") + [ Graph_service.Deletion_in_progress Deleting_mirror + ; Deletion_failed Deleting_mirror + ] ;; let test_stable_startup_presentation () = let open Journal_startup in let module P = Journal_startup.Presentation in - let observe ?manager ?(graph=graph_state ()) ?(graph_ready=false) - ?(feed_loaded=false) ?(password_pending=false) ?(graph_error=false) t = - P.step t (Observe {manager;graph;graph_ready;feed_loaded;password_pending;graph_error}) in + let observe + ?manager + ?(graph = graph_state ()) + ?(graph_ready = false) + ?(feed_loaded = false) + ?(password_pending = false) + ?(graph_error = false) + t + = + P.step + t + (Observe { manager; graph; graph_ready; feed_loaded; password_pending; graph_error }) + in let timer t = Option.get (P.loading_request t) in - let account t result = P.step t (Account (P.account_ticket t,result)) in - let cases=[ - "anonymous local restore is not signed out",(fun () -> - let phase=Journal_startup.derive ~snapshot:(startup_snapshot ~authenticated:false ~restoring_local:true ()) - ~graph:(graph_state ~phase:Graph_closed ()) in - require (phase.phase=Restoring_local) "local restore flashes signed-out UI"; - let phase=Journal_startup.derive ~snapshot:(startup_snapshot ~authenticated:false ~failure:During_local_restore ()) - ~graph:(graph_state ~phase:Graph_closed ()) in - require (phase.phase=Failed) "anonymous restore failure hides recovery"); - "initial and unresolved account stay blank",(fun () -> - let t=P.create () in - require (P.screen t=Waiting && not(P.loading_indicator t) && P.loading_request t<>None) - "initial startup mounts transient Timeline or immediate spinner"; - let t=observe ~manager:(startup_snapshot ~authenticated:false ()) ~graph:(graph_state ~phase:Graph_closed ()) t in - require (P.screen t=Waiting) "account discovery flashes sign-in"; - let t=account t Local in - let t=observe ~manager:(startup_snapshot ~authenticated:false ~restoring_local:true ()) ~graph:(graph_state ~phase:Graph_closed ()) t in - require (P.screen t=Waiting) "local binding restore flashes intermediate page"); - "one 100ms grace spans catalog restore and first feed",(fun () -> - let t=observe ~manager:(startup_snapshot ~catalog_loading:true ()) ~graph:(graph_state ~phase:Graph_closed ()) (P.create ()) in - let key=timer t in - let early=P.step t (Elapsed(key,0.099)) in - require (not(P.loading_indicator early)) "99ms shows loading"; - require (P.step early (Elapsed(key,0.099999999))==early) - "one nanosecond before 100ms shows loading"; - let t=observe ~manager:(startup_snapshot ~restoring_local:true ()) early in - require (timer t=key) "busy phase restarts grace"; - let slow=P.step t (Elapsed(key,0.1)) in - require (P.screen slow=Waiting && P.loading_indicator slow) "100ms waiting has no loading"; - require (P.step slow (Elapsed(key,0.2))==slow) "duplicate elapsed changes owner"; - let ready=observe ~manager:(startup_snapshot ()) ~graph_ready:true ~feed_loaded:true slow in - require (P.screen ready=Timeline && not(P.loading_indicator ready)) "ready body retains loading"; - require (P.step ready (Elapsed(key,2.))==ready) "late timer changes ready Timeline"); - "Graph_open waits for first feed and an empty feed completes",(fun () -> - let t=observe ~manager:(startup_snapshot ()) ~graph_ready:true (P.create ()) in - require (P.screen t=Waiting) "Graph_open mounts feed's transient loading page"; - let ready=observe ~manager:(startup_snapshot ~timeline_presentation_pending:true ()) - ~graph_ready:true ~feed_loaded:true t in - require (P.screen ready=Timeline) "presentation handshake or empty feed deadlocks Timeline"; - let refreshing=observe ~manager:(startup_snapshot ~sync_phase:Pulling ~catalog_loading:true ()) - ~graph_ready:true ~feed_loaded:true ready in - require (P.screen refreshing=Timeline) "cached background refresh hides existing content"); - "required input and errors remain immediately available",(fun () -> - let t=account (P.create ()) No_user in - require (match P.screen t with Interaction {phase=Signed_out;_}->true|_->false) "signed-out result is hidden"; - List.iter(fun phase -> - let snapshot=match phase with - | Awaiting_selection -> startup_snapshot ~awaiting_selection:true () - | Awaiting_e2ee_password -> startup_snapshot ~awaiting_e2ee_password:true () - | _ -> startup_snapshot ~failure:During_bootstrap () in - let t=observe ~manager:snapshot ~graph:(graph_state ~phase:Graph_closed ()) (P.create ()) in - require (match P.screen t with Interaction s->s.phase=phase|_->false) "required input/failure is masked"; - require (not(P.loading_indicator t)) "required input is covered by startup loading") - [Awaiting_selection;Awaiting_e2ee_password;Failed]); - "unlock submission retains its form and does not remount busy page",(fun () -> - let t=observe ~manager:(startup_snapshot ~awaiting_e2ee_password:true ()) - ~graph:(graph_state ~phase:Graph_closed ()) (P.create ()) in - let pending=observe ~manager:(startup_snapshot ~bootstrapping:true ()) - ~graph:(graph_state ~phase:Graph_closed ()) ~password_pending:true t in - require (match P.screen pending with Interaction {phase=Awaiting_e2ee_password;_}->true|_->false) - "password submission flashes a different waiting page"); - "account lookup failure Retry and stale replies",(fun () -> - let initial=P.create () in let old=P.account_ticket initial in - let failed=P.step initial (Account_error old) in - require (P.screen failed=Account_failure) "lookup failure has no explicit recovery"; - let retry=P.step failed Retry in - require (P.screen retry=Waiting && not(P.loading_indicator retry) && P.account_ticket retry<>old) - "Retry does not start fresh discovery/grace"; - require (P.step retry (Account(old,No_user))==retry) "old account result replaces retry"; - require (P.step retry (Account_error old)==retry) "old lookup failure replaces retry"); - "local offline recovery is independent of online auth and assets",(fun () -> - let t=account (P.create ()) Local in - let t=P.step t (Account_error(P.account_ticket t)) in - let ready=observe ~manager:(startup_snapshot ~sync_phase:Offline ()) - ~graph_ready:true ~feed_loaded:true t in - require (P.screen ready=Timeline) "valid cached Timeline waits for online auth/assets"); - "graph replacement rejects old ready data and old timer",(fun () -> - let ready=observe ~manager:(startup_snapshot ()) ~graph_ready:true ~feed_loaded:true (P.create ()) in - let other={ (startup_snapshot ~graph_generation:8 ()) with selected_graph=Some graph_id } in - let waiting=observe ~manager:other ~graph:(graph_state ~generation:7 ()) - ~graph_ready:true ~feed_loaded:true ready in - require (P.screen waiting=Waiting) "stale graph falsely publishes ready data"; - let key=timer waiting in - let picker=observe ~manager:(startup_snapshot ~awaiting_selection:true ~graph_generation:8 ()) - ~graph:(graph_state ~phase:Graph_closed ~generation:8 ()) waiting in - let fresh=observe ~manager:(startup_snapshot ~graph_generation:9 ~restoring_local:true ()) - ~graph:(graph_state ~phase:Graph_opening ~generation:9 ()) picker in - require (timer fresh<>key && P.step fresh (Elapsed(key,1.))==fresh) - "old graph journey timer enables new loading"); - "background interruption fences elapsed and resumes a fresh grace",(fun () -> - let t=observe ~manager:(startup_snapshot ~restoring_local:true ()) (P.create ()) in - let old=timer t in let inactive=P.step t (Foreground false) in - require (P.loading_request inactive=None && P.step inactive (Elapsed(old,1.))==inactive) - "background keeps an active loading timer"; - let resumed=P.step inactive (Foreground true) in - require (timer resumed<>old && not(P.loading_indicator resumed)) "resume reuses old grace"; - require (P.step resumed (Elapsed(old,3600.))==resumed) "late background timer changes resume"; - let key=timer resumed in - require (P.step resumed (Elapsed(key,0.099999999))==resumed) - "resume counts suspended time towards its fresh grace"; - require (P.loading_indicator (P.step resumed (Elapsed(key,0.1)))) - "resume does not show loading after its fresh 100ms grace"); - "negative/NaN elapsed and explicit signout are safe",(fun () -> - let t=observe ~manager:(startup_snapshot ~restoring_local:true ()) (P.create ()) in - let key=timer t in - require (P.step t (Elapsed(key,-1.))==t && P.step t (Elapsed(key,nan))==t) "invalid elapsed changes owner"; - let signed=P.step t P.Signed_out in - require (match P.screen signed with Interaction {phase=Signed_out;_}->true|_->false) "signout entry is hidden"; - require (P.step signed (Elapsed(key,2.))==signed) "signout leaves stale timer active") - ] in - let failures=ref [] in - List.iter(fun(name,test)->try test () with error-> - Printf.eprintf "STARTUP_RED %s: %s\n%!" name (Printexc.to_string error);failures:=name::!failures) cases; - require (!failures=[]) "Startup regressions: %s" (String.concat ", " (List.rev !failures)) + let account t result = P.step t (Account (P.account_ticket t, result)) in + let cases = + [ ( "anonymous local restore is not signed out" + , fun () -> + let phase = + Journal_startup.derive + ~snapshot:(startup_snapshot ~authenticated:false ~restoring_local:true ()) + ~graph:(graph_state ~phase:Graph_closed ()) + in + require (phase.phase = Restoring_local) "local restore flashes signed-out UI"; + let phase = + Journal_startup.derive + ~snapshot: + (startup_snapshot ~authenticated:false ~failure:During_local_restore ()) + ~graph:(graph_state ~phase:Graph_closed ()) + in + require (phase.phase = Failed) "anonymous restore failure hides recovery" ) + ; ( "initial and unresolved account stay blank" + , fun () -> + let t = P.create () in + require + (P.screen t = Waiting + && (not (P.loading_indicator t)) + && P.loading_request t <> None) + "initial startup mounts transient Timeline or immediate spinner"; + let t = + observe + ~manager:(startup_snapshot ~authenticated:false ()) + ~graph:(graph_state ~phase:Graph_closed ()) + t + in + require (P.screen t = Waiting) "account discovery flashes sign-in"; + let t = account t Local in + let t = + observe + ~manager:(startup_snapshot ~authenticated:false ~restoring_local:true ()) + ~graph:(graph_state ~phase:Graph_closed ()) + t + in + require (P.screen t = Waiting) "local binding restore flashes intermediate page" + ) + ; ( "one 100ms grace spans catalog restore and first feed" + , fun () -> + let t = + observe + ~manager:(startup_snapshot ~catalog_loading:true ()) + ~graph:(graph_state ~phase:Graph_closed ()) + (P.create ()) + in + let key = timer t in + let early = P.step t (Elapsed (key, 0.099)) in + require (not (P.loading_indicator early)) "99ms shows loading"; + require + (P.step early (Elapsed (key, 0.099999999)) == early) + "one nanosecond before 100ms shows loading"; + let t = observe ~manager:(startup_snapshot ~restoring_local:true ()) early in + require (timer t = key) "busy phase restarts grace"; + let slow = P.step t (Elapsed (key, 0.1)) in + require + (P.screen slow = Waiting && P.loading_indicator slow) + "100ms waiting has no loading"; + require + (P.step slow (Elapsed (key, 0.2)) == slow) + "duplicate elapsed changes owner"; + let ready = + observe + ~manager:(startup_snapshot ()) + ~graph_ready:true + ~feed_loaded:true + slow + in + require + (P.screen ready = Timeline && not (P.loading_indicator ready)) + "ready body retains loading"; + require + (P.step ready (Elapsed (key, 2.)) == ready) + "late timer changes ready Timeline" ) + ; ( "Graph_open waits for first feed and an empty feed completes" + , fun () -> + let t = + observe ~manager:(startup_snapshot ()) ~graph_ready:true (P.create ()) + in + require (P.screen t = Waiting) "Graph_open mounts feed's transient loading page"; + let ready = + observe + ~manager:(startup_snapshot ~timeline_presentation_pending:true ()) + ~graph_ready:true + ~feed_loaded:true + t + in + require + (P.screen ready = Timeline) + "presentation handshake or empty feed deadlocks Timeline"; + let refreshing = + observe + ~manager:(startup_snapshot ~sync_phase:Pulling ~catalog_loading:true ()) + ~graph_ready:true + ~feed_loaded:true + ready + in + require + (P.screen refreshing = Timeline) + "cached background refresh hides existing content" ) + ; ( "required input and errors remain immediately available" + , fun () -> + let t = account (P.create ()) No_user in + require + (match P.screen t with + | Interaction { phase = Signed_out; _ } -> true + | _ -> false) + "signed-out result is hidden"; + List.iter + (fun phase -> + let snapshot = + match phase with + | Awaiting_selection -> startup_snapshot ~awaiting_selection:true () + | Awaiting_e2ee_password -> + startup_snapshot ~awaiting_e2ee_password:true () + | _ -> startup_snapshot ~failure:During_bootstrap () + in + let t = + observe + ~manager:snapshot + ~graph:(graph_state ~phase:Graph_closed ()) + (P.create ()) + in + require + (match P.screen t with + | Interaction s -> s.phase = phase + | _ -> false) + "required input/failure is masked"; + require + (not (P.loading_indicator t)) + "required input is covered by startup loading") + [ Awaiting_selection; Awaiting_e2ee_password; Failed ] ) + ; ( "unlock submission retains its form and does not remount busy page" + , fun () -> + let t = + observe + ~manager:(startup_snapshot ~awaiting_e2ee_password:true ()) + ~graph:(graph_state ~phase:Graph_closed ()) + (P.create ()) + in + let pending = + observe + ~manager:(startup_snapshot ~bootstrapping:true ()) + ~graph:(graph_state ~phase:Graph_closed ()) + ~password_pending:true + t + in + require + (match P.screen pending with + | Interaction { phase = Awaiting_e2ee_password; _ } -> true + | _ -> false) + "password submission flashes a different waiting page" ) + ; ( "account lookup failure Retry and stale replies" + , fun () -> + let initial = P.create () in + let old = P.account_ticket initial in + let failed = P.step initial (Account_error old) in + require + (P.screen failed = Account_failure) + "lookup failure has no explicit recovery"; + let retry = P.step failed Retry in + require + (P.screen retry = Waiting + && (not (P.loading_indicator retry)) + && P.account_ticket retry <> old) + "Retry does not start fresh discovery/grace"; + require + (P.step retry (Account (old, No_user)) == retry) + "old account result replaces retry"; + require + (P.step retry (Account_error old) == retry) + "old lookup failure replaces retry" ) + ; ( "local offline recovery is independent of online auth and assets" + , fun () -> + let t = account (P.create ()) Local in + let t = P.step t (Account_error (P.account_ticket t)) in + let ready = + observe + ~manager:(startup_snapshot ~sync_phase:Offline ()) + ~graph_ready:true + ~feed_loaded:true + t + in + require + (P.screen ready = Timeline) + "valid cached Timeline waits for online auth/assets" ) + ; ( "graph replacement rejects old ready data and old timer" + , fun () -> + let ready = + observe + ~manager:(startup_snapshot ()) + ~graph_ready:true + ~feed_loaded:true + (P.create ()) + in + let other = + { (startup_snapshot ~graph_generation:8 ()) with + selected_graph = Some graph_id + } + in + let waiting = + observe + ~manager:other + ~graph:(graph_state ~generation:7 ()) + ~graph_ready:true + ~feed_loaded:true + ready + in + require (P.screen waiting = Waiting) "stale graph falsely publishes ready data"; + let key = timer waiting in + let picker = + observe + ~manager:(startup_snapshot ~awaiting_selection:true ~graph_generation:8 ()) + ~graph:(graph_state ~phase:Graph_closed ~generation:8 ()) + waiting + in + let fresh = + observe + ~manager:(startup_snapshot ~graph_generation:9 ~restoring_local:true ()) + ~graph:(graph_state ~phase:Graph_opening ~generation:9 ()) + picker + in + require + (timer fresh <> key && P.step fresh (Elapsed (key, 1.)) == fresh) + "old graph journey timer enables new loading" ) + ; ( "background interruption fences elapsed and resumes a fresh grace" + , fun () -> + let t = + observe ~manager:(startup_snapshot ~restoring_local:true ()) (P.create ()) + in + let old = timer t in + let inactive = P.step t (Foreground false) in + require + (P.loading_request inactive = None + && P.step inactive (Elapsed (old, 1.)) == inactive) + "background keeps an active loading timer"; + let resumed = P.step inactive (Foreground true) in + require + (timer resumed <> old && not (P.loading_indicator resumed)) + "resume reuses old grace"; + require + (P.step resumed (Elapsed (old, 3600.)) == resumed) + "late background timer changes resume"; + let key = timer resumed in + require + (P.step resumed (Elapsed (key, 0.099999999)) == resumed) + "resume counts suspended time towards its fresh grace"; + require + (P.loading_indicator (P.step resumed (Elapsed (key, 0.1)))) + "resume does not show loading after its fresh 100ms grace" ) + ; ( "negative/NaN elapsed and explicit signout are safe" + , fun () -> + let t = + observe ~manager:(startup_snapshot ~restoring_local:true ()) (P.create ()) + in + let key = timer t in + require + (P.step t (Elapsed (key, -1.)) == t && P.step t (Elapsed (key, nan)) == t) + "invalid elapsed changes owner"; + let signed = P.step t P.Signed_out in + require + (match P.screen signed with + | Interaction { phase = Signed_out; _ } -> true + | _ -> false) + "signout entry is hidden"; + require + (P.step signed (Elapsed (key, 2.)) == signed) + "signout leaves stale timer active" ) + ] + in + let failures = ref [] in + List.iter + (fun (name, test) -> + try test () with + | error -> + Printf.eprintf "STARTUP_RED %s: %s\n%!" name (Printexc.to_string error); + failures := name :: !failures) + cases; + require + (!failures = []) + "Startup regressions: %s" + (String.concat ", " (List.rev !failures)) ;; (* Startup recovery projection owns action choice, not Core challenge storage. *) let test_e2ee_recovery_requires_a_prepared_challenge () = - List.iter (fun ready -> - let state = Journal_startup.derive - ~snapshot:(startup_snapshot ~failure:During_e2ee ~awaiting_e2ee_password:ready ()) - ~graph:(graph_state ~phase:Graph_closed ()) in - require (state.phase = Failed) "E2EE failure was hidden"; - let expected = if ready then Journal_startup.Submit_e2ee_password - else Journal_startup.Begin_online_recovery in - require (Option.bind state.error (fun error -> error.Journal_startup.recovery) - = Some expected) "E2EE recovery offers password without a prepared challenge") [false;true] + List.iter + (fun ready -> + let state = + Journal_startup.derive + ~snapshot: + (startup_snapshot ~failure:During_e2ee ~awaiting_e2ee_password:ready ()) + ~graph:(graph_state ~phase:Graph_closed ()) + in + require (state.phase = Failed) "E2EE failure was hidden"; + let expected = + if ready + then Journal_startup.Submit_e2ee_password + else Journal_startup.Begin_online_recovery + in + require + (Option.bind state.error (fun error -> error.Journal_startup.recovery) + = Some expected) + "E2EE recovery offers password without a prepared challenge") + [ false; true ] ;; let test_online_key_recovery_can_present_its_password_challenge () = - List.iter (fun failed -> - let snapshot=startup_snapshot ~awaiting_e2ee_password:true ?failure:(if failed then Some Graph_service.During_e2ee else None) () in - let snapshot={snapshot with startup={snapshot.startup with - local_access=Graph_service.Local_unavailable Graph_service.Local_key_unavailable; - network_phase=Graph_service.Online_by_user}} in - let derived=Journal_startup.derive ~snapshot ~graph:(graph_state ~phase:Graph_closed ()) in - if failed then - require (Option.bind derived.error (fun e->e.Journal_startup.recovery)=Some Journal_startup.Submit_e2ee_password) - "failed prepared challenge must still permit password retry" - else require (derived.phase=Journal_startup.Awaiting_e2ee_password) - "explicit recovery must show its prepared password challenge" - ) [false;true] + List.iter + (fun failed -> + let snapshot = + startup_snapshot + ~awaiting_e2ee_password:true + ?failure:(if failed then Some Graph_service.During_e2ee else None) + () + in + let snapshot = + { snapshot with + startup = + { snapshot.startup with + local_access = + Graph_service.Local_unavailable Graph_service.Local_key_unavailable + ; network_phase = Graph_service.Online_by_user + } + } + in + let derived = + Journal_startup.derive ~snapshot ~graph:(graph_state ~phase:Graph_closed ()) + in + if failed + then + require + (Option.bind derived.error (fun e -> e.Journal_startup.recovery) + = Some Journal_startup.Submit_e2ee_password) + "failed prepared challenge must still permit password retry" + else + require + (derived.phase = Journal_startup.Awaiting_e2ee_password) + "explicit recovery must show its prepared password challenge") + [ false; true ] ;; let test_local_timeline_does_not_require_online_authentication () = - let snapshot = startup_snapshot ~authenticated:false ~timeline_presentation_pending:true () in - let snapshot = {snapshot with startup={snapshot.startup with local_access=Graph_service.Local_available}} in + let snapshot = + startup_snapshot ~authenticated:false ~timeline_presentation_pending:true () + in + let snapshot = + { snapshot with + startup = { snapshot.startup with local_access = Graph_service.Local_available } + } + in let module P = Journal_startup.Presentation in let t = P.create () in - let t = P.step t (Account(P.account_ticket t,Local)) in - let t = P.step t (Observe {manager=Some snapshot;graph=graph_state ();graph_ready=true; - feed_loaded=true;password_pending=false;graph_error=false}) in - require (P.screen t=P.Timeline) "usable local graph must render before authentication/appearance"; - let unavailable = {snapshot with startup={snapshot.startup with - local_access=Graph_service.Local_unavailable Graph_service.Local_mirror_unavailable; - restoring_local=false}} in - let t = P.step t (Observe {manager=Some unavailable;graph=graph_state ~phase:Graph_closed (); - graph_ready=false;feed_loaded=false;password_pending=false;graph_error=false}) in - require (match P.screen t with Interaction {phase=Failed;_}->true|_->false) + let t = P.step t (Account (P.account_ticket t, Local)) in + let t = + P.step + t + (Observe + { manager = Some snapshot + ; graph = graph_state () + ; graph_ready = true + ; feed_loaded = true + ; password_pending = false + ; graph_error = false + }) + in + require + (P.screen t = P.Timeline) + "usable local graph must render before authentication/appearance"; + let unavailable = + { snapshot with + startup = + { snapshot.startup with + local_access = + Graph_service.Local_unavailable Graph_service.Local_mirror_unavailable + ; restoring_local = false + } + } + in + let t = + P.step + t + (Observe + { manager = Some unavailable + ; graph = graph_state ~phase:Graph_closed () + ; graph_ready = false + ; feed_loaded = false + ; password_pending = false + ; graph_error = false + }) + in + require + (match P.screen t with + | Interaction { phase = Failed; _ } -> true + | _ -> false) "missing mirror must stop loading" ;; let () = + test_bootstrap_crypto_failure_is_terminal (); test_online_key_recovery_can_present_its_password_challenge (); test_local_timeline_does_not_require_online_authentication (); test_e2ee_recovery_requires_a_prepared_challenge (); diff --git a/test/worker_coordination_pure_test.ml b/test/worker_coordination_pure_test.ml index 881d1974..e52684a9 100644 --- a/test/worker_coordination_pure_test.ml +++ b/test/worker_coordination_pure_test.ml @@ -2,7 +2,7 @@ module W = Journal_worker_coordination.Worker_api module C = Journal_worker_coordination.Worker_pure_coordinator module S = Logseq_db_worker_lui.Logseq_db_worker_lui_service module Id = Journal_worker_ids -module Sync = Logseq_sync_pure_reducer.Core +module Sync = Logseq_sync_domain.Core let check condition message = if not condition then failwith message let uuid s = W.Uuid.of_string s |> Result.get_ok diff --git a/test/worker_coordination_runtime_test.ml b/test/worker_coordination_runtime_test.ml index aa0c2b98..179b355b 100644 --- a/test/worker_coordination_runtime_test.ml +++ b/test/worker_coordination_runtime_test.ml @@ -6,8 +6,7 @@ module Runtime = Logseq_db_worker_lui.Journal_worker_runtime module Id = Journal_worker_ids module Service = Logseq_db_worker_lui.Logseq_db_worker_lui_service module T = Logseq_db_worker_test_support.Test_support -module Sync = Logseq_sync_pure_reducer.Core -module SR = Logseq_sync_effect_runner.Effect_runner +module Sync = Logseq_sync_domain.Core let require condition message = if not condition then failwith message let uuid value = W.Uuid.of_string value |> Result.get_ok @@ -16,101 +15,166 @@ let uuid value = W.Uuid.of_string value |> Result.get_ok Consuming a retained reply must not replace a newer snapshot already seen. *) let remember_baseline baseline (value : Service.coordination_baseline) = match !baseline with - | Some current when current.Service.registration = value.registration - && current.baseline.version > value.baseline.version -> () + | Some current + when current.Service.registration = value.registration + && current.baseline.version > value.baseline.version -> () | _ -> baseline := Some value ;; -module Read=Worker_pure_read + +module Read = Worker_pure_read (* This Service-side source uses the real client's incarnation and its atomic baseline. It exercises the backend contract rather than fabricating an ACK. *) -let observation_read client baseline (owner:Service.timeline_scope) registration ?(maximum_items=1) ?(maximum_bytes=16384) query = - let module F=Read.For_worker in - let baseline=Option.get baseline in - let scope=List.find_map(fun((scope:Sync.graph_scope),_)-> - if scope.Logseq_sync_pure_reducer.Core.graph_id=owner.graph_id - && scope.graph_generation=owner.graph_generation - && scope.account.account_generation=owner.startup.account_generation then Some scope else None) - baseline.Service.baseline.state.residents |> Option.get in - let binding=F.binding ~serial:registration - ~incarnation:(Read.worker_incarnation ~runtime_epoch:(Mailbox.runtime_epoch client) ~worker_generation:(Mailbox.worker_generation client)) - ~origin:(Uri.to_string scope.account.managed_sync_origin) ~user:scope.account.user_id in - let session=F.session ~binding ~graph:scope.graph_id ~account:scope.account.account_generation - ~lifecycle:scope.account.lifecycle_generation ~generation:scope.graph_generation ~scope:owner ~asset_scope:scope ~limits:baseline.limits in - let budget=Read.read_budget session ~maximum_items ~maximum_bytes |> Result.get_ok in +let observation_read + client + baseline + (owner : Service.timeline_scope) + registration + ?(maximum_items = 1) + ?(maximum_bytes = 16384) + query + = + let module F = Read.For_worker in + let baseline = Option.get baseline in + let scope = + List.find_map + (fun ((scope : Sync.graph_scope), _) -> + if + scope.Logseq_sync_domain.Core.graph_id = owner.graph_id + && scope.graph_generation = owner.graph_generation + && scope.account.account_generation = owner.startup.account_generation + then Some scope + else None) + baseline.Service.baseline.state.residents + |> Option.get + in + let binding = + F.binding + ~serial:registration + ~incarnation: + (Read.worker_incarnation + ~runtime_epoch:(Mailbox.runtime_epoch client) + ~worker_generation:(Mailbox.worker_generation client)) + ~origin:(Uri.to_string scope.account.managed_sync_origin) + ~user:scope.account.user_id + in + let session = + F.session + ~binding + ~graph:scope.graph_id + ~account:scope.account.account_generation + ~lifecycle:scope.account.lifecycle_generation + ~generation:scope.graph_generation + ~scope:owner + ~asset_scope:scope + ~limits:baseline.limits + in + let budget = Read.read_budget session ~maximum_items ~maximum_bytes |> Result.get_ok in F.pack_query budget query ;; -let query_request client baseline owner registration request_id ?maximum_items ?maximum_bytes query = - Service.Coordination_query{registration;owner;request_id; - read=observation_read client baseline owner registration ?maximum_items ?maximum_bytes query} +let query_request + client + baseline + owner + registration + request_id + ?maximum_items + ?maximum_bytes + query + = + Service.Coordination_query + { registration + ; owner + ; request_id + ; read = + observation_read + client + baseline + owner + registration + ?maximum_items + ?maximum_bytes + query + } +;; + let mutation_request client baseline owner registration ~request_id ~mutation_id command = - let Read.Read_query(budget,_)=observation_read client baseline owner registration Read.Graph_info in - let session=Read.For_worker.read_budget_owner budget in - let intent=W.mutation_intent session ~mutation_id command |> Result.get_ok in - Service.Coordination_mutation{registration;owner;request_id; - intent=W.Mutation_intent intent} + let (Read.Read_query (budget, _)) = + observation_read client baseline owner registration Read.Graph_info + in + let session = Read.For_worker.read_budget_owner budget in + let intent = W.mutation_intent session ~mutation_id command |> Result.get_ok in + Service.Coordination_mutation + { registration; owner; request_id; intent = W.Mutation_intent intent } +;; let second_graph = uuid "9c000000-0000-4000-8000-000000000003" let client_id = uuid "9c000000-0000-4000-8000-000000000001" -let crypto = - SR.crypto - ~encrypt_aes_gcm:(fun ~key:_ ~plaintext:_ -> Error "unused") - ~decrypt_aes_gcm:(fun ~key:_ ~iv:_ ~ciphertext:_ -> Error "unused") - |> Result.get_ok -;; - -let secrets = - SR.secrets - ~unlock_private_key: - (fun - ~managed_sync_origin:_ ~user_id:_ ~password:_ ~private_key_package:_ -> - Error "unused") - ~unlock_graph_key:(fun ~managed_sync_origin:_ ~user_id:_ ~encrypted_graph_key:_ -> - Error "unused") - ~load_wrapped_graph_key:(fun ~managed_sync_origin:_ ~user_id:_ ~graph_id:_ -> - Error (SR.Wrapped_graph_key_unavailable "unused")) - ~verify_and_save_wrapped_graph_key: - (fun - ~managed_sync_origin:_ ~user_id:_ ~graph_id:_ ~encrypted_graph_key:_ -> - Error "unused") - ~delete_account_secrets:(fun ~managed_sync_origin:_ ~user_id:_ -> Ok ()) - |> Result.get_ok +let host_asset_demand + host + ~receive + ~(owner : Service.timeline_scope) + ~graph_generation + ~name + priority + assets + = + let module H = Journal_worker_coordination_host in + match + Option.bind + (C.projection (H.view host)) + (fun projection -> + List.find_map + (fun resident -> + if resident.W.phase = W.Open && W.matches_scope resident.session owner + then Some resident.session + else None) + (W.residents projection)) + with + | Some session when owner.graph_generation = graph_generation -> + (match W.asset_consumer session ~name with + | Error _ -> None + | Ok consumer -> + (match + H.request host session (W.Replace_asset_demand (consumer, priority, assets)) + with + | Error _ -> None + | Ok ticket -> + H.watch host ticket ~f:(function + | W.Answered (Ok demand) -> receive ticket demand + | _ -> ()); + Some ticket)) + | _ -> None ;; -let host_asset_demand host ~receive ~(owner:Service.timeline_scope) ~graph_generation ~name priority assets = - let module H=Journal_worker_coordination_host in - match Option.bind(C.projection(H.view host))(fun projection->List.find_map(fun resident-> - if resident.W.phase=W.Open && W.matches_scope resident.session owner then Some resident.session else None)(W.residents projection)) with - |Some session when owner.graph_generation=graph_generation-> - (match W.asset_consumer session ~name with Error _->None|Ok consumer-> - match H.request host session(W.Replace_asset_demand(consumer,priority,assets)) with - |Error _->None|Ok ticket->H.watch host ticket ~f:(function - |W.Answered(Ok demand)->receive ticket demand - |_->()); - Some ticket) - |_->None - -let host_asset_release host (ticket,demand) = - let module H=Journal_worker_coordination_host in +let host_asset_release host (ticket, demand) = + let module H = Journal_worker_coordination_host in match demand with - |None->H.post host(C.Cancel ticket);true - |Some demand-> - match H.request host (W.asset_demand_session demand)(W.Release_asset_demand demand) with - |Error _->false|Ok ticket->H.watch host ticket ~f:(fun _->());true + | None -> + H.post host (C.Cancel ticket); + true + | Some demand -> + (match + H.request host (W.asset_demand_session demand) (W.Release_asset_demand demand) + with + | Error _ -> false + | Ok ticket -> + H.watch host ticket ~f:(fun _ -> ()); + true) +;; let favorite_page_id = uuid "9c000000-0000-4000-8000-000000000100" let favorite_member n = uuid (Printf.sprintf "9c000000-0000-4000-8000-%012d" (100 + n)) - -let byte_root n = uuid (Printf.sprintf "9c000000-0000-4000-8000-%012d" (700+n)) -let quota_root n = uuid (Printf.sprintf "9c000000-0000-4000-8000-%012d" (500+n)) +let byte_root n = uuid (Printf.sprintf "9c000000-0000-4000-8000-%012d" (700 + n)) +let quota_root n = uuid (Printf.sprintf "9c000000-0000-4000-8000-%012d" (500 + n)) let preview_root = uuid "9c000000-0000-4000-8000-000000000400" -let preview_child n = uuid (Printf.sprintf "9c000000-0000-4000-8000-%012d" (400+n)) -let summary_child n = uuid (Printf.sprintf "9c000000-0000-4000-8000-%012d" (300+n)) -let journal_page n = uuid (Printf.sprintf "9c000000-0000-4000-8000-%012d" (200+n)) +let preview_child n = uuid (Printf.sprintf "9c000000-0000-4000-8000-%012d" (400 + n)) +let summary_child n = uuid (Printf.sprintf "9c000000-0000-4000-8000-%012d" (300 + n)) +let journal_page n = uuid (Printf.sprintf "9c000000-0000-4000-8000-%012d" (200 + n)) -let seed_coordination_read_data ?(assets=false) path = +let seed_coordination_read_data ?(assets = false) path = let module Storage = Logseq_db_storage.Logseq_sqlite_storage in let module Session = Logseq_db_storage.Storage_session in let connection = Storage.open_database path |> Result.get_ok in @@ -165,67 +229,147 @@ let seed_coordination_read_data ?(assets=false) path = ] @ member 1 Test_support.authoritative_block_uuid @ member 2 Test_support.page_uuid - @ List.concat_map (fun n -> - let node = Datascript.Temp_id ("synthetic-journal-" ^ string_of_int n) in - Datascript.[ - Add (node,"block/uuid",Uuid (W.Uuid.to_string (journal_page n))); - Add (node,"block/name",String ("synthetic-journal-" ^ string_of_int n)); - Add (node,"block/title",String "Synthetic journal"); - Add (node,"block/journal-day",Int64 (if n=4 then 20260831L else if n=3 then 20260829L else 20260830L)); - Add (node,"logseq.property/public?",Bool true)]) [1;2;3;4] + @ List.concat_map + (fun n -> + let node = Datascript.Temp_id ("synthetic-journal-" ^ string_of_int n) in + Datascript. + [ Add (node, "block/uuid", Uuid (W.Uuid.to_string (journal_page n))) + ; Add + (node, "block/name", String ("synthetic-journal-" ^ string_of_int n)) + ; Add (node, "block/title", String "Synthetic journal") + ; Add + ( node + , "block/journal-day" + , Int64 + (if n = 4 + then 20260831L + else if n = 3 + then 20260829L + else 20260830L) ) + ; Add (node, "logseq.property/public?", Bool true) + ]) + [ 1; 2; 3; 4 ] + in + let ops = + ops + @ List.concat_map + (fun n -> + let node = + Datascript.Temp_id ("synthetic-summary-child-" ^ string_of_int n) + in + Datascript. + [ Add (node, "block/uuid", Uuid (W.Uuid.to_string (summary_child n))) + ; Add (node, "block/title", String "Synthetic summary child") + ; Add (node, "block/order", String ("a" ^ string_of_int n)) + ; Add + ( node + , "block/parent" + , Ref_to (lookup Test_support.authoritative_block_uuid) ) + ; Add (node, "block/page", Ref_to (lookup Test_support.page_uuid)) + ]) + [ 1; 2 ] + in + let node = Datascript.Temp_id "synthetic-preview-root" in + let preview = + Datascript. + [ Add (node, "block/uuid", Uuid (W.Uuid.to_string preview_root)) + ; Add (node, "block/title", String "Synthetic preview root") + ; Add (node, "block/order", String "a") + ; Add (node, "block/parent", Ref_to (lookup (journal_page 2))) + ; Add (node, "block/page", Ref_to (lookup (journal_page 2))) + ] + in + let ops = + ops + @ preview + @ List.concat_map + (fun n -> + let child = + Datascript.Temp_id ("synthetic-preview-child-" ^ string_of_int n) + in + Datascript. + [ Add (child, "block/uuid", Uuid (W.Uuid.to_string (preview_child n))) + ; Add (child, "block/title", String ("Preview child " ^ string_of_int n)) + ; Add (child, "block/order", String ("a" ^ string_of_int n)) + ; Add (child, "block/parent", Ref_to node) + ; Add (child, "block/page", Ref_to (lookup (journal_page 2))) + ]) + [ 1; 2 ] + in + let ops = + ops + @ List.concat_map + (fun index -> + let n = index + 1 in + let node = + Datascript.Temp_id ("synthetic-quota-root-" ^ string_of_int n) + in + Datascript. + [ Add (node, "block/uuid", Uuid (W.Uuid.to_string (quota_root n))) + ; Add + ( node + , "block/title" + , String (if n <= 20 then "" else "Quota root " ^ string_of_int n) + ) + ; Add (node, "block/order", String (Printf.sprintf "a%04d" n)) + ; Add (node, "block/parent", Ref_to (lookup (journal_page 3))) + ; Add (node, "block/page", Ref_to (lookup (journal_page 3))) + ]) + (List.init 60 Fun.id) + in + let ops = + ops + @ List.concat_map + (fun index -> + let n = index + 1 in + let node = + Datascript.Temp_id ("synthetic-byte-root-" ^ string_of_int n) + in + Datascript. + [ Add (node, "block/uuid", Uuid (W.Uuid.to_string (byte_root n))) + ; Add + ( node + , "block/title" + , String + ((if n <= 30 + then "Before byte boundary " + else String.make 3000 'x') + ^ string_of_int n) ) + ; Add (node, "block/order", String (Printf.sprintf "a%04d" n)) + ; Add (node, "block/parent", Ref_to (lookup (journal_page 4))) + ; Add (node, "block/page", Ref_to (lookup (journal_page 4))) + ]) + (List.init 60 Fun.id) + in + let ops = + if not assets + then ops + else + ops + @ List.concat_map + (fun n -> + let node = + Datascript.Temp_id ("typed-scan-external-asset-" ^ string_of_int n) + in + Datascript. + [ Add + ( node + , "block/uuid" + , Uuid (Printf.sprintf "9c000000-0000-4000-8000-%012d" (900 + n)) + ) + ; Add (node, "block/title", String "Synthetic external attachment") + ; Add (node, "block/parent", Ref_to (lookup (journal_page 1))) + ; Add (node, "block/page", Ref_to (lookup (journal_page 1))) + ; Add (node, "block/order", String ("a" ^ string_of_int n)) + ; Add (node, "block/tags", Ref_to (Ident "logseq.class/Asset")) + ; Add (lookup (journal_page 1), "block/refs", Ref_to node) + ; Add + ( node + , "logseq.property.asset/external-url" + , String ("https://example.invalid/" ^ string_of_int n) ) + ]) + [ 1; 2; 3 ] in - let ops = ops @ List.concat_map (fun n -> - let node=Datascript.Temp_id ("synthetic-summary-child-" ^ string_of_int n) in - Datascript.[ - Add (node,"block/uuid",Uuid (W.Uuid.to_string (summary_child n))); - Add (node,"block/title",String "Synthetic summary child"); - Add (node,"block/order",String ("a" ^ string_of_int n)); - Add (node,"block/parent",Ref_to (lookup Test_support.authoritative_block_uuid)); - Add (node,"block/page",Ref_to (lookup Test_support.page_uuid))]) [1;2] in - let node=Datascript.Temp_id "synthetic-preview-root" in - let preview = Datascript.[ - Add (node,"block/uuid",Uuid (W.Uuid.to_string preview_root)); - Add (node,"block/title",String "Synthetic preview root"); - Add (node,"block/order",String "a"); - Add (node,"block/parent",Ref_to (lookup (journal_page 2))); - Add (node,"block/page",Ref_to (lookup (journal_page 2)))] in - let ops=ops @ preview @ List.concat_map (fun n -> - let child=Datascript.Temp_id ("synthetic-preview-child-" ^ string_of_int n) in - Datascript.[ - Add (child,"block/uuid",Uuid (W.Uuid.to_string (preview_child n))); - Add (child,"block/title",String ("Preview child " ^ string_of_int n)); - Add (child,"block/order",String ("a" ^ string_of_int n)); - Add (child,"block/parent",Ref_to node); - Add (child,"block/page",Ref_to (lookup (journal_page 2)))]) [1;2] in - let ops=ops @ List.concat_map (fun index -> - let n=index+1 in - let node=Datascript.Temp_id ("synthetic-quota-root-" ^ string_of_int n) in - Datascript.[ - Add (node,"block/uuid",Uuid (W.Uuid.to_string (quota_root n))); - Add (node,"block/title",String (if n<=20 then "" else "Quota root " ^ string_of_int n)); - Add (node,"block/order",String (Printf.sprintf "a%04d" n)); - Add (node,"block/parent",Ref_to (lookup (journal_page 3))); - Add (node,"block/page",Ref_to (lookup (journal_page 3)))]) (List.init 60 Fun.id) in - let ops=ops @ List.concat_map (fun index -> - let n=index+1 in - let node=Datascript.Temp_id ("synthetic-byte-root-" ^ string_of_int n) in - Datascript.[ - Add (node,"block/uuid",Uuid (W.Uuid.to_string (byte_root n))); - Add (node,"block/title",String ((if n<=30 then "Before byte boundary " else String.make 3000 'x') ^ string_of_int n)); - Add (node,"block/order",String (Printf.sprintf "a%04d" n)); - Add (node,"block/parent",Ref_to (lookup (journal_page 4))); - Add (node,"block/page",Ref_to (lookup (journal_page 4)))]) (List.init 60 Fun.id) in - let ops=if not assets then ops else ops @ List.concat_map(fun n-> - let node=Datascript.Temp_id ("typed-scan-external-asset-" ^ string_of_int n) in - Datascript.[ - Add(node,"block/uuid",Uuid(Printf.sprintf "9c000000-0000-4000-8000-%012d" (900+n))); - Add(node,"block/title",String "Synthetic external attachment"); - Add(node,"block/parent",Ref_to(lookup(journal_page 1))); - Add(node,"block/page",Ref_to(lookup(journal_page 1))); - Add(node,"block/order",String("a" ^ string_of_int n)); - Add(node,"block/tags",Ref_to(Ident "logseq.class/Asset")); - Add(lookup(journal_page 1),"block/refs",Ref_to node); - Add(node,"logseq.property.asset/external-url",String("https://example.invalid/" ^ string_of_int n))]) [1;2;3] in let staged = Session.stage_transact session ~authoritative_before:database ops |> Result.get_ok @@ -279,21 +423,42 @@ let seed_status_catalog support = ;; let prepare_atomic_append_fixture support = - let module D=Logseq_overlay_db.Database in - let module J=Transit_core.Json in - Eio_main.run(fun _->Eio.Switch.run(fun sw-> - let inspection=D.inspect_mirror ~application_support_directory:support ~graph_id:Test_support.graph_uuid |> Result.get_ok in - let database=D.open_ ~sw (Test_support.dependencies ~behavior:"atomic append fixture") inspection - ~graph_name:"atomic append fixture" |> Result.get_ok in - Fun.protect ~finally:(fun()->ignore(D.close database))(fun()-> - let web=J.String "existing-web-append-day" in - let add attribute value=Test_support.status_add web attribute value in - Test_support.apply_test_transaction database ~cursor:2 - [Test_support.status_add (J.Keyword "logseq.class/Journal") - "logseq.property.journal/title-format" (J.String "yyyy-MM-dd"); - add "block/uuid" (J.Uuid "9f000000-0000-4000-8000-000000002100"); - add "block/title" (J.String "Existing Web day title");add "block/name" (J.String "existing-web-append-day"); - add "block/journal-day" (J.Int 20261103);add "block/tags" (J.Keyword "logseq.class/Journal")]))) + let module D = Logseq_overlay_db.Database in + let module J = Transit_core.Json in + Eio_main.run (fun _ -> + Eio.Switch.run (fun sw -> + let inspection = + D.inspect_mirror + ~application_support_directory:support + ~graph_id:Test_support.graph_uuid + |> Result.get_ok + in + let database = + D.open_ + ~sw + (Test_support.dependencies ~behavior:"atomic append fixture") + inspection + ~graph_name:"atomic append fixture" + |> Result.get_ok + in + Fun.protect + ~finally:(fun () -> ignore (D.close database)) + (fun () -> + let web = J.String "existing-web-append-day" in + let add attribute value = Test_support.status_add web attribute value in + Test_support.apply_test_transaction + database + ~cursor:2 + [ Test_support.status_add + (J.Keyword "logseq.class/Journal") + "logseq.property.journal/title-format" + (J.String "yyyy-MM-dd") + ; add "block/uuid" (J.Uuid "9f000000-0000-4000-8000-000000002100") + ; add "block/title" (J.String "Existing Web day title") + ; add "block/name" (J.String "existing-web-append-day") + ; add "block/journal-day" (J.Int 20261103) + ; add "block/tags" (J.Keyword "logseq.class/Journal") + ]))) ;; let with_service_fixture @@ -311,7 +476,9 @@ let with_service_fixture let first_path = Test_support.seed_mirror root in if favorites then seed_coordination_read_data ~assets first_path; ignore (Test_support.seed_mirror ~graph_id:second_graph root); - List.iter(fun graph_id->ignore(Test_support.seed_mirror ~graph_id root))extra_graphs; + List.iter + (fun graph_id -> ignore (Test_support.seed_mirror ~graph_id root)) + extra_graphs; if statuses then seed_status_catalog root; prepare root; let graph : Sync.graph = @@ -336,11 +503,15 @@ let with_service_fixture ~user_id:"user-1" ~graphs: ([ graph - ; { graph with - graph_id = second_graph - ; name = "Second coordination synthetic" - } - ] @ List.map(fun graph_id->{graph with graph_id;name="Extra synthetic graph"})extra_graphs) + ; { graph with + graph_id = second_graph + ; name = "Second coordination synthetic" + } + ] + @ List.map + (fun graph_id -> + { graph with graph_id; name = "Extra synthetic graph" }) + extra_graphs) ~selected_graph:(Some graph.graph_id) |> Sync.encode_catalog_cache)); let deps = @@ -349,58 +520,94 @@ let with_service_fixture (Logseq_overlay_db.Database.dependencies ~epoch_ms ~monotonic_ns:(fun () -> 1L) - ~limits:(let limits=Logseq_overlay_db.Database.dependency_limits fixture.overlay in - match change_max_items with None->limits|Some change_max_items->{limits with change_max_items}) + ~limits: + (let limits = + Logseq_overlay_db.Database.dependency_limits fixture.overlay + in + match change_max_items with + | None -> limits + | Some change_max_items -> { limits with change_max_items }) |> Result.get_ok) - ~crypto - ~secrets - ~tls_authenticator:(SR.system_tls_authenticator () |> Result.get_ok) in f deps fixture.config) ;; -let with_client_sessions ?epoch_ms ?favorites ?assets ?statuses ?change_max_items ?prepare f = - with_service_fixture ?epoch_ms ?favorites ?assets ?statuses ?change_max_items ?prepare (fun deps config -> - let current = ref None and epoch = ref 8999L in - let stop () = Option.iter Runtime.stop !current; current := None in - let start () = - stop (); - epoch := Int64.succ !epoch; - let client = Runtime.start ~runtime_epoch:(Id.Runtime.Epoch.of_int64 !epoch) - (Service.create ~dependencies:deps) config |> Result.get_ok in - current := Some client; - client - in - Fun.protect ~finally:stop (fun () -> f start)) +let with_client_sessions + ?epoch_ms + ?favorites + ?assets + ?statuses + ?change_max_items + ?prepare + f + = + with_service_fixture + ?epoch_ms + ?favorites + ?assets + ?statuses + ?change_max_items + ?prepare + (fun deps config -> + let current = ref None + and epoch = ref 8999L in + let stop () = + Option.iter Runtime.stop !current; + current := None + in + let start () = + stop (); + epoch := Int64.succ !epoch; + let client = + Runtime.start + ~runtime_epoch:(Id.Runtime.Epoch.of_int64 !epoch) + (Service.create ~dependencies:deps) + config + |> Result.get_ok + in + current := Some client; + client + in + Fun.protect ~finally:stop (fun () -> f start)) ;; let with_client ?epoch_ms ?favorites ?assets ?statuses ?change_max_items ?prepare f = - with_client_sessions ?epoch_ms ?favorites ?assets ?statuses ?change_max_items ?prepare + with_client_sessions + ?epoch_ms + ?favorites + ?assets + ?statuses + ?change_max_items + ?prepare (fun start -> f (start ())) ;; let with_client_output_wait ?epoch_ms ?prepare ?change_max_items f = let input, output = Unix.pipe ~cloexec:true () in Fun.protect - ~finally:(fun () -> Unix.close input; Unix.close output) + ~finally:(fun () -> + Unix.close input; + Unix.close output) (fun () -> Unix.set_nonblock input; Unix.set_nonblock output; - let signal = Bytes.make 1 '\000' and buffer = Bytes.create 4096 in + let signal = Bytes.make 1 '\000' + and buffer = Bytes.create 4096 in with_client ?epoch_ms ?prepare ?change_max_items (fun client -> Mailbox.Private.set_output_wakeup client (fun () -> try ignore (Unix.write output signal 0 1) with | Unix.Unix_error ((Unix.EAGAIN | Unix.EWOULDBLOCK), _, _) -> ()); let wait ~pump ~timeout ~label predicate = let deadline = Unix.gettimeofday () +. timeout in - while not (predicate ()) && Unix.gettimeofday () < deadline do + while (not (predicate ())) && Unix.gettimeofday () < deadline do (try ignore (Unix.read input buffer 0 (Bytes.length buffer)) with | Unix.Unix_error ((Unix.EAGAIN | Unix.EWOULDBLOCK), _, _) -> ()); pump (); let remaining = deadline -. Unix.gettimeofday () in - if not (predicate ()) && remaining > 0. then - (try ignore (Unix.select [input] [] [] (min remaining 0.01)) with - | Unix.Unix_error (Unix.EINTR, _, _) -> ()) + if (not (predicate ())) && remaining > 0. + then ( + try ignore (Unix.select [ input ] [] [] (min remaining 0.01)) with + | Unix.Unix_error (Unix.EINTR, _, _) -> ()) done; require (predicate ()) label in @@ -413,294 +620,680 @@ let with_client_output_wait ?epoch_ms ?prepare ?change_max_items f = reducer. Exercise the public runner with a real admitted request, then replace only its transport outcome; the reducer never receives a fabricated error. *) let test_cleanup_transport_causes () = - let failures=ref [] in - let check condition message=if not condition then failures:=message::!failures in - let run stage outcome suffix = with_client (fun client -> - let events=Queue.create () and state=ref(C.initial ~client_id) in - let runner=R.create ~client ~managed_origin:"https://api.logseq.io" ~user_id:"user-1" - ~post:(fun event->Queue.add event events) ~publish:(fun _->()) |> Result.get_ok in - let injecting=ref false and captured=ref [] and delivery_errors=ref [] and imported=ref None in - Mailbox.on_event client(fun event-> - let event=match event,!injecting with - |Mailbox.Response response,true->Mailbox.Response{response with outcome} - |_->event in - let delivery=R.on_event runner event in - if !injecting then delivery_errors:=delivery.error::!delivery_errors); - let apply transition=state:=transition.C.next;List.iter(fun instruction-> - (match instruction,stage with - |C.Release_observation_registration _,"Observation release" - |C.Finalize_observation_registration _,"Observation finalization" - |C.Release_binding _,"Binding unregister" - |C.Send(_,_,W.Import _),"Import"->injecting:=true - |_->());R.submit runner instruction)transition.instructions in - let pump ()= - Mailbox.Private.deliver client ~max_events:64; - let count=Queue.length events in - for _=1 to count do - let event=Queue.take events in - if !injecting then captured:=event::!captured else apply(C.step !state event) - done in - let wait message predicate= - let deadline=Unix.gettimeofday()+.4. in - while not(predicate()) && Unix.gettimeofday()match C.bootstrap(C.view !state)with W.Bootstrap_not_ready _->true|_->false); - ignore(Mailbox.send client(Service.Client_command(Restore_local_account{user_id="user-1"}))); - let session ()=Option.bind(C.projection(C.view !state))(fun projection-> - List.find_map(fun resident->if resident.W.phase=W.Open then Some resident.session else None)(W.residents projection)) in - wait "cleanup fixture session" (fun()->Option.is_some(session())); - let session=Option.get(session()) in - pump(); - let push notice=R.on_event runner(Mailbox.Push{ - runtime_epoch=Mailbox.runtime_epoch client;worker_generation=Mailbox.worker_generation client; - push_sequence=Id.Worker.Push_sequence.of_int64 999L;topic=Service.asset_topic; - payload=Service.Asset_notice(Read.For_worker.session_asset_scope session,notice)}) in - require((push Service.Asset_capacity_available).error=None) - "valid asset notice became a transport error"; - check((push(Service.Asset_demand_accepted "foreign-demand-owner")).error=Some W.Revoked_session) - "asset notice decoding erased its typed ownership cause"; - if stage="Binding unregister" then apply(C.step !state C.Shutdown_requested) - else if stage="Import" then ( - let source:Logseq_db_types.Asset_import.t={ - operation=uuid "9c000000-0000-4000-8000-000000009921"; - asset=uuid "9c000000-0000-4000-8000-000000009922";target=Test_support.authoritative_block_uuid; - local_mutation=uuid "9c000000-0000-4000-8000-000000009923"; - metadata_mutation=uuid "9c000000-0000-4000-8000-000000009924"; - replace_reference=None;source_file="/synthetic-missing/import.png";title="Synthetic import";file_type="png"} in - let intent=W.asset_import_intent session source |> Result.get_ok in - imported:=Some intent; - let _,plan=C.request !state session(W.Import intent) |> Result.get_ok in apply plan) - else ( - let budget=W.read_budget session ~maximum_items:1 ~maximum_bytes:65536 |> Result.get_ok in - let ticket,plan=C.request !state session(W.Observe(budget,W.Graph_info)) |> Result.get_ok in - apply plan;wait "cleanup fixture observation" (fun()->Option.is_some(C.reply(C.view !state)ticket)); - let handle=match C.reply(C.view !state)ticket with Some(W.Answered(Ok value))->value.W.observation - |_->failwith "cleanup fixture observation failed" in - apply(C.step !state(C.Forget ticket)); - let released,plan=C.request !state session(W.Release handle) |> Result.get_ok in - apply plan; - if stage="Observation finalization" then ( - wait "cleanup fixture release" (fun()->Option.is_some(C.reply(C.view !state)released)); - apply(C.step !state(C.Forget released)); - apply(C.step !state(C.Observation_returned handle)))); - if stage="Import" then ( - wait "Import completion missing" (fun()->List.exists(function C.Completed _->true|_->false)!captured); - check(List.mem (Some(W.Service_failed("Import: "^suffix))) !delivery_errors) - "Import transport failure erased its diagnostic cause"; - check(List.exists(function - |C.Completed(_,W.Interrupted(W.Transport_lost cause))->cause=W.Service_failed("Import: "^suffix) - |_->false)!captured) "Import reply erased its typed interruption cause"; - List.iter(fun event->state:=(C.step !state event).next)(List.rev !captured); - check(C.import_progress(C.view !state)(Option.get !imported)=Some W.Import_unknown) - "Import transport failure falsely settled its unknown outcome") - else ( - let result=function - |C.Observation_release_completed(_,_,result) when stage="Observation release"->Some result - |C.Observation_finalize_completed(_,_,result) when stage="Observation finalization"->Some result - |C.Binding_release_completed(_,result) when stage="Binding unregister"->Some result - |_->None in - wait (stage^" completion missing") (fun()->List.exists(fun event->Option.is_some(result event))!captured); - check(List.exists(fun event->result event=Some(Error(W.Service_failed(stage^": "^suffix))))!captured) - (stage^" erased its backend cause"))) in - List.iter(fun stage->List.iter(fun(outcome,suffix)->run stage outcome suffix) - [Mailbox.Failed "synthetic cleanup backend refused retirement","synthetic cleanup backend refused retirement"; - Mailbox.Cancelled,"cancelled before retirement acknowledgement"; - Mailbox.Shutdown,"worker stopped before retirement acknowledgement"]) - ["Observation release";"Observation finalization";"Binding unregister"]; - run "Import" (Mailbox.Failed "synthetic import transport lost") "synthetic import transport lost"; - require(!failures=[]) (String.concat "; " (List.rev !failures)) + let failures = ref [] in + let check condition message = if not condition then failures := message :: !failures in + let run stage outcome suffix = + with_client (fun client -> + let events = Queue.create () + and state = ref (C.initial ~client_id) in + let runner = + R.create + ~client + ~managed_origin:"https://api.logseq.io" + ~user_id:"user-1" + ~post:(fun event -> Queue.add event events) + ~publish:(fun _ -> ()) + |> Result.get_ok + in + let injecting = ref false + and captured = ref [] + and delivery_errors = ref [] + and imported = ref None in + Mailbox.on_event client (fun event -> + let event = + match event, !injecting with + | Mailbox.Response response, true -> Mailbox.Response { response with outcome } + | _ -> event + in + let delivery = R.on_event runner event in + if !injecting then delivery_errors := delivery.error :: !delivery_errors); + let apply transition = + state := transition.C.next; + List.iter + (fun instruction -> + (match instruction, stage with + | C.Release_observation_registration _, "Observation release" + | C.Finalize_observation_registration _, "Observation finalization" + | C.Release_binding _, "Binding unregister" + | C.Send (_, _, W.Import _), "Import" -> injecting := true + | _ -> ()); + R.submit runner instruction) + transition.instructions + in + let pump () = + Mailbox.Private.deliver client ~max_events:64; + let count = Queue.length events in + for _ = 1 to count do + let event = Queue.take events in + if !injecting + then captured := event :: !captured + else apply (C.step !state event) + done + in + let wait message predicate = + let deadline = Unix.gettimeofday () +. 4. in + while (not (predicate ())) && Unix.gettimeofday () < deadline do + pump (); + Thread.delay 0.001 + done; + require (predicate ()) message + in + wait "cleanup fixture registration" (fun () -> + match C.bootstrap (C.view !state) with + | W.Bootstrap_not_ready _ -> true + | _ -> false); + ignore + (Mailbox.send + client + (Service.Client_command (Restore_local_account { user_id = "user-1" }))); + let session () = + Option.bind + (C.projection (C.view !state)) + (fun projection -> + List.find_map + (fun resident -> + if resident.W.phase = W.Open then Some resident.session else None) + (W.residents projection)) + in + wait "cleanup fixture session" (fun () -> Option.is_some (session ())); + let session = Option.get (session ()) in + pump (); + let push notice = + R.on_event + runner + (Mailbox.Push + { runtime_epoch = Mailbox.runtime_epoch client + ; worker_generation = Mailbox.worker_generation client + ; push_sequence = Id.Worker.Push_sequence.of_int64 999L + ; topic = Service.asset_topic + ; payload = + Service.Asset_notice (Read.For_worker.session_asset_scope session, notice) + }) + in + require + ((push Service.Asset_capacity_available).error = None) + "valid asset notice became a transport error"; + check + ((push (Service.Asset_demand_accepted "foreign-demand-owner")).error + = Some W.Revoked_session) + "asset notice decoding erased its typed ownership cause"; + if stage = "Binding unregister" + then apply (C.step !state C.Shutdown_requested) + else if stage = "Import" + then ( + let source : Logseq_db_types.Asset_import.t = + { operation = uuid "9c000000-0000-4000-8000-000000009921" + ; asset = uuid "9c000000-0000-4000-8000-000000009922" + ; target = Test_support.authoritative_block_uuid + ; local_mutation = uuid "9c000000-0000-4000-8000-000000009923" + ; metadata_mutation = uuid "9c000000-0000-4000-8000-000000009924" + ; replace_reference = None + ; source_file = "/synthetic-missing/import.png" + ; title = "Synthetic import" + ; file_type = "png" + } + in + let intent = W.asset_import_intent session source |> Result.get_ok in + imported := Some intent; + let _, plan = C.request !state session (W.Import intent) |> Result.get_ok in + apply plan) + else ( + let budget = + W.read_budget session ~maximum_items:1 ~maximum_bytes:65536 |> Result.get_ok + in + let ticket, plan = + C.request !state session (W.Observe (budget, W.Graph_info)) |> Result.get_ok + in + apply plan; + wait "cleanup fixture observation" (fun () -> + Option.is_some (C.reply (C.view !state) ticket)); + let handle = + match C.reply (C.view !state) ticket with + | Some (W.Answered (Ok value)) -> value.W.observation + | _ -> failwith "cleanup fixture observation failed" + in + apply (C.step !state (C.Forget ticket)); + let released, plan = + C.request !state session (W.Release handle) |> Result.get_ok + in + apply plan; + if stage = "Observation finalization" + then ( + wait "cleanup fixture release" (fun () -> + Option.is_some (C.reply (C.view !state) released)); + apply (C.step !state (C.Forget released)); + apply (C.step !state (C.Observation_returned handle)))); + if stage = "Import" + then ( + wait "Import completion missing" (fun () -> + List.exists + (function + | C.Completed _ -> true + | _ -> false) + !captured); + check + (List.mem (Some (W.Service_failed ("Import: " ^ suffix))) !delivery_errors) + "Import transport failure erased its diagnostic cause"; + check + (List.exists + (function + | C.Completed (_, W.Interrupted (W.Transport_lost cause)) -> + cause = W.Service_failed ("Import: " ^ suffix) + | _ -> false) + !captured) + "Import reply erased its typed interruption cause"; + List.iter (fun event -> state := (C.step !state event).next) (List.rev !captured); + check + (C.import_progress (C.view !state) (Option.get !imported) + = Some W.Import_unknown) + "Import transport failure falsely settled its unknown outcome") + else ( + let result = function + | C.Observation_release_completed (_, _, result) + when stage = "Observation release" -> Some result + | C.Observation_finalize_completed (_, _, result) + when stage = "Observation finalization" -> Some result + | C.Binding_release_completed (_, result) when stage = "Binding unregister" -> + Some result + | _ -> None + in + wait (stage ^ " completion missing") (fun () -> + List.exists (fun event -> Option.is_some (result event)) !captured); + check + (List.exists + (fun event -> + result event = Some (Error (W.Service_failed (stage ^ ": " ^ suffix)))) + !captured) + (stage ^ " erased its backend cause"))) + in + List.iter + (fun stage -> + List.iter + (fun (outcome, suffix) -> run stage outcome suffix) + [ ( Mailbox.Failed "synthetic cleanup backend refused retirement" + , "synthetic cleanup backend refused retirement" ) + ; Mailbox.Cancelled, "cancelled before retirement acknowledgement" + ; Mailbox.Shutdown, "worker stopped before retirement acknowledgement" + ]) + [ "Observation release"; "Observation finalization"; "Binding unregister" ]; + run + "Import" + (Mailbox.Failed "synthetic import transport lost") + "synthetic import transport lost"; + require (!failures = []) (String.concat "; " (List.rev !failures)) ;; let test_application_current_close ~replacement () = - let third=uuid "9c000000-0000-4000-8000-000000000004" - and fourth=uuid "9c000000-0000-4000-8000-000000000005" in - let prepare root=List.iter(fun graph_id-> - let path=Filename.concat root("logseq-db-worker/synced-graphs/" ^ W.Uuid.to_string graph_id ^ "/db.sqlite") in - seed_coordination_read_data ~assets:true path)[second_graph;third;fourth] in - with_service_fixture ~favorites:true ~assets:true ~extra_graphs:[third;fourth] ~prepare (fun deps config -> - let client=ref None and requests=Queue.create() in - let calendar_sampler=Journal_calendar.Sampler.create ~clock:(fun()->1788048000.) ~localtime:Unix.gmtime () in - let hooks=Application.For_testing.app_with_service ~calendar_sampler ~on_client:(fun value->client:=Some value) - ~on_platform_request:(fun packet->Queue.add(Bytes.copy packet)requests) - (Service.create ~dependencies:deps) in - Fun.protect ~finally:(fun()->ignore(hooks.dispose());Option.iter Runtime.stop !client)(fun()-> - let controls=Hashtbl.create 128 and parents=Hashtbl.create 128 and appearances=Queue.create() in - let selected=ref None and residents=ref [] in - let rec drop id= - Hashtbl.to_seq parents |> List.of_seq |> List.iter(fun(child,parent)->if parent=id then drop child); - Hashtbl.remove parents id;Hashtbl.remove controls id in - let patch_path=Filename.concat config.Logseq_db_worker.Config.application_support_directory "mounted-ui-patch.json" in - let consume source=if source<>"" then ( - Out_channel.with_open_bin patch_path (fun out->output_string out source); - let fields=function `Assoc fields->fields|_->failwith "invalid UI patch object" in - let int=function `Int value->value|_->failwith "invalid UI patch node" in - let ops=match List.assoc "ops" (fields(T.read_json patch_path))with - |`List ops->ops|_->failwith "invalid UI patch operations" in - List.iter(fun op->let fields=fields op in let get name=List.assoc name fields in - match get "op" with - |`String "create-extension"-> - Hashtbl.replace controls(int(get "id"))["_extension",(match get "identifier" with `String value->value|_->"")] - |`String("drop-node"|"drop-extension"|"detach-subtree")->drop(int(get "id")) - |`String("insert-child"|"move-child")->Hashtbl.replace parents(int(get "child"))(int(get "parent")) - |`String"remove-child"->Hashtbl.remove parents(int(get "child")) - |`String("set-prop"|"set-extension-prop")-> - let id=int(get "id") in - (match get "property",get "value" with - |`String "appear-enabled",`Bool true->Queue.add id appearances - |`String key,`Bool value-> - let old=Option.value(Hashtbl.find_opt controls id) ~default:[] in - Hashtbl.replace controls id ((key,string_of_bool value)::List.remove_assoc key old) - |`String key,`String value-> - let old=Option.value(Hashtbl.find_opt controls id) ~default:[] in - Hashtbl.replace controls id ((key,value)::List.remove_assoc key old) - |_->()) - |_->())ops) in - let rec mounted id= id=hooks.root_node() || match Hashtbl.find_opt parents id with None->false|Some parent->mounted parent in - let find key value=Hashtbl.fold(fun id fields found-> - if mounted id && List.assoc_opt key fields=Some value then Some(max id(Option.value found ~default:0))else found)controls None in - let native_authenticated=ref false and token_requested=ref false and registrations=ref 0 in - let packet tag json=let packet=Bytes.make(32+String.length json)'\000' in - Bytes.blit_string "LJP2" 0 packet 0 4;Bytes.set_uint16_le packet 4 2; - Bytes.set_uint16_le packet 6 tag;Bytes.set_int32_le packet 24(Int32.of_int(String.length json)); - Bytes.blit_string json 0 packet 32(String.length json);Bytes.to_string packet in - let answer_native ()=while not(Queue.is_empty requests)do - let request=Queue.take requests in - let response=match Bytes.get_uint16_le request 6 with - |20->Some(21,"{\"userId\":\"user-1\",\"managedSyncOrigin\":\"https://api.logseq.io\"}") - |6->Some(7,"{\"userId\":\"user-1\"}") - |22-> - if not !native_authenticated then ( - native_authenticated:=true; - let scope=Bytes.sub_string request 32 (Bytes.length request-32) in - hooks.platform_event(packet 7("{\"userId\":\"user-1\",\"startupScope\":" ^ scope ^ "}"))); - Some(23,"{\"presented\":true}") - |_->None in - Option.iter(fun(tag,json)->hooks.platform_response(packet tag json))response - done in - let wait label predicate= - let deadline=Unix.gettimeofday()+.5. in - while not(predicate()) && Unix.gettimeofday() - Option.fold ~none:false ~some:(fun graph->contains payload(W.Uuid.to_string graph))!selected - |_->match Hashtbl.find_opt parents id with None->true|Some parent->current_host parent in - let press key value= - let clickable ()=Hashtbl.fold(fun id fields found-> - if mounted id && current_host id && List.assoc_opt key fields=Some value && List.assoc_opt "press-enabled" fields=Some "true" - then Some id else found)controls None in - wait ("missing control: " ^ value)(fun()->Option.is_some(clickable())); - consume(hooks.dispatch(Press(Option.get(clickable())))) in - consume(hooks.init 2 2 (Journal_startup.encode config |> Result.get_ok |> Bytes.to_string)); - let worker=Option.get !client in - Mailbox.on_event worker(function Mailbox.Push{payload=Service.Client_state_changed state;_}-> - selected:=state.snapshot.selected_graph;residents:=state.snapshot.resident_graphs - |Mailbox.Push{payload=Service.Need_id_token _;_}->token_requested:=true - |Mailbox.Response{outcome=Completed(Service.Coordination_registered _);_}->incr registrations|_->()); - let send command=match Mailbox.send worker(Service.Client_command command)with - |Mailbox.Accepted _->()|_->failwith "real app fixture command refused" in - wait "real app restores first graph" (fun()-> !selected=Some Test_support.graph_uuid); - send(Configure_resident_graphs[Test_support.graph_uuid;second_graph;third]); - wait "real app admits three resident scopes" (fun()->List.length !residents=3); - press "accessibility-label" "Favorites"; - press "text" "Switch graph"; - wait "native identity reaches the deliberately blocked token request" (fun()-> !token_requested); - require (!registrations=1) "same-user authentication preserves the original app registration"; - if Option.is_none(find "accessibility-identifier" "graph-picker-list")then press "text" "Switch graph"; - List.iter(fun graph_id-> - press "accessibility-identifier" ("graph-picker:" ^ W.Uuid.to_string graph_id); - wait "user selection becomes current" (fun()-> !selected=Some graph_id); - wait "selection closes picker" (fun()->Option.is_none(find "accessibility-identifier" "graph-picker-list")); - press "text" "Switch graph"; - wait "selected graph opens picker" (fun()->Option.is_some(find "accessibility-identifier" "graph-picker-list")))[second_graph;third]; - if replacement then ( - press "accessibility-identifier" ("graph-picker:" ^ W.Uuid.to_string fourth); - press "accessibility-identifier" ("graph-resident-replace:" ^ W.Uuid.to_string third)) - else press "accessibility-identifier" ("graph-resident-remove:" ^ W.Uuid.to_string third); - let target=if replacement then fourth else Test_support.graph_uuid in - wait "real current close settles and selects retained/replacement graph" (fun()-> !selected=Some target && not(List.mem third !residents)); - press "text" "Switch graph"; - wait "retained graph responds to picker after retirement" (fun()->Option.is_some(find "accessibility-identifier" "graph-picker-list")); - press "accessibility-identifier" "graph-picker-close"; - press "accessibility-label" "Journals"; - press "accessibility-identifier" "journal-capture-open"; - wait "current graph opens a real Capture editor after retirement" (fun()->Option.is_some(find "accessibility-identifier" "field.composer")))) + let third = uuid "9c000000-0000-4000-8000-000000000004" + and fourth = uuid "9c000000-0000-4000-8000-000000000005" in + let prepare root = + List.iter + (fun graph_id -> + let path = + Filename.concat + root + ("logseq-db-worker/synced-graphs/" ^ W.Uuid.to_string graph_id ^ "/db.sqlite") + in + seed_coordination_read_data ~assets:true path) + [ second_graph; third; fourth ] + in + with_service_fixture + ~favorites:true + ~assets:true + ~extra_graphs:[ third; fourth ] + ~prepare + (fun deps config -> + let client = ref None + and requests = Queue.create () in + let calendar_sampler = + Journal_calendar.Sampler.create + ~clock:(fun () -> 1788048000.) + ~localtime:Unix.gmtime + () + in + let hooks = + Application.For_testing.app_with_service + ~calendar_sampler + ~on_client:(fun value -> client := Some value) + ~on_platform_request:(fun packet -> Queue.add (Bytes.copy packet) requests) + (Service.create ~dependencies:deps) + in + Fun.protect + ~finally:(fun () -> + ignore (hooks.dispose ()); + Option.iter Runtime.stop !client) + (fun () -> + let controls = Hashtbl.create 128 + and parents = Hashtbl.create 128 + and appearances = Queue.create () in + let selected = ref None + and residents = ref [] in + let rec drop id = + Hashtbl.to_seq parents + |> List.of_seq + |> List.iter (fun (child, parent) -> if parent = id then drop child); + Hashtbl.remove parents id; + Hashtbl.remove controls id + in + let patch_path = + Filename.concat + config.Logseq_db_worker.Config.application_support_directory + "mounted-ui-patch.json" + in + let consume source = + if source <> "" + then ( + Out_channel.with_open_bin patch_path (fun out -> output_string out source); + let fields = function + | `Assoc fields -> fields + | _ -> failwith "invalid UI patch object" + in + let int = function + | `Int value -> value + | _ -> failwith "invalid UI patch node" + in + let ops = + match List.assoc "ops" (fields (T.read_json patch_path)) with + | `List ops -> ops + | _ -> failwith "invalid UI patch operations" + in + List.iter + (fun op -> + let fields = fields op in + let get name = List.assoc name fields in + match get "op" with + | `String "create-extension" -> + Hashtbl.replace + controls + (int (get "id")) + [ ( "_extension" + , match get "identifier" with + | `String value -> value + | _ -> "" ) + ] + | `String ("drop-node" | "drop-extension" | "detach-subtree") -> + drop (int (get "id")) + | `String ("insert-child" | "move-child") -> + Hashtbl.replace parents (int (get "child")) (int (get "parent")) + | `String "remove-child" -> + Hashtbl.remove parents (int (get "child")) + | `String ("set-prop" | "set-extension-prop") -> + let id = int (get "id") in + (match get "property", get "value" with + | `String "appear-enabled", `Bool true -> Queue.add id appearances + | `String key, `Bool value -> + let old = + Option.value (Hashtbl.find_opt controls id) ~default:[] + in + Hashtbl.replace + controls + id + ((key, string_of_bool value) :: List.remove_assoc key old) + | `String key, `String value -> + let old = + Option.value (Hashtbl.find_opt controls id) ~default:[] + in + Hashtbl.replace + controls + id + ((key, value) :: List.remove_assoc key old) + | _ -> ()) + | _ -> ()) + ops) + in + let rec mounted id = + id = hooks.root_node () + || + match Hashtbl.find_opt parents id with + | None -> false + | Some parent -> mounted parent + in + let find key value = + Hashtbl.fold + (fun id fields found -> + if mounted id && List.assoc_opt key fields = Some value + then Some (max id (Option.value found ~default:0)) + else found) + controls + None + in + let native_authenticated = ref false + and token_requested = ref false + and registrations = ref 0 in + let packet tag json = + let packet = Bytes.make (32 + String.length json) '\000' in + Bytes.blit_string "LJP2" 0 packet 0 4; + Bytes.set_uint16_le packet 4 2; + Bytes.set_uint16_le packet 6 tag; + Bytes.set_int32_le packet 24 (Int32.of_int (String.length json)); + Bytes.blit_string json 0 packet 32 (String.length json); + Bytes.to_string packet + in + let answer_native () = + while not (Queue.is_empty requests) do + let request = Queue.take requests in + let response = + match Bytes.get_uint16_le request 6 with + | 20 -> + Some + ( 21 + , "{\"userId\":\"user-1\",\"managedSyncOrigin\":\"https://api.logseq.io\"}" + ) + | 6 -> Some (7, "{\"userId\":\"user-1\"}") + | 22 -> + if not !native_authenticated + then ( + native_authenticated := true; + let scope = + Bytes.sub_string request 32 (Bytes.length request - 32) + in + hooks.platform_event + (packet + 7 + ("{\"userId\":\"user-1\",\"startupScope\":" ^ scope ^ "}"))); + Some (23, "{\"presented\":true}") + | _ -> None + in + Option.iter + (fun (tag, json) -> hooks.platform_response (packet tag json)) + response + done + in + let wait label predicate = + let deadline = Unix.gettimeofday () +. 5. in + while (not (predicate ())) && Unix.gettimeofday () < deadline do + answer_native (); + while not (Queue.is_empty appearances) do + consume (hooks.dispatch (Appear (Queue.take appearances))) + done; + consume (hooks.pump ()); + Thread.delay 0.001 + done; + require (predicate ()) label + in + let contains source part = + let rec loop n = + n + String.length part <= String.length source + && (String.sub source n (String.length part) = part || loop (n + 1)) + in + loop 0 + in + let rec current_host id = + let fields = Option.value (Hashtbl.find_opt controls id) ~default:[] in + match List.assoc_opt "payload" fields with + | Some payload + when String.starts_with ~prefix:"{\"mode\":\"timeline-host\"" payload -> + Option.fold + ~none:false + ~some:(fun graph -> contains payload (W.Uuid.to_string graph)) + !selected + | _ -> + (match Hashtbl.find_opt parents id with + | None -> true + | Some parent -> current_host parent) + in + let press key value = + let clickable () = + Hashtbl.fold + (fun id fields found -> + if + mounted id + && current_host id + && List.assoc_opt key fields = Some value + && List.assoc_opt "press-enabled" fields = Some "true" + then Some id + else found) + controls + None + in + wait ("missing control: " ^ value) (fun () -> Option.is_some (clickable ())); + consume (hooks.dispatch (Press (Option.get (clickable ())))) + in + consume + (hooks.init + 2 + 2 + (Journal_startup.encode config |> Result.get_ok |> Bytes.to_string)); + let worker = Option.get !client in + Mailbox.on_event worker (function + | Mailbox.Push { payload = Service.Client_state_changed state; _ } -> + selected := state.snapshot.selected_graph; + residents := state.snapshot.resident_graphs + | Mailbox.Push { payload = Service.Need_id_token _; _ } -> + token_requested := true + | Mailbox.Response + { outcome = Completed (Service.Coordination_registered _); _ } -> + incr registrations + | _ -> ()); + let send command = + match Mailbox.send worker (Service.Client_command command) with + | Mailbox.Accepted _ -> () + | _ -> failwith "real app fixture command refused" + in + wait "real app restores first graph" (fun () -> + !selected = Some Test_support.graph_uuid); + send + (Configure_resident_graphs [ Test_support.graph_uuid; second_graph; third ]); + wait "real app admits three resident scopes" (fun () -> + List.length !residents = 3); + press "accessibility-label" "Favorites"; + press "text" "Switch graph"; + wait + "native identity reaches the deliberately blocked token request" + (fun () -> !token_requested); + require + (!registrations = 1) + "same-user authentication preserves the original app registration"; + if Option.is_none (find "accessibility-identifier" "graph-picker-list") + then press "text" "Switch graph"; + List.iter + (fun graph_id -> + press + "accessibility-identifier" + ("graph-picker:" ^ W.Uuid.to_string graph_id); + wait "user selection becomes current" (fun () -> + !selected = Some graph_id); + wait "selection closes picker" (fun () -> + Option.is_none (find "accessibility-identifier" "graph-picker-list")); + press "text" "Switch graph"; + wait "selected graph opens picker" (fun () -> + Option.is_some (find "accessibility-identifier" "graph-picker-list"))) + [ second_graph; third ]; + if replacement + then ( + press "accessibility-identifier" ("graph-picker:" ^ W.Uuid.to_string fourth); + press + "accessibility-identifier" + ("graph-resident-replace:" ^ W.Uuid.to_string third)) + else + press + "accessibility-identifier" + ("graph-resident-remove:" ^ W.Uuid.to_string third); + let target = if replacement then fourth else Test_support.graph_uuid in + wait + "real current close settles and selects retained/replacement graph" + (fun () -> !selected = Some target && not (List.mem third !residents)); + press "text" "Switch graph"; + wait "retained graph responds to picker after retirement" (fun () -> + Option.is_some (find "accessibility-identifier" "graph-picker-list")); + press "accessibility-identifier" "graph-picker-close"; + press "accessibility-label" "Journals"; + press "accessibility-identifier" "journal-capture-open"; + wait "current graph opens a real Capture editor after retirement" (fun () -> + Option.is_some (find "accessibility-identifier" "field.composer")))) ;; (* Actual Service/DB/Sync ownership is not expressible by injecting Core completions. Exercise the real retained listener, close, consume and reopen. *) let test_real_host_close_keeps_other_resident_and_reopens () = - with_client(fun client-> - let events=Queue.create() and state=ref(C.initial ~client_id) in - let runner=R.create ~client ~managed_origin:"https://api.logseq.io" ~user_id:"user-1" - ~post:(fun event->Queue.add event events) ~publish:(fun _->()) |> Result.get_ok in - let apply transition=state:=transition.C.next;List.iter(R.submit runner)transition.instructions in - let host_responses=ref 0 in - Mailbox.on_event client(fun event-> - let delivery=R.on_event runner event in + with_client (fun client -> + let events = Queue.create () + and state = ref (C.initial ~client_id) in + let runner = + R.create + ~client + ~managed_origin:"https://api.logseq.io" + ~user_id:"user-1" + ~post:(fun event -> Queue.add event events) + ~publish:(fun _ -> ()) + |> Result.get_ok + in + let apply transition = + state := transition.C.next; + List.iter (R.submit runner) transition.instructions + in + let host_responses = ref 0 in + Mailbox.on_event client (fun event -> + let delivery = R.on_event runner event in match event with - |Mailbox.Response{outcome=Completed(Service.Coordination_host_prepared _|Coordination_host_applied _);_}-> + | Mailbox.Response + { outcome = + Completed + (Service.Coordination_host_prepared _ | Coordination_host_applied _) + ; _ + } -> incr host_responses; - require(delivery.disposition=R.Consumed)"actual host receipt escaped its exclusive response owner" - - |_->()); - let pump()=Mailbox.Private.deliver client ~max_events:64; - let count=Queue.length events in for _=1 to count do apply(C.step !state(Queue.take events)) done in - let wait label predicate= - let deadline=Unix.gettimeofday()+.4. in - while not(predicate()) && Unix.gettimeofday()[]|Some projection-> - W.residents projection |> List.filter_map(fun resident->if resident.W.phase=W.Open then Some resident.session else None) in - wait "host initial actual DB did not open" (fun()->List.length(open_sessions())=1); - ignore(Mailbox.send client(Service.Client_command(Configure_resident_graphs[Test_support.graph_uuid;second_graph]))); - wait "host two actual residents did not open" (fun()->List.length(open_sessions())=2); - let first=List.find(fun session->W.graph_id session=Test_support.graph_uuid)(open_sessions()) - and second=List.find(fun session->W.graph_id session=second_graph)(open_sessions()) in - let budget=W.read_budget first ~maximum_items:1 ~maximum_bytes:65536 |> Result.get_ok in - let observed,registration=C.request !state first(W.Observe(budget,W.Graph_info)) |> Result.get_ok in - apply registration;wait "actual host listener not registered"(fun()->Option.is_some(C.reply(C.view !state)observed)); - let observation=match C.reply(C.view !state)observed with Some(W.Answered(Ok value))->value.W.observation - |_->failwith "actual host observation did not sample" in - let id,prepare=C.prepare_host_change !state(W.Graph_will_close(W.graph_id first)) |> Result.get_ok in + require + (delivery.disposition = R.Consumed) + "actual host receipt escaped its exclusive response owner" + | _ -> ()); + let pump () = + Mailbox.Private.deliver client ~max_events:64; + let count = Queue.length events in + for _ = 1 to count do + apply (C.step !state (Queue.take events)) + done + in + let wait label predicate = + let deadline = Unix.gettimeofday () +. 4. in + while (not (predicate ())) && Unix.gettimeofday () < deadline do + pump (); + Thread.delay 0.001 + done; + require (predicate ()) label + in + ignore + (Mailbox.send + client + (Service.Client_command (Restore_local_account { user_id = "user-1" }))); + let open_sessions () = + match C.projection (C.view !state) with + | None -> [] + | Some projection -> + W.residents projection + |> List.filter_map (fun resident -> + if resident.W.phase = W.Open then Some resident.session else None) + in + wait "host initial actual DB did not open" (fun () -> + List.length (open_sessions ()) = 1); + ignore + (Mailbox.send + client + (Service.Client_command + (Configure_resident_graphs [ Test_support.graph_uuid; second_graph ]))); + wait "host two actual residents did not open" (fun () -> + List.length (open_sessions ()) = 2); + let first = + List.find + (fun session -> W.graph_id session = Test_support.graph_uuid) + (open_sessions ()) + and second = + List.find (fun session -> W.graph_id session = second_graph) (open_sessions ()) + in + let budget = + W.read_budget first ~maximum_items:1 ~maximum_bytes:65536 |> Result.get_ok + in + let observed, registration = + C.request !state first (W.Observe (budget, W.Graph_info)) |> Result.get_ok + in + apply registration; + wait "actual host listener not registered" (fun () -> + Option.is_some (C.reply (C.view !state) observed)); + let observation = + match C.reply (C.view !state) observed with + | Some (W.Answered (Ok value)) -> value.W.observation + | _ -> failwith "actual host observation did not sample" + in + let id, prepare = + C.prepare_host_change !state (W.Graph_will_close (W.graph_id first)) + |> Result.get_ok + in apply prepare; - let receipt=ref None in - wait "actual host prepare did not settle"(fun()->match C.handoff(C.view !state)id with - |Some(W.Safe_to_proceed value)->receipt:=Some value;true - |Some(W.Handoff_failed(_,(W.Service_failed message|W.Protocol_mismatch message)))->failwith message - |Some(W.Handoff_failed _)->failwith "actual host prepare failed" - |_->false); - require(C.observed(C.view !state)observation=None)"host Safe published before its listener retirement ACK"; - let query session= - let budget=W.read_budget session ~maximum_items:1 ~maximum_bytes:65536 |> Result.get_ok in - let ticket,transition=C.request !state session(W.Query(budget,W.Graph_info)) |> Result.get_ok in - apply transition;wait "unrelated resident read did not finish"(fun()->Option.is_some(C.reply(C.view !state)ticket)); - let info=match C.reply(C.view !state)ticket with Some(W.Answered(Ok info))->info|_->failwith "actual host read failed" in - apply(C.step !state(C.Forget ticket));info in - require((query second).graph_uuid=second_graph)"closing first resident interrupted the other actual DB"; - let receipt=Option.get !receipt in - apply(C.authorize_host_change !state receipt |> Result.get_ok); - wait "actual host consume did not acknowledge"(fun()->match C.handoff(C.view !state)id with - |Some(W.Host_change_applied _)->true - |Some(W.Handoff_failed(_,(W.Service_failed message|W.Protocol_mismatch message)))->failwith message - |Some(W.Handoff_failed _)->failwith "actual host consume failed"|_->false); - require(C.validate_session !state first=Error W.Revoked_session)"actual consume revived its old execution"; - ignore(Mailbox.send client(Service.Client_command(Configure_resident_graphs[Test_support.graph_uuid;second_graph]))); - wait "actual graph could not reopen after consumed close"(fun()->List.length(open_sessions())=2); - let reopened=List.find(fun session->W.graph_id session=Test_support.graph_uuid)(open_sessions()) in - require(not(W.equal_session first reopened))"actual reopen reused old execution generation"; - require((query reopened).graph_uuid=Test_support.graph_uuid)"new execution remained fenced after close"; - require(Result.is_error(C.authorize_host_change !state receipt))"old safe permission consumed after real reopen"; - require(!host_responses=2)"host prepare/consume did not own exactly two real responses"; - apply(C.step !state C.Shutdown_requested); - wait "real host coordination did not drain"(fun()->C.shutdown_state(C.view !state)=C.Stopped)) + let receipt = ref None in + wait "actual host prepare did not settle" (fun () -> + match C.handoff (C.view !state) id with + | Some (W.Safe_to_proceed value) -> + receipt := Some value; + true + | Some + (W.Handoff_failed (_, (W.Service_failed message | W.Protocol_mismatch message))) + -> failwith message + | Some (W.Handoff_failed _) -> failwith "actual host prepare failed" + | _ -> false); + require + (C.observed (C.view !state) observation = None) + "host Safe published before its listener retirement ACK"; + let query session = + let budget = + W.read_budget session ~maximum_items:1 ~maximum_bytes:65536 |> Result.get_ok + in + let ticket, transition = + C.request !state session (W.Query (budget, W.Graph_info)) |> Result.get_ok + in + apply transition; + wait "unrelated resident read did not finish" (fun () -> + Option.is_some (C.reply (C.view !state) ticket)); + let info = + match C.reply (C.view !state) ticket with + | Some (W.Answered (Ok info)) -> info + | _ -> failwith "actual host read failed" + in + apply (C.step !state (C.Forget ticket)); + info + in + require + ((query second).graph_uuid = second_graph) + "closing first resident interrupted the other actual DB"; + let receipt = Option.get !receipt in + apply (C.authorize_host_change !state receipt |> Result.get_ok); + wait "actual host consume did not acknowledge" (fun () -> + match C.handoff (C.view !state) id with + | Some (W.Host_change_applied _) -> true + | Some + (W.Handoff_failed (_, (W.Service_failed message | W.Protocol_mismatch message))) + -> failwith message + | Some (W.Handoff_failed _) -> failwith "actual host consume failed" + | _ -> false); + require + (C.validate_session !state first = Error W.Revoked_session) + "actual consume revived its old execution"; + ignore + (Mailbox.send + client + (Service.Client_command + (Configure_resident_graphs [ Test_support.graph_uuid; second_graph ]))); + wait "actual graph could not reopen after consumed close" (fun () -> + List.length (open_sessions ()) = 2); + let reopened = + List.find + (fun session -> W.graph_id session = Test_support.graph_uuid) + (open_sessions ()) + in + require + (not (W.equal_session first reopened)) + "actual reopen reused old execution generation"; + require + ((query reopened).graph_uuid = Test_support.graph_uuid) + "new execution remained fenced after close"; + require + (Result.is_error (C.authorize_host_change !state receipt)) + "old safe permission consumed after real reopen"; + require + (!host_responses = 2) + "host prepare/consume did not own exactly two real responses"; + apply (C.step !state C.Shutdown_requested); + wait "real host coordination did not drain" (fun () -> + C.shutdown_state (C.view !state) = C.Stopped)) ;; let test_client_to_typed_read () = @@ -792,28 +1385,60 @@ let test_client_to_typed_read () = wait (fun () -> Option.is_some (C.reply (C.view !state) ticket)); ticket, C.reply (C.view !state) ticket |> Option.get in - let request operation=request_in session operation in - let asset_consumer=W.asset_consumer session ~name:"runtime-synthetic-demand" |> Result.get_ok in - let asset=Logseq_db_types.Asset_descriptor.create - ~uuid:(uuid "9c000000-0000-4000-8000-000000000881") ~source:(Managed None) - ~current_checksum:None ~size:None ~dimensions:None |> Result.get_ok in - let before_demand=C.view !state in - let demand_ticket,demand_reply=request(W.Replace_asset_demand(asset_consumer,W.Foreground,[asset])) in - let asset_demand=match demand_reply with W.Answered(Ok demand)->demand - |_->failwith "actual Service rejected typed scoped asset demand" in - wait(fun()->List.exists(fun notice->match W.asset_notice_data notice with - | Service.Asset_demand_accepted name->name="runtime-synthetic-demand" - | _->false)(C.asset_notices(C.view !state)asset_demand)); - require(List.for_all(fun notice->Option.exists(W.equal_asset_consumer asset_consumer)(W.asset_notice_consumer notice)) - (C.asset_notices(C.view !state)asset_demand)) "actual asset push crossed consumer identity"; - require(Rrbvec.exists(fun notice->match W.asset_notice_data notice with - |Service.Asset_demand_accepted name->name="runtime-synthetic-demand"|_->false) - (C.asset_updates ~previous:(Some before_demand)(C.view !state))) + let request operation = request_in session operation in + let asset_consumer = + W.asset_consumer session ~name:"runtime-synthetic-demand" |> Result.get_ok + in + let asset = + Logseq_db_types.Asset_descriptor.create + ~uuid:(uuid "9c000000-0000-4000-8000-000000000881") + ~source:(Managed None) + ~current_checksum:None + ~size:None + ~dimensions:None + |> Result.get_ok + in + let before_demand = C.view !state in + let demand_ticket, demand_reply = + request (W.Replace_asset_demand (asset_consumer, W.Foreground, [ asset ])) + in + let asset_demand = + match demand_reply with + | W.Answered (Ok demand) -> demand + | _ -> failwith "actual Service rejected typed scoped asset demand" + in + wait (fun () -> + List.exists + (fun notice -> + match W.asset_notice_data notice with + | Service.Asset_demand_accepted name -> name = "runtime-synthetic-demand" + | _ -> false) + (C.asset_notices (C.view !state) asset_demand)); + require + (List.for_all + (fun notice -> + Option.exists + (W.equal_asset_consumer asset_consumer) + (W.asset_notice_consumer notice)) + (C.asset_notices (C.view !state) asset_demand)) + "actual asset push crossed consumer identity"; + require + (Rrbvec.exists + (fun notice -> + match W.asset_notice_data notice with + | Service.Asset_demand_accepted name -> name = "runtime-synthetic-demand" + | _ -> false) + (C.asset_updates ~previous:(Some before_demand) (C.view !state))) "actual Service acceptance did not reach the typed asset outlet"; - let release_demand,released_demand=request(W.Release_asset_demand asset_demand) in - require(released_demand=W.Answered(Ok())) "actual scoped asset demand retirement failed"; - require(C.asset_notices(C.view !state)asset_demand=[]) "retired actual asset demand retained data authority"; - apply(C.step !state(C.Forget demand_ticket));apply(C.step !state(C.Forget release_demand)); + let release_demand, released_demand = request (W.Release_asset_demand asset_demand) in + require + (released_demand = W.Answered (Ok ())) + "actual scoped asset demand retirement failed"; + require + (C.asset_notices (C.view !state) asset_demand = []) + "retired actual asset demand retained data authority"; + apply (C.step !state (C.Forget demand_ticket)); + apply (C.step !state (C.Forget release_demand)); let info_ticket, info = request (W.Query (budget, W.Graph_info)) in (match info with | W.Answered (Ok info) -> @@ -823,51 +1448,83 @@ let test_client_to_typed_read () = | _ -> failwith "graph info failed"); (* Actual client -> interpreter -> pure registration -> sampled read and ACK. The source values below come from Service, never a synthetic projection. *) - let observed_ticket,observed_reply=request(W.Observe(budget,W.Graph_info)) in - let observed=match observed_reply with - | W.Answered(Ok value)->value - | _->failwith "real App observation registration failed" in - require(C.observation_handle(C.view !state) observed_ticket=Some observed.observation) + let observed_ticket, observed_reply = request (W.Observe (budget, W.Graph_info)) in + let observed = + match observed_reply with + | W.Answered (Ok value) -> value + | _ -> failwith "real App observation registration failed" + in + require + (C.observation_handle (C.view !state) observed_ticket = Some observed.observation) "interpreter did not forward actual registration metadata"; - require(C.observed(C.view !state) observed.observation=Some observed) + require + (C.observed (C.view !state) observed.observation = Some observed) "real observed DTO did not reach pure retained state"; - apply(C.step !state(C.Forget observed_ticket)); - require(C.observed(C.view !state) observed.observation=Some observed) + apply (C.step !state (C.Forget observed_ticket)); + require + (C.observed (C.view !state) observed.observation = Some observed) "host result disposal unregistered the live interest"; - let refresh,refreshed_reply=request(W.Refresh observed.observation) in - let refreshed=match refreshed_reply with - | W.Answered(Ok value)->value|_->failwith "real App Refresh failed" in - require(W.compare_observation_epoch refreshed.epoch observed.epoch>0) + let refresh, refreshed_reply = request (W.Refresh observed.observation) in + let refreshed = + match refreshed_reply with + | W.Answered (Ok value) -> value + | _ -> failwith "real App Refresh failed" + in + require + (W.compare_observation_epoch refreshed.epoch observed.epoch > 0) "real Refresh did not advance the backend epoch"; - apply(C.step !state(C.Forget refresh)); - let released,released_reply=request(W.Release observed.observation) in - require(released_reply=W.Answered(Ok())) "real Release did not receive a physical ACK"; - require(C.observed(C.view !state) observed.observation=None) "released interest remained visible"; - apply(C.step !state(C.Forget released)); - let tiny=W.read_budget session ~maximum_items:1 ~maximum_bytes:1 |> Result.get_ok in - let failed_ticket,failed_reply=request(W.Observe(tiny,W.Graph_info)) in - require(match failed_reply with W.Answered(Error _)->true|_->false) + apply (C.step !state (C.Forget refresh)); + let released, released_reply = request (W.Release observed.observation) in + require + (released_reply = W.Answered (Ok ())) + "real Release did not receive a physical ACK"; + require + (C.observed (C.view !state) observed.observation = None) + "released interest remained visible"; + apply (C.step !state (C.Forget released)); + let tiny = W.read_budget session ~maximum_items:1 ~maximum_bytes:1 |> Result.get_ok in + let failed_ticket, failed_reply = request (W.Observe (tiny, W.Graph_info)) in + require + (match failed_reply with + | W.Answered (Error _) -> true + | _ -> false) "tiny actual observation unexpectedly fit its byte budget"; - let failed_handle=match C.observation_handle(C.view !state) failed_ticket with - | Some handle->handle|_->failwith "actual read error erased registration metadata" in - let _,_=request(W.Release failed_handle) in - apply(C.step !state(C.Forget failed_ticket)); - let cancel_ticket,cancel_plan=C.request !state session(W.Observe(budget,W.Graph_info)) |> Result.get_ok in + let failed_handle = + match C.observation_handle (C.view !state) failed_ticket with + | Some handle -> handle + | _ -> failwith "actual read error erased registration metadata" + in + let _, _ = request (W.Release failed_handle) in + apply (C.step !state (C.Forget failed_ticket)); + let cancel_ticket, cancel_plan = + C.request !state session (W.Observe (budget, W.Graph_info)) |> Result.get_ok + in apply cancel_plan; (* Admission is queued by the real interpreter, but has not yet been reduced. *) - apply(C.step !state(C.Cancel cancel_ticket)); - wait(fun ()->C.reply(C.view !state) cancel_ticket<>None); - wait(fun ()->List.exists(function - | Mailbox.Response{outcome=Completed(Service.Coordination_observation_released value);_},R.Consumed -> - W.equal_request_id(W.request_id value.observation_id)(C.ticket_id cancel_ticket) - | _->false) !deliveries); - require(C.observation_handle(C.view !state) cancel_ticket=None) + apply (C.step !state (C.Cancel cancel_ticket)); + wait (fun () -> C.reply (C.view !state) cancel_ticket <> None); + wait (fun () -> + List.exists + (function + | ( Mailbox.Response + { outcome = Completed (Service.Coordination_observation_released value) + ; _ + } + , R.Consumed ) -> + W.equal_request_id + (W.request_id value.observation_id) + (C.ticket_id cancel_ticket) + | _ -> false) + !deliveries); + require + (C.observation_handle (C.view !state) cancel_ticket = None) "canceled registration adopted a late active handle"; - require(match C.reply(C.view !state) cancel_ticket with - | Some(W.Interrupted W.Cancelled)->true|_->false) + require + (match C.reply (C.view !state) cancel_ticket with + | Some (W.Interrupted W.Cancelled) -> true + | _ -> false) "canceled observation returned a successful sampled result"; - apply(C.step !state(C.Forget cancel_ticket)); - + apply (C.step !state (C.Forget cancel_ticket)); let _, admission = request (W.Query (budget, W.Admission)) in (match admission with | W.Answered (Ok value) -> @@ -879,10 +1536,17 @@ let test_client_to_typed_read () = require (definitions = W.Answered (Ok [ None ])) "missing status definition lost ordered slot"; - let business = Journal_graph_runtime.create ~validate_session:(fun session -> C.validate_session ~owner !state session) () in + let business = + Journal_graph_runtime.create + ~validate_session:(fun session -> C.validate_session ~owner !state session) + () + in let original = Journal_graph_runtime.start business in require - (Journal_graph_runtime.worker_read business (W.Uuid.of_string "98000000-0000-4000-8000-000000009999" |> Result.get_ok)=None) + (Journal_graph_runtime.worker_read + business + (W.Uuid.of_string "98000000-0000-4000-8000-000000009999" |> Result.get_ok) + = None) "an unplanned business request acquired a typed query"; (match Journal_graph_runtime.worker_read business original with | Some (Journal_graph_runtime.Read read) -> @@ -893,7 +1557,7 @@ let test_client_to_typed_read () = Journal_graph_runtime.receive_worker_read business ~request_id:original - (Journal_graph_runtime.Query_result(read.query,value)) + (Journal_graph_runtime.Query_result (read.query, value)) in require (List.exists @@ -903,7 +1567,8 @@ let test_client_to_typed_read () = output.responses) "typed graph info did not reach actual business projection" | _ -> failwith "typed business graph-info read failed") - | Some (Journal_graph_runtime.Acknowledge _ | Confirm_readback _) -> failwith "graph-info dispatch became a non-query" + | Some (Journal_graph_runtime.Acknowledge _ | Confirm_readback _) -> + failwith "graph-info dispatch became a non-query" | None -> failwith "business graph-info path was not migrated"); let _, page = request @@ -954,7 +1619,13 @@ let test_client_to_typed_read () = with | Error (W.Invalid_input _) -> () | _ -> failwith "foreign parent accepted continuation"); - let summary_budget = W.read_budget session ~maximum_items:Worker_pure_data.maximum_page_size ~maximum_bytes:65536 |> Result.get_ok in + let summary_budget = + W.read_budget + session + ~maximum_items:Worker_pure_data.maximum_page_size + ~maximum_bytes:65536 + |> Result.get_ok + in let _, summary = request (W.Query @@ -964,10 +1635,13 @@ let test_client_to_typed_read () = in (match summary with | W.Answered (Ok summary) -> - require (summary.next=None) "test projection cannot infer exact child count from Partial"; + require + (summary.next = None) + "test projection cannot infer exact child count from Partial"; require (Option.is_some summary.page) "atomic summary lost owning page"; (match summary.lookup, summary.page with - | Worker_pure_data.V2_present_block { value; revision }, Some (Worker_pure_data.V2_present_page { page; _ }) -> + | ( Worker_pure_data.V2_present_block { value; revision } + , Some (Worker_pure_data.V2_present_page { page; _ }) ) -> require (revision <> "" && value.block.title = "Authoritative block") "full DTO lost source/revision"; @@ -999,8 +1673,10 @@ let test_client_to_typed_read () = !clock) ; plan_read = (fun ~source_epoch:_ read query -> - let original=uuid (Printf.sprintf "9c000000-0000-4000-8000-%012Ld" !clock) in - planned := Some (read,query); + let original = + uuid (Printf.sprintf "9c000000-0000-4000-8000-%012Ld" !clock) + in + planned := Some (read, query); original) } in @@ -1036,7 +1712,9 @@ let test_client_to_typed_read () = summary_page (count + 1) | Confirmed value -> require ((not more) && count >= 1) "summary completed before all real pages"; - require (Rrbvec.length value.children = 2) "summary lost exact child cardinality"; + require + (Rrbvec.length value.children = 2) + "summary lost exact child cardinality"; List.iter (fun n -> require @@ -1075,7 +1753,7 @@ let test_client_to_typed_read () = List.find (fun request -> match Journal_graph_runtime.worker_read business request with - | Some(Journal_graph_runtime.Read{query=W.Task_statuses _;_}) -> true + | Some (Journal_graph_runtime.Read { query = W.Task_statuses _; _ }) -> true | _ -> false) output.requests in @@ -1089,10 +1767,11 @@ let test_client_to_typed_read () = let _, reply = request (W.Query (budget, read.query)) in (match reply with | W.Answered (Ok value) -> - let result = Journal_graph_runtime.Query_result(read.query,value) in + let result = Journal_graph_runtime.Query_result (read.query, value) in let more = match result with - | Journal_graph_runtime.Query_result(W.Task_statuses _,value) -> Option.is_some value.next + | Journal_graph_runtime.Query_result (W.Task_statuses _, value) -> + Option.is_some value.next | _ -> failwith "catalog result lost typed correspondence" in let output = @@ -1116,7 +1795,8 @@ let test_client_to_typed_read () = List.find (fun request -> match Journal_graph_runtime.worker_read business request with - | Some(Journal_graph_runtime.Read{query=W.Task_statuses _;_}) -> true + | Some (Journal_graph_runtime.Read { query = W.Task_statuses _; _ }) + -> true | _ -> false) output.requests in @@ -1162,38 +1842,61 @@ let test_client_to_typed_read () = let record_check read_limit receipts = require (read_limit = 1) "Timeline_check must use its single aggregate row budget"; let count = Rrbvec.length receipts in - require (count > 0 && count <= W.Timeline.maximum_receipts) + require + (count > 0 && count <= W.Timeline.maximum_receipts) "Timeline_check lost its independent retained-receipt bound"; receipt_checks := count :: !receipt_checks in let checked_results receipts value = - require (Rrbvec.length value.W.Timeline.results = Rrbvec.length receipts) + require + (Rrbvec.length value.W.Timeline.results = Rrbvec.length receipts) "real receipt validation lost scope correspondence"; - require (Rrbvec.for_all Result.is_ok value.results) + require + (Rrbvec.for_all Result.is_ok value.results) "unchanged synthetic Timeline scopes failed same-snapshot validation" in - let rec settle_checks (output : Journal_graph_runtime.output) : Journal_graph_runtime.output = - List.fold_left (fun (kept:Journal_graph_runtime.output) original -> - match Journal_graph_runtime.worker_read business original with - | Some (Journal_graph_runtime.Read {query=W.Timeline_check receipts;maximum_items}) -> - record_check maximum_items receipts; - require (not (List.exists (fun response -> match response.Journal_graph_runtime.payload with - | Feed_loaded {complete=true;_} -> true | _ -> false) output.responses)) - "unverified Timeline assembly was published before its receipt check"; - let budget = W.read_budget session ~maximum_items ~maximum_bytes:65536 |> Result.get_ok in - let ticket, reply = request (W.Query (budget,W.Timeline_check receipts)) in - (match reply with - | W.Answered (Ok checked) -> - checked_results receipts checked; - let next = Journal_graph_runtime.receive_worker_read - ?origin:(C.query_origin (C.view !state) ticket) business ~request_id:original - (Journal_graph_runtime.Query_result(W.Timeline_check receipts,checked)) - |> settle_checks in - {Journal_graph_runtime.requests=kept.requests @ next.requests; - responses=kept.responses @ next.responses} - | _ -> failwith "real typed Timeline receipt check failed") - | _ -> {kept with requests=kept.requests @ [original]}) - {Journal_graph_runtime.requests=[];responses=output.responses} output.requests + let rec settle_checks (output : Journal_graph_runtime.output) + : Journal_graph_runtime.output + = + List.fold_left + (fun (kept : Journal_graph_runtime.output) original -> + match Journal_graph_runtime.worker_read business original with + | Some + (Journal_graph_runtime.Read + { query = W.Timeline_check receipts; maximum_items }) -> + record_check maximum_items receipts; + require + (not + (List.exists + (fun response -> + match response.Journal_graph_runtime.payload with + | Feed_loaded { complete = true; _ } -> true + | _ -> false) + output.responses)) + "unverified Timeline assembly was published before its receipt check"; + let budget = + W.read_budget session ~maximum_items ~maximum_bytes:65536 |> Result.get_ok + in + let ticket, reply = request (W.Query (budget, W.Timeline_check receipts)) in + (match reply with + | W.Answered (Ok checked) -> + checked_results receipts checked; + let next = + Journal_graph_runtime.receive_worker_read + ?origin:(C.query_origin (C.view !state) ticket) + business + ~request_id:original + (Journal_graph_runtime.Query_result + (W.Timeline_check receipts, checked)) + |> settle_checks + in + { Journal_graph_runtime.requests = kept.requests @ next.requests + ; responses = kept.responses @ next.responses + } + | _ -> failwith "real typed Timeline receipt check failed") + | _ -> { kept with requests = kept.requests @ [ original ] }) + { Journal_graph_runtime.requests = []; responses = output.responses } + output.requests in let dates = Journal_graph_runtime.submit @@ -1208,7 +1911,10 @@ let test_client_to_typed_read () = List.find (fun r -> match Journal_graph_runtime.worker_read business r with - | Some(Journal_graph_runtime.Read{query=W.Timeline_read {query=W.Timeline.Dates _;_};_}) -> true + | Some + (Journal_graph_runtime.Read + { query = W.Timeline_read { query = W.Timeline.Dates _; _ }; _ }) -> + true | _ -> false) o.requests in @@ -1218,11 +1924,18 @@ let test_client_to_typed_read () = let ticket, reply = request (W.Query (budget, read.query)) in (match reply with | W.Answered (Ok value) -> - let result = Journal_graph_runtime.Query_result(read.query,value) in - let (items, next) : Worker_pure_data.tree_member list * Journal_graph_projection.root_cursor option = + let result = Journal_graph_runtime.Query_result (read.query, value) in + let (items, next) + : Worker_pure_data.tree_member list + * Journal_graph_projection.root_cursor option + = match result with - | Journal_graph_runtime.Query_result(W.Timeline_read {query=W.Timeline.Roots _;_},Ok packet) -> - Rrbvec.to_list packet.items, (match packet.next with W.Timeline.End -> None | More cursor -> Some cursor) + | Journal_graph_runtime.Query_result + (W.Timeline_read { query = W.Timeline.Roots _; _ }, Ok packet) -> + ( Rrbvec.to_list packet.items + , (match packet.next with + | W.Timeline.End -> None + | More cursor -> Some cursor) ) | _ -> failwith "root lost typed result" in ( items @@ -1231,7 +1944,8 @@ let test_client_to_typed_read () = ?origin:(C.query_origin (C.view !state) ticket) business ~request_id:original - result |> settle_checks ) + result + |> settle_checks ) | _ -> failwith "actual root query failed") | _ -> failwith "production root dispatch bypassed coordinator" in @@ -1244,11 +1958,15 @@ let test_client_to_typed_read () = let ticket, reply = request (W.Query (budget, read.query)) in (match reply with | W.Answered (Ok value) -> - let result = Journal_graph_runtime.Query_result(read.query,value) in + let result = Journal_graph_runtime.Query_result (read.query, value) in let (items, more) : Worker_pure_data.journal_item list * bool = match result with - | Journal_graph_runtime.Query_result(W.Timeline_read {query=W.Timeline.Dates _;_},Ok packet) -> - Rrbvec.to_list packet.items, (match packet.next with W.Timeline.End -> false | More _ -> true) + | Journal_graph_runtime.Query_result + (W.Timeline_read { query = W.Timeline.Dates _; _ }, Ok packet) -> + ( Rrbvec.to_list packet.items + , (match packet.next with + | W.Timeline.End -> false + | More _ -> true) ) | _ -> failwith "date result lost typed family" in require (List.length items = 1) "real date page lost item bound"; @@ -1278,7 +1996,10 @@ let test_client_to_typed_read () = List.find (fun r -> match Journal_graph_runtime.worker_read business r with - | Some(Journal_graph_runtime.Read{query=W.Timeline_read {query=W.Timeline.Roots _;_};_}) -> true + | Some + (Journal_graph_runtime.Read + { query = W.Timeline_read { query = W.Timeline.Roots _; _ }; _ }) + -> true | _ -> false) output.requests in @@ -1294,7 +2015,11 @@ let test_client_to_typed_read () = List.find (fun r -> match Journal_graph_runtime.worker_read business r with - | Some(Journal_graph_runtime.Read{query=W.Timeline_read {query=W.Timeline.Dates _;_};_}) -> true + | Some + (Journal_graph_runtime.Read + { query = W.Timeline_read { query = W.Timeline.Dates _; _ } + ; _ + }) -> true | _ -> false) output.requests in @@ -1318,7 +2043,11 @@ let test_client_to_typed_read () = List.find (fun r -> match Journal_graph_runtime.worker_read business r with - | Some(Journal_graph_runtime.Read{query=W.Timeline_read {query=W.Timeline.Preview_children _;_};_}) -> true + | Some + (Journal_graph_runtime.Read + { query = W.Timeline_read { query = W.Timeline.Preview_children _; _ } + ; _ + }) -> true | _ -> false) output.Journal_graph_runtime.requests in @@ -1329,11 +2058,15 @@ let test_client_to_typed_read () = let ticket, reply = request (W.Query (budget, read.query)) in (match reply with | W.Answered (Ok value) -> - let result = Journal_graph_runtime.Query_result(read.query,value) in + let result = Journal_graph_runtime.Query_result (read.query, value) in let more = match result with - | Journal_graph_runtime.Query_result(W.Timeline_read {query=W.Timeline.Preview_children _;_},Ok packet) -> - (match packet.next with W.Timeline.End -> false | More _ -> true) + | Journal_graph_runtime.Query_result + (W.Timeline_read { query = W.Timeline.Preview_children _; _ }, Ok packet) + -> + (match packet.next with + | W.Timeline.End -> false + | More _ -> true) | _ -> failwith "preview lost typed child result" in let output = @@ -1341,7 +2074,8 @@ let test_client_to_typed_read () = ?origin:(C.query_origin (C.view !state) ticket) business ~request_id:original - result |> settle_checks + result + |> settle_checks in if more then preview_page (page + 1) (child_request output) @@ -1356,7 +2090,8 @@ let test_client_to_typed_read () = (fun (day : Journal_graph_projection.day_feed) -> List.find_opt (fun (entry : Journal_graph_projection.timeline_entry) -> - Journal_model.id entry.block = W.Uuid.to_string preview_root) + Journal_model.id entry.block + = W.Uuid.to_string preview_root) day.entries) feed.days | _ -> None) @@ -1374,10 +2109,13 @@ let test_client_to_typed_read () = let queue = Queue.create () and responses = ref [] and root_reads = ref [] in - let remaining = ref (match demand with - | Journal_graph_request.Load_feed {root_limit;_} -> root_limit - | Load_day_blocks {limit;_} -> limit - | _ -> failwith "Expected a bounded Timeline root demand") in + let remaining = + ref + (match demand with + | Journal_graph_request.Load_feed { root_limit; _ } -> root_limit + | Load_day_blocks { limit; _ } -> limit + | _ -> failwith "Expected a bounded Timeline root demand") + in let add output = responses := List.rev_append output.Journal_graph_runtime.responses !responses; List.iter (fun request -> Queue.add request queue) output.requests @@ -1389,8 +2127,13 @@ let test_client_to_typed_read () = require (!turns < 1000) "actual root quota read did not converge"; let original = Queue.take queue in (match Journal_graph_runtime.worker_read business original with - | Some(Journal_graph_runtime.Read{query=W.Timeline_read {query=W.Timeline.Roots _;_};maximum_items=limit}) -> - require (limit = min !remaining Worker_pure_data.maximum_page_size) + | Some + (Journal_graph_runtime.Read + { query = W.Timeline_read { query = W.Timeline.Roots _; _ } + ; maximum_items = limit + }) -> + require + (limit = min !remaining Worker_pure_data.maximum_page_size) "Timeline root admission changed the exact remaining visible quota budget"; root_reads := limit :: !root_reads | _ -> ()); @@ -1404,17 +2147,24 @@ let test_client_to_typed_read () = (match reply with | W.Answered (Ok value) -> (match read.query with - | W.Timeline_check receipts -> record_check read.maximum_items receipts; checked_results receipts value - | W.Timeline_read {query=W.Timeline.Roots _;_} -> + | W.Timeline_check receipts -> + record_check read.maximum_items receipts; + checked_results receipts value + | W.Timeline_read { query = W.Timeline.Roots _; _ } -> (match value with | Ok packet -> - require (Rrbvec.length packet.items <= read.maximum_items) + require + (Rrbvec.length packet.items <= read.maximum_items) "real Timeline root packet exceeded its unchanged admitted budget"; (* These declared fixtures contain only ordinary roots and blank roots; the latter consume no visible quota. *) - remaining := !remaining - Rrbvec.fold_left - (fun count (member:Worker_pure_data.tree_member) -> - count + if member.value.block.title = "" then 0 else 1) 0 packet.items + remaining + := !remaining + - Rrbvec.fold_left + (fun count (member : Worker_pure_data.tree_member) -> + count + if member.value.block.title = "" then 0 else 1) + 0 + packet.items | Error error -> failwith (W.Timeline.error_message error)) | _ -> ()); add @@ -1422,7 +2172,7 @@ let test_client_to_typed_read () = ?origin:(C.query_origin (C.view !state) ticket) business ~request_id:original - (Journal_graph_runtime.Query_result(read.query,value))) + (Journal_graph_runtime.Query_result (read.query, value))) | _ -> failwith "actual root quota typed query failed") | _ -> failwith "actual root quota path bypassed coordinator" done; @@ -1446,9 +2196,13 @@ let test_client_to_typed_read () = ; request_generation = 330L }) in - require (reads = [ 30; 20 ]) "Timeline root packets lost the remaining visible quota budget"; + require + (reads = [ 30; 20 ]) + "Timeline root packets lost the remaining visible quota budget"; let day = List.hd first.days in - require (List.length day.entries = 30) "actual first Timeline batch exceeded visible quota"; + require + (List.length day.entries = 30) + "actual first Timeline batch exceeded visible quota"; (match (Option.get day.continuation).protocol_cursor with | Some (Journal_graph_projection.Root_tree_cursor _) -> () | _ -> failwith "actual root cursor lost its sealed Timeline continuation"); @@ -1461,7 +2215,10 @@ let test_client_to_typed_read () = ; request_generation = 331L }) in - require (reads = [30]) "sealed Timeline continuation did not read its final ten roots at the requested budget"; + require + (reads = [ 30 ]) + "sealed Timeline continuation did not read its final ten roots at the requested \ + budget"; let entries = List.concat_map (fun (day : Journal_graph_projection.day_feed) -> day.entries) @@ -1478,7 +2235,8 @@ let test_client_to_typed_read () = require (List.length (List.sort_uniq String.compare all) = 40) "actual Timeline continuation lost or repeated visible roots"; - require (all = List.init 40 (fun index -> W.Uuid.to_string (quota_root (index+21)))) + require + (all = List.init 40 (fun index -> W.Uuid.to_string (quota_root (index + 21)))) "actual Timeline batches lost, reordered or replaced seeded visible roots"; let first, reads = drive_feed @@ -1491,14 +2249,20 @@ let test_client_to_typed_read () = }) in require - (reads <> [] && List.hd reads = 30 && List.for_all (fun limit -> limit > 0 && limit <= 30) reads) + (reads <> [] + && List.hd reads = 30 + && List.for_all (fun limit -> limit > 0 && limit <= 30) reads) "byte-budget fixture lost its bounded remaining root demand"; let day = List.hd first.days in require (List.length day.entries = 30 && day.continuation <> None) "byte-budget first page lost continuation"; - let first_byte_ids = List.map (fun (entry:Journal_graph_projection.timeline_entry) -> - Journal_model.id entry.block) day.entries in + let first_byte_ids = + List.map + (fun (entry : Journal_graph_projection.timeline_entry) -> + Journal_model.id entry.block) + day.entries + in let second, reads = drive_feed ~maximum_bytes:16384 @@ -1510,9 +2274,10 @@ let test_client_to_typed_read () = }) in require - (List.length reads > 1 && List.hd reads = 30 && - List.for_all (fun limit -> limit > 0 && limit <= 30) reads && - List.sort_uniq (fun a b -> Int.compare b a) reads = reads) + (List.length reads > 1 + && List.hd reads = 30 + && List.for_all (fun limit -> limit > 0 && limit <= 30) reads + && List.sort_uniq (fun a b -> Int.compare b a) reads = reads) "byte-fitting Timeline packets lost stable, decreasing remaining-demand budgets"; let day = List.hd second.days in require @@ -1524,11 +2289,18 @@ let test_client_to_typed_read () = String.length (Journal_model.source entry.block) >= 3001) day.entries) "byte fitting truncated canonical DTO source text"; - let all_byte_ids = first_byte_ids @ List.map (fun (entry:Journal_graph_projection.timeline_entry) -> - Journal_model.id entry.block) day.entries in - require (all_byte_ids = List.init 60 (fun index -> W.Uuid.to_string (byte_root (index+1)))) + let all_byte_ids = + first_byte_ids + @ List.map + (fun (entry : Journal_graph_projection.timeline_entry) -> + Journal_model.id entry.block) + day.entries + in + require + (all_byte_ids = List.init 60 (fun index -> W.Uuid.to_string (byte_root (index + 1)))) "byte-fitting Timeline pagination lost, reordered or duplicated canonical roots"; - require (List.exists (fun count -> count >= 42) !receipt_checks) + require + (List.exists (fun count -> count >= 42) !receipt_checks) "real combined root/preview reads lost retained End-scope receipts"; let favorites_request : Journal_graph_request.favorites_request = { graph_generation = owner.graph_generation @@ -1545,7 +2317,7 @@ let test_client_to_typed_read () = List.find (fun r -> match Journal_graph_runtime.worker_read business r with - | Some(Journal_graph_runtime.Read{query=W.Favorites _;_}) -> true + | Some (Journal_graph_runtime.Read { query = W.Favorites _; _ }) -> true | _ -> false) planned.requests in @@ -1559,11 +2331,17 @@ let test_client_to_typed_read () = Journal_graph_runtime.receive_worker_read business ~request_id:original - (Journal_graph_runtime.Query_result(read.query,value)) + (Journal_graph_runtime.Query_result (read.query, value)) in - List.iter (fun r -> match Journal_graph_runtime.worker_read business r with - | Some(Journal_graph_runtime.Read{query=W.Child_members{parent;_};_}) when W.Uuid.equal parent favorite_page_id -> membership:=Some r - | _ -> ()) output.requests; + List.iter + (fun r -> + match Journal_graph_runtime.worker_read business r with + | Some + (Journal_graph_runtime.Read + { query = W.Child_members { parent; _ }; _ }) + when W.Uuid.equal parent favorite_page_id -> membership := Some r + | _ -> ()) + output.requests; List.find_map (fun response -> match response.Journal_graph_runtime.payload with @@ -1602,15 +2380,18 @@ let test_client_to_typed_read () = let _, reply = request_query (W.Query (budget, read.query)) in (match reply with | W.Answered (Ok value) -> - let result = Journal_graph_runtime.Query_result(read.query,value) in + let result = Journal_graph_runtime.Query_result (read.query, value) in let items, more = match result with - | Journal_graph_runtime.Query_result(W.Child_members _,value) -> + | Journal_graph_runtime.Query_result (W.Child_members _, value) -> value.items, Option.is_some value.next | _ -> failwith "Favorite membership lost typed result" in let seen = - seen @ List.map (fun (item : Worker_pure_data.child_member) -> item.value.block.uuid) items + seen + @ List.map + (fun (item : Worker_pure_data.child_member) -> item.value.block.uuid) + items in let output = Journal_graph_runtime.receive_worker_read @@ -1619,13 +2400,19 @@ let test_client_to_typed_read () = result in if more - then membership_page (page + 1) seen (List.find - (fun request -> - match Journal_graph_runtime.worker_read business request with - | Some (Journal_graph_runtime.Read { query = W.Child_members { parent; _ }; _ }) -> - W.Uuid.equal parent favorite_page_id - | _ -> false) - output.requests) + then + membership_page + (page + 1) + seen + (List.find + (fun request -> + match Journal_graph_runtime.worker_read business request with + | Some + (Journal_graph_runtime.Read + { query = W.Child_members { parent; _ }; _ }) -> + W.Uuid.equal parent favorite_page_id + | _ -> false) + output.requests) else ( require (page = 1 && List.length seen = 2) @@ -1659,14 +2446,20 @@ let test_client_to_typed_read () = List.find (fun r -> match Journal_graph_runtime.worker_read business r with - | Some(Journal_graph_runtime.Read{query=W.Outline_block _;_}) | Some(Journal_graph_runtime.Read{query=W.Page _;_}) | Some(Journal_graph_runtime.Read{query=W.Child_members _;_}) -> true + | Some (Journal_graph_runtime.Read { query = W.Outline_block _; _ }) + | Some (Journal_graph_runtime.Read { query = W.Page _; _ }) + | Some (Journal_graph_runtime.Read { query = W.Child_members _; _ }) -> + true | _ -> false) output.requests in (match Journal_graph_runtime.worker_read business original with | Some (Journal_graph_runtime.Read read) -> let budget = - W.read_budget session ~maximum_items:read.maximum_items ~maximum_bytes:65536 + W.read_budget + session + ~maximum_items:read.maximum_items + ~maximum_bytes:65536 |> Result.get_ok in let _, reply = request_query (W.Query (budget, read.query)) in @@ -1677,7 +2470,7 @@ let test_client_to_typed_read () = (Journal_graph_runtime.receive_worker_read business ~request_id:original - (Journal_graph_runtime.Query_result(read.query,value))) + (Journal_graph_runtime.Query_result (read.query, value))) | _ -> failwith "actual Detail typed query failed") | _ -> failwith "actual Detail query bypassed coordinator") in @@ -1719,7 +2512,8 @@ let test_client_to_typed_read () = [ 1; 2 ]; let revision = match summary with - | W.Answered (Ok { lookup = Worker_pure_data.V2_present_block { revision; _ }; _ }) -> revision + | W.Answered (Ok { lookup = Worker_pure_data.V2_present_block { revision; _ }; _ }) + -> revision | _ -> assert false in let intent = @@ -1740,56 +2534,132 @@ let test_client_to_typed_read () = | Error (W.Invalid_input message) -> failwith message | Error _ -> failwith "durable intent rejected" in - let wire = W.encode_durable_intent(W.Mutation_intent intent) in + let wire = W.encode_durable_intent (W.Mutation_intent intent) in let restored = W.restore_durable_intent session wire |> Result.get_ok in require (W.encode_durable_intent restored = wire) "complete mutation changed on restore"; - let observed_block_ticket,observed_block_reply=request(W.Observe(budget, - W.Outline_block{uuid=Test_support.authoritative_block_uuid;revision=None})) in - let observed_block=match observed_block_reply with - | W.Answered(Ok value)->value|_->failwith "actual block observation failed" in - let module Detail_observation=Journal_detail.Pure_observation in - let context:Detail_observation.context={entry_id="actual-service-detail"; - request_generation=350L;root=Test_support.authoritative_block_uuid;owner;maximum_children=128; - original=uuid "9c000000-0000-4000-8000-0000000000d0"} in - let presentation_model,_=Detail_observation.planned Detail_observation.empty context observed_block_ticket in - let presentation_model,initial_effects=Detail_observation.inspect presentation_model(C.view !state) in - require(Rrbvec.exists(function Detail_observation.Initial(_,sample)-> - (match sample.value with Worker_pure_data.V2_present_block{value;_}->value.block.title="Authoritative block"|_->false) - | _->false)initial_effects) "actual Service sample did not reach the pure Detail outlet"; - apply(C.step !state(C.Forget observed_block_ticket)); + let observed_block_ticket, observed_block_reply = + request + (W.Observe + ( budget + , W.Outline_block + { uuid = Test_support.authoritative_block_uuid; revision = None } )) + in + let observed_block = + match observed_block_reply with + | W.Answered (Ok value) -> value + | _ -> failwith "actual block observation failed" + in + let module Detail_observation = Journal_detail.Pure_observation in + let context : Detail_observation.context = + { entry_id = "actual-service-detail" + ; request_generation = 350L + ; root = Test_support.authoritative_block_uuid + ; owner + ; maximum_children = 128 + ; original = uuid "9c000000-0000-4000-8000-0000000000d0" + } + in + let presentation_model, _ = + Detail_observation.planned Detail_observation.empty context observed_block_ticket + in + let presentation_model, initial_effects = + Detail_observation.inspect presentation_model (C.view !state) + in + require + (Rrbvec.exists + (function + | Detail_observation.Initial (_, sample) -> + (match sample.value with + | Worker_pure_data.V2_present_block { value; _ } -> + value.block.title = "Authoritative block" + | _ -> false) + | _ -> false) + initial_effects) + "actual Service sample did not reach the pure Detail outlet"; + apply (C.step !state (C.Forget observed_block_ticket)); let mutation_ticket, mutation_reply = request (W.Mutate intent) in let receipt = match mutation_reply with | W.Answered (Ok (W.Submission_committed receipt)) -> receipt | _ -> failwith "real durable mutation did not return commit evidence" in - wait(fun ()->match C.observed(C.view !state) observed_block.observation with - | Some {value=Worker_pure_data.V2_present_block {value;_};_}->value.block.title="Durable coordinated edit" - | _->false); - let observed_after=Option.get(C.observed(C.view !state) observed_block.observation) in - require(W.compare_observation_epoch observed_after.epoch observed_block.epoch>0) + wait (fun () -> + match C.observed (C.view !state) observed_block.observation with + | Some { value = Worker_pure_data.V2_present_block { value; _ }; _ } -> + value.block.title = "Durable coordinated edit" + | _ -> false); + let observed_after = + Option.get (C.observed (C.view !state) observed_block.observation) + in + require + (W.compare_observation_epoch observed_after.epoch observed_block.epoch > 0) "actual write invalidation did not advance retained observation"; - let presentation_model,update_effects=Detail_observation.inspect presentation_model(C.view !state) in - let actual=Rrbvec.find_opt(function Detail_observation.Updated _->true|_->false)update_effects |> Option.get in - let sample=match actual with Detail_observation.Updated(_,sample)->sample|_->assert false in - let business_request=Journal_graph_runtime.plan_observed_block business ~root:context.root in - let applied=Journal_graph_runtime.receive_worker_read business ~request_id:business_request - ~origin:(W.observation_session sample.observation,W.observation_budget sample.observation) - (Journal_graph_runtime.Query_result(W.Outline_block{uuid=context.root;revision=None},sample.value)) in - require(List.exists(fun response->match response.Journal_graph_runtime.payload with - | Block_updated{block;_}->Journal_model.id block=W.Uuid.to_string context.root - && Journal_model.source block="Durable coordinated edit" - | _->false)applied.responses) "actual observed DTO did not enter the existing Detail projection"; - let presentation_model,cleanup=Detail_observation.retain presentation_model Rrbvec.empty in - require(Detail_observation.context_count presentation_model=0) "retired actual Detail kept presentation ownership"; - let handle=Rrbvec.find_opt(function Detail_observation.Release _->true|_->false)cleanup |> Option.get - |> function Detail_observation.Release handle->handle|_->assert false in - require(W.same_observation handle observed_block.observation<>None) "actual Detail cleanup changed registration"; - let release_ticket,released=request(W.Release handle) in - require(released=W.Answered(Ok())) "updated observation did not physically release"; - apply(C.step !state(C.Forget release_ticket)); + let presentation_model, update_effects = + Detail_observation.inspect presentation_model (C.view !state) + in + let actual = + Rrbvec.find_opt + (function + | Detail_observation.Updated _ -> true + | _ -> false) + update_effects + |> Option.get + in + let sample = + match actual with + | Detail_observation.Updated (_, sample) -> sample + | _ -> assert false + in + let business_request = + Journal_graph_runtime.plan_observed_block business ~root:context.root + in + let applied = + Journal_graph_runtime.receive_worker_read + business + ~request_id:business_request + ~origin: + ( W.observation_session sample.observation + , W.observation_budget sample.observation ) + (Journal_graph_runtime.Query_result + (W.Outline_block { uuid = context.root; revision = None }, sample.value)) + in + require + (List.exists + (fun response -> + match response.Journal_graph_runtime.payload with + | Block_updated { block; _ } -> + Journal_model.id block = W.Uuid.to_string context.root + && Journal_model.source block = "Durable coordinated edit" + | _ -> false) + applied.responses) + "actual observed DTO did not enter the existing Detail projection"; + let presentation_model, cleanup = + Detail_observation.retain presentation_model Rrbvec.empty + in + require + (Detail_observation.context_count presentation_model = 0) + "retired actual Detail kept presentation ownership"; + let handle = + Rrbvec.find_opt + (function + | Detail_observation.Release _ -> true + | _ -> false) + cleanup + |> Option.get + |> function + | Detail_observation.Release handle -> handle + | _ -> assert false + in + require + (W.same_observation handle observed_block.observation <> None) + "actual Detail cleanup changed registration"; + let release_ticket, released = request (W.Release handle) in + require + (released = W.Answered (Ok ())) + "updated observation did not physically release"; + apply (C.step !state (C.Forget release_ticket)); let original = W.receipt_revisions receipt in require (Rrbvec.to_list (W.receipt_targets receipt) @@ -1797,9 +2667,14 @@ let test_client_to_typed_read () = "durable receipt lost canonical target"; apply (C.step !state (C.Forget mutation_ticket)); require - (Option.is_some (C.intent_progress (C.view !state)(W.Mutation_intent intent))) + (Option.is_some (C.intent_progress (C.view !state) (W.Mutation_intent intent))) "forgetting ticket erased durable outcome"; - let _, again = request (match restored with W.Mutation_intent intent->W.Mutate intent|Append_intent intent->W.Append intent) in + let _, again = + request + (match restored with + | W.Mutation_intent intent -> W.Mutate intent + | Append_intent intent -> W.Append intent) + in (match again with | W.Answered (Ok (W.Submission_committed repeated)) -> require @@ -1894,7 +2769,7 @@ let test_client_to_typed_read () = ~some:(fun p -> W.equal_session (W.presentation_session p) session) (selected ())) "focus roundtrip revoked resident execution instead of presentation"; - (match C.intent_progress (C.view !state)(W.Mutation_intent intent) with + (match C.intent_progress (C.view !state) (W.Mutation_intent intent) with | Some (W.Committed_pending_presentation preserved) -> require (W.receipt_revisions preserved = original) @@ -1908,92 +2783,213 @@ let test_client_to_typed_read () = (match reconfirmed with | W.Answered (Ok _) -> () | _ -> failwith "new explicit Confirm could not use preserved receipt"); - let source_file=Filename.temp_file "journal-coordination-synthetic-import-" ".png" in - Out_channel.with_open_bin source_file(fun output->output_string output "file"); - let source:Logseq_db_types.Asset_import.t= - {operation=uuid "9c000000-0000-4000-8000-000000000885"; - asset=uuid "9c000000-0000-4000-8000-000000000886";target=Test_support.authoritative_block_uuid; - replace_reference=None;local_mutation=uuid "9c000000-0000-4000-8000-000000000887"; - metadata_mutation=uuid "9c000000-0000-4000-8000-000000000888"; - source_file;title="Synthetic typed import";file_type="png"} in - let import_bytes,original_import_asset=Fun.protect ~finally:(fun()->if Sys.file_exists source_file then Sys.remove source_file)(fun()-> - let intent=W.asset_import_intent session source |> Result.get_ok in - let imported_ticket,reply=request(W.Import intent) in - let imported=match reply with W.Answered(Ok(W.Imported imported))->imported - |W.Answered(Error(W.Service_failed message))->failwith("actual typed import failed: "^message) - |_->failwith "actual typed import did not return its durable source receipt" in - let ordinary_consumer=W.asset_consumer session ~name:"media:actual-imported-bytes" |> Result.get_ok in - let demand,demand_reply=request(W.Replace_asset_demand(ordinary_consumer,W.Foreground,[(W.import_data imported).asset])) in - let ordinary_handle=match demand_reply with W.Answered(Ok demand)->demand - |_->failwith "ordinary typed demand did not reach the actual Service" in - let active_handle=ref ordinary_handle in - let ordinary_file()=List.find_map W.ready_asset_file(C.asset_notices(C.view !state) !active_handle) in - wait(fun()->Option.is_some(ordinary_file())); - let ordinary_acquire,ordinary_reply=request(W.Acquire_asset_file(Option.get(ordinary_file()))) in - let ordinary_lease=match ordinary_reply with W.Answered(Ok(Some lease))->lease - |_->failwith "ordinary Ready capability did not acquire actual retained bytes" in - require(In_channel.with_open_bin(W.file_path ordinary_lease)In_channel.input_all="file") - "ordinary typed acquisition did not retain the imported local version"; - let module Descriptor=Logseq_db_types.Asset_descriptor in - let remote_version=Descriptor.version - ~checksum:(Logseq_sync_effect_runner.Asset_codec.checksum "synthetic published second version") - ~file_type:"png" |> Result.get_ok in - let changed=Descriptor.create ~uuid:source.asset ~source:(Managed(Some remote_version)) - ~current_checksum:None ~size:None ~dimensions:None |> Result.get_ok in - let replacement_demand,replacement_reply=request(W.Replace_asset_demand(ordinary_consumer,W.Foreground,[changed])) in - let replacement_handle=match replacement_reply with W.Answered(Ok demand)->demand - |_->failwith "same-UUID remote version did not replace its typed demand" in - active_handle:=replacement_handle; - wait(fun()->List.exists(fun notice->match W.asset_notice_data notice with - |Service.Asset_availability{descriptor;_}->descriptor=changed|_->false) - (C.asset_notices(C.view !state)replacement_handle)); - require(ordinary_file()=None) "new remote version incorrectly adopted the cached old Ready"; - require(In_channel.with_open_bin(W.file_path ordinary_lease)In_channel.input_all="file") - "replacing the transfer version released the renderer's independent old lease"; - let ordinary_release,ordinary_released=request(W.Release_file ordinary_lease) in - require(ordinary_released=W.Answered(Ok())) "ordinary display lease did not acknowledge its physical release"; - let demand_release,demand_released=request(W.Release_asset_demand replacement_handle) in - require(demand_released=W.Answered(Ok())) "ordinary demand did not settle its physical transfer owner"; - apply(C.step !state(C.Forget demand));apply(C.step !state(C.Forget replacement_demand)); - apply(C.step !state(C.Forget ordinary_release));apply(C.step !state(C.Forget demand_release)); - apply(C.step !state(C.Forget ordinary_acquire)); - let preview=W.import_preview imported |> Option.get in - require(Sys.file_exists(W.file_path preview)) "actual imported preview did not own staged bytes"; - require((W.import_data imported).operation=source.operation) "actual import changed its stable operation"; - Sys.remove source_file; - let acquire,reply=request(W.Acquire_imported_file imported) in - let retained=match reply with W.Answered(Ok(Some lease))->lease|_->failwith "actual imported source could not be re-retained after picker removal" in - require(W.file_lease_id retained<>W.file_lease_id preview) "two real acquisitions shared one lease identity"; - let release,reply=request(W.Release_file retained) in - require(reply=W.Answered(Ok())) "actual retained lease release did not ACK"; - let released_previews=ref [] in - let media=Journal_media_runtime.create ~read:(fun _->false) - ~release_file:(fun lease->released_previews:=lease::!released_previews;true) - ~release_consumer:(fun _->failwith "preview-only fixture emitted a display release") - ~demand:(fun ~graph_generation:_ ~name:_ _ _->failwith "preview-only fixture emitted a display demand") - ~acquire:(fun _ _->failwith "preview-only fixture acquired a display file") - ~retry_asset:(fun ~graph_generation:_ _->failwith "preview-only fixture retried a network asset") - ~changed:(fun _ _->())() in - Journal_media_runtime.reset media ~graph_generation:(Some(W.import_data imported).graph_generation); - Journal_media_runtime.imported media ~current:true imported; - Journal_media_runtime.retire_coordinated_previews media; - Journal_media_runtime.reset media ~graph_generation:None; - require(List.length !released_previews=1)"binding cleanup and later reset released the same preview twice"; - require(List.exists(fun lease->W.file_lease_id lease=W.file_lease_id preview)!released_previews) - "typed Media preview cleanup lost its original lease"; - let release_preview,reply=request(W.Release_file preview) in - require(reply=W.Answered(Ok())) "actual preview lease release did not ACK"; - apply(C.step !state(C.Forget imported_ticket));apply(C.step !state(C.Forget acquire)); - apply(C.step !state(C.Forget release));apply(C.step !state(C.Forget release_preview)); - W.encode_asset_import_intent intent,(W.import_data imported).asset) in - let source_page,source_reply=request(W.Recover_import_inputs{after=None;maximum_items=2;maximum_bytes=65536}) in - let source_rows=match source_reply with W.Answered(Ok page)->page.inputs|_->failwith "actual Service source recovery failed" in - require(Rrbvec.length source_rows=1)"actual source recovery lost its original input"; - let original_source=Rrbvec.nth source_rows 0 in - require(W.encode_asset_import_intent original_source.intent=import_bytes && original_source.prepared<>None && not original_source.acknowledged) + let source_file = + Filename.temp_file "journal-coordination-synthetic-import-" ".png" + in + Out_channel.with_open_bin source_file (fun output -> output_string output "file"); + let source : Logseq_db_types.Asset_import.t = + { operation = uuid "9c000000-0000-4000-8000-000000000885" + ; asset = uuid "9c000000-0000-4000-8000-000000000886" + ; target = Test_support.authoritative_block_uuid + ; replace_reference = None + ; local_mutation = uuid "9c000000-0000-4000-8000-000000000887" + ; metadata_mutation = uuid "9c000000-0000-4000-8000-000000000888" + ; source_file + ; title = "Synthetic typed import" + ; file_type = "png" + } + in + let import_bytes, original_import_asset = + Fun.protect + ~finally:(fun () -> if Sys.file_exists source_file then Sys.remove source_file) + (fun () -> + let intent = W.asset_import_intent session source |> Result.get_ok in + let imported_ticket, reply = request (W.Import intent) in + let imported = + match reply with + | W.Answered (Ok (W.Imported imported)) -> imported + | W.Answered (Error (W.Service_failed message)) -> + failwith ("actual typed import failed: " ^ message) + | _ -> + failwith "actual typed import did not return its durable source receipt" + in + let ordinary_consumer = + W.asset_consumer session ~name:"media:actual-imported-bytes" |> Result.get_ok + in + let demand, demand_reply = + request + (W.Replace_asset_demand + (ordinary_consumer, W.Foreground, [ (W.import_data imported).asset ])) + in + let ordinary_handle = + match demand_reply with + | W.Answered (Ok demand) -> demand + | _ -> failwith "ordinary typed demand did not reach the actual Service" + in + let active_handle = ref ordinary_handle in + let ordinary_file () = + List.find_map + W.ready_asset_file + (C.asset_notices (C.view !state) !active_handle) + in + wait (fun () -> Option.is_some (ordinary_file ())); + let ordinary_acquire, ordinary_reply = + request (W.Acquire_asset_file (Option.get (ordinary_file ()))) + in + let ordinary_lease = + match ordinary_reply with + | W.Answered (Ok (Some lease)) -> lease + | _ -> + failwith "ordinary Ready capability did not acquire actual retained bytes" + in + require + (In_channel.with_open_bin (W.file_path ordinary_lease) In_channel.input_all + = "file") + "ordinary typed acquisition did not retain the imported local version"; + let module Descriptor = Logseq_db_types.Asset_descriptor in + let remote_version = + Descriptor.version + ~checksum: + (Logseq_sync_native_crypto.Asset_codec.checksum + "synthetic published second version") + ~file_type:"png" + |> Result.get_ok + in + let changed = + Descriptor.create + ~uuid:source.asset + ~source:(Managed (Some remote_version)) + ~current_checksum:None + ~size:None + ~dimensions:None + |> Result.get_ok + in + let replacement_demand, replacement_reply = + request + (W.Replace_asset_demand (ordinary_consumer, W.Foreground, [ changed ])) + in + let replacement_handle = + match replacement_reply with + | W.Answered (Ok demand) -> demand + | _ -> failwith "same-UUID remote version did not replace its typed demand" + in + active_handle := replacement_handle; + wait (fun () -> + List.exists + (fun notice -> + match W.asset_notice_data notice with + | Service.Asset_availability { descriptor; _ } -> descriptor = changed + | _ -> false) + (C.asset_notices (C.view !state) replacement_handle)); + require + (ordinary_file () = None) + "new remote version incorrectly adopted the cached old Ready"; + require + (In_channel.with_open_bin (W.file_path ordinary_lease) In_channel.input_all + = "file") + "replacing the transfer version released the renderer's independent old \ + lease"; + let ordinary_release, ordinary_released = + request (W.Release_file ordinary_lease) + in + require + (ordinary_released = W.Answered (Ok ())) + "ordinary display lease did not acknowledge its physical release"; + let demand_release, demand_released = + request (W.Release_asset_demand replacement_handle) + in + require + (demand_released = W.Answered (Ok ())) + "ordinary demand did not settle its physical transfer owner"; + apply (C.step !state (C.Forget demand)); + apply (C.step !state (C.Forget replacement_demand)); + apply (C.step !state (C.Forget ordinary_release)); + apply (C.step !state (C.Forget demand_release)); + apply (C.step !state (C.Forget ordinary_acquire)); + let preview = W.import_preview imported |> Option.get in + require + (Sys.file_exists (W.file_path preview)) + "actual imported preview did not own staged bytes"; + require + ((W.import_data imported).operation = source.operation) + "actual import changed its stable operation"; + Sys.remove source_file; + let acquire, reply = request (W.Acquire_imported_file imported) in + let retained = + match reply with + | W.Answered (Ok (Some lease)) -> lease + | _ -> + failwith + "actual imported source could not be re-retained after picker removal" + in + require + (W.file_lease_id retained <> W.file_lease_id preview) + "two real acquisitions shared one lease identity"; + let release, reply = request (W.Release_file retained) in + require + (reply = W.Answered (Ok ())) + "actual retained lease release did not ACK"; + let released_previews = ref [] in + let media = + Journal_media_runtime.create + ~read:(fun _ -> false) + ~release_file:(fun lease -> + released_previews := lease :: !released_previews; + true) + ~release_consumer:(fun _ -> + failwith "preview-only fixture emitted a display release") + ~demand:(fun ~graph_generation:_ ~name:_ _ _ -> + failwith "preview-only fixture emitted a display demand") + ~acquire:(fun _ _ -> + failwith "preview-only fixture acquired a display file") + ~retry_asset:(fun ~graph_generation:_ _ -> + failwith "preview-only fixture retried a network asset") + ~changed:(fun _ _ -> ()) + () + in + Journal_media_runtime.reset + media + ~graph_generation:(Some (W.import_data imported).graph_generation); + Journal_media_runtime.imported media ~current:true imported; + Journal_media_runtime.retire_coordinated_previews media; + Journal_media_runtime.reset media ~graph_generation:None; + require + (List.length !released_previews = 1) + "binding cleanup and later reset released the same preview twice"; + require + (List.exists + (fun lease -> W.file_lease_id lease = W.file_lease_id preview) + !released_previews) + "typed Media preview cleanup lost its original lease"; + let release_preview, reply = request (W.Release_file preview) in + require (reply = W.Answered (Ok ())) "actual preview lease release did not ACK"; + apply (C.step !state (C.Forget imported_ticket)); + apply (C.step !state (C.Forget acquire)); + apply (C.step !state (C.Forget release)); + apply (C.step !state (C.Forget release_preview)); + W.encode_asset_import_intent intent, (W.import_data imported).asset) + in + let source_page, source_reply = + request + (W.Recover_import_inputs + { after = None; maximum_items = 2; maximum_bytes = 65536 }) + in + let source_rows = + match source_reply with + | W.Answered (Ok page) -> page.inputs + | _ -> failwith "actual Service source recovery failed" + in + require + (Rrbvec.length source_rows = 1) + "actual source recovery lost its original input"; + let original_source = Rrbvec.nth source_rows 0 in + require + (W.encode_asset_import_intent original_source.intent = import_bytes + && original_source.prepared <> None + && not original_source.acknowledged) "actual source recovery changed original input or Prepared membership"; - let old_source_proof=Option.get original_source.prepared in - apply(C.step !state(C.Forget source_page)); + let old_source_proof = Option.get original_source.prepared in + apply (C.step !state (C.Forget source_page)); apply (C.step !state C.Retry_bootstrap); require (Result.is_error (C.request !state session (W.Query (budget, W.Graph_info)))) @@ -2004,40 +3000,89 @@ let test_client_to_typed_read () = ~some:(fun next -> not (W.equal_session next session)) (current_session ())); let replacement = Option.get (current_session ()) in - let restored_import=W.restore_asset_import_intent replacement import_bytes |> Result.get_ok in - require(Result.is_error(C.request !state replacement(W.Acknowledge_import_source old_source_proof))) + let restored_import = + W.restore_asset_import_intent replacement import_bytes |> Result.get_ok + in + require + (Result.is_error + (C.request !state replacement (W.Acknowledge_import_source old_source_proof))) "old source proof authorized ACK across binding replacement"; - let source_check,source_reply=request_in replacement(W.Confirm_import_source restored_import) in - let original_source=match source_reply with W.Answered(Ok(Some item))->item|_->failwith "cold read-only original source confirmation failed" in - require(W.encode_asset_import_intent original_source.intent=import_bytes && not original_source.acknowledged) + let source_check, source_reply = + request_in replacement (W.Confirm_import_source restored_import) + in + let original_source = + match source_reply with + | W.Answered (Ok (Some item)) -> item + | _ -> failwith "cold read-only original source confirmation failed" + in + require + (W.encode_asset_import_intent original_source.intent = import_bytes + && not original_source.acknowledged) "cold source confirmation replaced input or fabricated ACK"; - let current_proof=Option.get original_source.prepared in - require(W.import_source_proof_matches restored_import(W.prepared_import_adoption current_proof)) + let current_proof = Option.get original_source.prepared in + require + (W.import_source_proof_matches + restored_import + (W.prepared_import_adoption current_proof)) "cold source proof lost the exact original input adoption identity"; - apply(C.step !state(C.Forget source_check)); - let source_ack,source_reply=request_in replacement(W.Acknowledge_import_source current_proof) in - require(source_reply=W.Answered(Ok()))"actual source proof ACK failed"; - apply(C.step !state(C.Forget source_ack)); - let pending_sources,source_reply=request_in replacement(W.Recover_import_inputs{after=None;maximum_items=2;maximum_bytes=65536}) in - require(match source_reply with W.Answered(Ok page)->Rrbvec.is_empty page.inputs && page.continuation=None|_->false) + apply (C.step !state (C.Forget source_check)); + let source_ack, source_reply = + request_in replacement (W.Acknowledge_import_source current_proof) + in + require (source_reply = W.Answered (Ok ())) "actual source proof ACK failed"; + apply (C.step !state (C.Forget source_ack)); + let pending_sources, source_reply = + request_in + replacement + (W.Recover_import_inputs + { after = None; maximum_items = 2; maximum_bytes = 65536 }) + in + require + (match source_reply with + | W.Answered (Ok page) -> Rrbvec.is_empty page.inputs && page.continuation = None + | _ -> false) "actual source ACK retained pending membership or fabricated More"; - apply(C.step !state(C.Forget pending_sources)); - let original_check,source_reply=request_in replacement(W.Confirm_import_source restored_import) in - require(match source_reply with W.Answered(Ok(Some item))->item.acknowledged && item.prepared<>None - && W.encode_asset_import_intent item.intent=import_bytes|_->false) + apply (C.step !state (C.Forget pending_sources)); + let original_check, source_reply = + request_in replacement (W.Confirm_import_source restored_import) + in + require + (match source_reply with + | W.Answered (Ok (Some item)) -> + item.acknowledged + && item.prepared <> None + && W.encode_asset_import_intent item.intent = import_bytes + | _ -> false) "actual ACK erased the original Prepared proof or complete input"; - apply(C.step !state(C.Forget original_check)); - let restored_ticket,restored_reply=request_in replacement(W.Import restored_import) in - let restored_imported=match restored_reply with W.Answered(Ok(W.Imported imported))->imported - |_->failwith "new binding could not reconcile the original staged import after picker removal" in - require((W.import_data restored_imported).asset=original_import_asset) "import reconciliation changed original descriptor/source bytes"; - require((W.import_data restored_imported).operation=source.operation && (W.import_data restored_imported).target=source.target) + apply (C.step !state (C.Forget original_check)); + let restored_ticket, restored_reply = + request_in replacement (W.Import restored_import) + in + let restored_imported = + match restored_reply with + | W.Answered (Ok (W.Imported imported)) -> imported + | _ -> + failwith + "new binding could not reconcile the original staged import after picker \ + removal" + in + require + ((W.import_data restored_imported).asset = original_import_asset) + "import reconciliation changed original descriptor/source bytes"; + require + ((W.import_data restored_imported).operation = source.operation + && (W.import_data restored_imported).target = source.target) "import reconciliation changed original operation or target"; - let preview=W.import_preview restored_imported |> Option.get in - require(Sys.file_exists(W.file_path preview)) "reconciled import lost durable staged source"; - let released,reply=request_in replacement(W.Release_file preview) in - require(reply=W.Answered(Ok())) "new-binding source preview did not physically release"; - apply(C.step !state(C.Forget restored_ticket));apply(C.step !state(C.Forget released)); + let preview = W.import_preview restored_imported |> Option.get in + require + (Sys.file_exists (W.file_path preview)) + "reconciled import lost durable staged source"; + let released, reply = request_in replacement (W.Release_file preview) in + require + (reply = W.Answered (Ok ())) + "new-binding source preview did not physically release"; + apply (C.step !state (C.Forget restored_ticket)); + apply (C.step !state (C.Forget released)); let next_budget = W.read_budget replacement ~maximum_items:1 ~maximum_bytes:65536 |> Result.get_ok in @@ -2056,7 +3101,7 @@ let test_client_to_typed_read () = "late cancelled read revived a forgotten consumer") ;; -let test_read_completion_rechecks_graph_owner ?(change=`Graph) () = +let test_read_completion_rechecks_graph_owner ?(change = `Graph) () = let armed = Atomic.make false and entered = Atomic.make false and release = Atomic.make false in @@ -2074,7 +3119,8 @@ let test_read_completion_rechecks_graph_owner ?(change=`Graph) () = ~finally:(fun () -> Atomic.set release true) (fun () -> let responses = Hashtbl.create 16 - and baseline = ref None and authority_revoked = ref false in + and baseline = ref None + and authority_revoked = ref false in Mailbox.on_event client (function | Mailbox.Response { request_id; outcome; _ } -> Hashtbl.replace responses request_id outcome @@ -2091,7 +3137,9 @@ let test_read_completion_rechecks_graph_owner ?(change=`Graph) () = Mailbox.Private.deliver client ~max_events:64; Thread.delay 0.001 done; - require (predicate ()) ("Service completion ownership race did not settle: " ^ stage) + require + (predicate ()) + ("Service completion ownership race did not settle: " ^ stage) in let send request = match Mailbox.send client request with @@ -2106,12 +3154,16 @@ let test_read_completion_rechecks_graph_owner ?(change=`Graph) () = (* The worker can publish Open before this test consumes the retained registration result. Exercise that real delivery order explicitly. *) wait "live graph open before retained registration" (fun () -> - Option.exists (fun value -> - List.exists (fun (_, state) -> - state.Logseq_db_worker.phase = Logseq_db_worker.Graph_open) - value.Service.baseline.state.residents) !baseline); + Option.exists + (fun value -> + List.exists + (fun (_, state) -> + state.Logseq_db_worker.phase = Logseq_db_worker.Graph_open) + value.Service.baseline.state.residents) + !baseline); (match Hashtbl.find responses id with - | Completed (Service.Coordination_registered value) -> remember_baseline baseline value + | Completed (Service.Coordination_registered value) -> + remember_baseline baseline value | _ -> assert false); wait "initial graph open" (fun () -> match !baseline with @@ -2122,34 +3174,76 @@ let test_read_completion_rechecks_graph_owner ?(change=`Graph) () = value.baseline.state.residents | _ -> false); let owner = Option.get (Option.get !baseline).baseline.state.sync.selected in - let read=send(query_request client !baseline owner registration - (uuid "99000000-0000-4000-8000-000000000001") ~maximum_bytes:Worker_pure_data.maximum_response_bytes - (Read.Outline_block{uuid=Test_support.authoritative_block_uuid;revision=None})) in + let read = + send + (query_request + client + !baseline + owner + registration + (uuid "99000000-0000-4000-8000-000000000001") + ~maximum_bytes:Worker_pure_data.maximum_response_bytes + (Read.Outline_block + { uuid = Test_support.authoritative_block_uuid; revision = None })) + in wait "initial block read" (fun () -> Hashtbl.mem responses read); let revision = match Hashtbl.find responses read with - |Completed(Service.Coordination_queried{answer=Read.Query_answer(_,Read.Outline_block _,Ok(Worker_pure_data.V2_present_block{revision;_}));_}) - -> revision + | Completed + (Service.Coordination_queried + { answer = + Read.Query_answer + ( _ + , Read.Outline_block _ + , Ok (Worker_pure_data.V2_present_block { revision; _ }) ) + ; _ + }) -> revision | _ -> failwith "race fixture block missing" in Atomic.set armed true; - let module Wire=Logseq_db_worker_contract.Worker_pure_mutation in - let wire=Wire.create_mutation ~origin:(Uri.of_string "https://api.logseq.io") - ~user:"user-1" ~graph:Test_support.graph_uuid - ~mutation_id:(uuid "99000000-0000-4000-8000-000000000003") - (Wire.Save_block{block=Test_support.authoritative_block_uuid;title="Synthetic held writer"; - preconditions={blocks=[Test_support.authoritative_block_uuid,revision];pages=[];scopes=[]}}) - |>Result.get_ok |>fun intent->Wire.Mutation_intent intent in - let writer=send(Service.Coordination_mutation{registration;owner; - request_id=uuid "99000000-0000-4000-8000-000000000002";intent=wire})in + let module Wire = Logseq_db_worker_contract.Worker_pure_mutation in + let wire = + Wire.create_mutation + ~origin:(Uri.of_string "https://api.logseq.io") + ~user:"user-1" + ~graph:Test_support.graph_uuid + ~mutation_id:(uuid "99000000-0000-4000-8000-000000000003") + (Wire.Save_block + { block = Test_support.authoritative_block_uuid + ; title = "Synthetic held writer" + ; preconditions = + { blocks = [ Test_support.authoritative_block_uuid, revision ] + ; pages = [] + ; scopes = [] + } + }) + |> Result.get_ok + |> fun intent -> Wire.Mutation_intent intent + in + let writer = + send + (Service.Coordination_mutation + { registration + ; owner + ; request_id = uuid "99000000-0000-4000-8000-000000000002" + ; intent = wire + }) + in wait "writer entered DB gate" (fun () -> Atomic.get entered); - let read=send(query_request client !baseline owner registration - (uuid "99000000-0000-4000-8000-000000000004") Read.Graph_info)in + let read = + send + (query_request + client + !baseline + owner + registration + (uuid "99000000-0000-4000-8000-000000000004") + Read.Graph_info) + in (* Cancelling the outer writer releases its Service lane, not its accepted durable DB operation. Keep that operation held while the second lane is waiting on the real snapshot mutex. *) Mailbox.cancel client ~request_id:writer; - wait "read request fiber started" (fun () -> (Mailbox.Private.metrics (Mailbox.Private.pack_client client)) .active_request_fibers @@ -2165,26 +3259,37 @@ let test_read_completion_rechecks_graph_owner ?(change=`Graph) () = | Some value -> value.baseline.state.sync.selected = None | _ -> false) | `Same | `Changed | `Signout -> - let user_id=match change with `Same->Some "user-1"|`Changed->Some "other-user"|_->None in - let auth=send(Service.Client_command(Reconcile_authenticated_user{user_id})) in - if change=`Same then wait "same identity confirmed" (fun()->Hashtbl.mem responses auth) - else wait "different authority revoked" (fun()-> !authority_revoked)); + let user_id = + match change with + | `Same -> Some "user-1" + | `Changed -> Some "other-user" + | _ -> None + in + let auth = + send (Service.Client_command (Reconcile_authenticated_user { user_id })) + in + if change = `Same + then wait "same identity confirmed" (fun () -> Hashtbl.mem responses auth) + else wait "different authority revoked" (fun () -> !authority_revoked)); Atomic.set release true; wait "retired read completion" (fun () -> Hashtbl.mem responses read); require (match Hashtbl.find responses read with - | Completed(Service.Coordination_queried{answer=Read.Query_answer(_,Read.Graph_info,Ok info);_}) - when change=`Same -> info.graph_uuid=Test_support.graph_uuid - | Completed(Service.Coordination_query_revoked _) | Failed _ when change<>`Same -> true + | Completed + (Service.Coordination_queried + { answer = Read.Query_answer (_, Read.Graph_info, Ok info); _ }) + when change = `Same -> info.graph_uuid = Test_support.graph_uuid + | (Completed (Service.Coordination_query_revoked _) | Failed _) + when change <> `Same -> true | _ -> false) "in-flight read did not respect the actual identity/graph transition"; - ( - wait "cancelled writer completion" (fun () -> Hashtbl.mem responses writer); - require - (Hashtbl.find responses writer = Mailbox.Cancelled) - "cancelled durable attempt was reported as known rollback"; - Atomic.set armed false; - if change=`Graph then ( + wait "cancelled writer completion" (fun () -> Hashtbl.mem responses writer); + require + (Hashtbl.find responses writer = Mailbox.Cancelled) + "cancelled durable attempt was reported as known rollback"; + Atomic.set armed false; + if change = `Graph + then ( ignore (send (Service.Client_command (Select_graph Test_support.graph_uuid))); wait "replacement graph open" (fun () -> match !baseline with @@ -2213,454 +3318,19 @@ let test_read_completion_rechecks_graph_owner ?(change=`Graph) () = match Hashtbl.find responses retry with | Completed (Service.Coordination_mutation_result - { outcome = - Wire.Committed committed - ; _ - }) -> + { outcome = Wire.Committed committed; _ }) -> require (committed.status = Worker_pure_data.V2_applied) "unknown write replay invented Already_applied/current revision"; require - (committed.before - <> committed.after) + (committed.before <> committed.after) "unknown accepted write lost original commit evidence" - | _ -> failwith "durable unknown did not reconcile across close/reopen")))) + | _ -> failwith "durable unknown did not reconcile across close/reopen"))) ;; let test_host_durable_recovery_confirmation_ack () = - let module Host=Journal_worker_coordination_host in - with_client ~statuses:true ~prepare:prepare_atomic_append_fixture(fun client-> - let jobs=Queue.create() in - let host=Host.create ~client ~client_id ~managed_origin:"https://api.logseq.io" ~user_id:"user-1" - ~enqueue:(fun job->Queue.add job jobs) ~changed:(fun _->()) |> Result.get_ok in - let consumed=ref 0 in - Mailbox.on_event client(fun event->let delivery=Host.on_event host event in - if delivery.R.disposition=R.Consumed then incr consumed); - let pump()=Mailbox.Private.deliver client ~max_events:64; - let count=Queue.length jobs in for _=1 to count do (Queue.take jobs)() done in - let wait predicate=let deadline=Unix.gettimeofday()+.6. in - while not(predicate()) && Unix.gettimeofday()Option.exists(fun projection->Option.is_some(W.selected projection) - && List.exists(fun resident->resident.W.phase=W.Open)(W.residents projection)) - (C.projection(Host.view host))); - let presentation=Option.get(Option.bind(C.projection(Host.view host))W.selected) in - let session=W.presentation_session presentation in - let call_in : type a. W.session -> a W.request -> a W.reply = fun session operation-> - let before= !consumed in - let ticket=Host.request host session operation |> Result.get_ok in - let result=ref None in - Host.watch host ticket ~f:(fun reply->result:=Some reply); - require(!result=None) "typed durable callback recursively completed admission"; - wait(fun()-> !result<>None); - require(!consumed=before+1) "typed lifecycle response has duplicate or missing owner"; - Option.get !result in - let call : type a. a W.request -> a W.reply = fun operation->call_in session operation in - let value : type a. a W.reply -> a = function W.Answered(Ok value)->value|_->failwith "typed lifecycle operation failed" in - let budget=W.read_budget session ~maximum_items:8 ~maximum_bytes:16384 |> Result.get_ok in - let read()=value(call(W.Query(budget,W.Outline_block{uuid=Test_support.authoritative_block_uuid;revision=None}))) in - let revision=match read() with Worker_pure_data.V2_present_block{revision;_}->revision|_->assert false in - let intent=W.mutation_intent session ~mutation_id:(uuid "9f000000-0000-4000-8000-000000001000") - (W.Save_block{block=Test_support.authoritative_block_uuid;title="Typed applied readback"; - preconditions={blocks=[Test_support.authoritative_block_uuid,revision];pages=[];scopes=[]}}) |> Result.get_ok in - let receipt=match value(call(W.Mutate intent)) with W.Submission_committed receipt->receipt|_->failwith "typed write lost proof" in - require(Result.is_error(Host.request host session(W.Acknowledge_intent(W.Mutation_intent intent)))) "typed unconfirmed commit admitted ACK"; - let recover()=value(call(W.Recover_mutations{after=None;maximum_items=8;maximum_bytes=W.maximum_mutation_recovery_bytes})) in - let page=recover() in - require(Rrbvec.length page.items=1) "typed recovery lost the persisted intent"; - require(match(Rrbvec.nth page.items 0).progress with W.Committed_pending_presentation actual-> - W.receipt_revisions actual=W.receipt_revisions receipt|_->false) "typed recovery replaced original proof"; - let confirmation=value(call(W.Confirm(presentation,receipt))) in - let ui_applied=match W.confirmation_primary confirmation with - | W.Confirmed_block{lookup=Worker_pure_data.V2_present_block{value;_};_}->value.block.title="Typed applied readback" - | _->false in - require ui_applied "canonical DTO was not applied to the consuming projection"; - ignore(value(call(W.Acknowledge_intent(W.Mutation_intent intent)))); - require(C.intent_progress(Host.view host)(W.Mutation_intent intent)=None) "actual typed ACK did not retire pure ownership"; - require(Rrbvec.is_empty(recover()).items) "actual typed ACK did not settle database recovery entry"; - let before=read() in - require(match value(call(W.Mutate intent)) with W.Submission_committed original-> - W.receipt_revisions original=W.receipt_revisions receipt|_->false) "typed ACK removed original idempotency proof"; - require(read()=before) "typed same-intent reconciliation executed a second write"; - let revision=match read() with Worker_pure_data.V2_present_block{revision;_}->revision|_->assert false in - let status=Test_support.status_uuid 2 in - let status_intent=W.mutation_intent session ~mutation_id:(uuid "9f000000-0000-4000-8000-000000001001") - (W.Set_task_status{block=Test_support.authoritative_block_uuid;status; - preconditions={blocks=[Test_support.authoritative_block_uuid,revision];pages=[];scopes=[]}}) |> Result.get_ok in - let status_receipt=match value(call(W.Mutate status_intent)) with - | W.Submission_committed receipt->receipt|_->failwith "real typed status commit failed" in - let status_confirmation=value(call(W.Confirm(presentation,status_receipt))) in - require(match W.confirmation_primary status_confirmation with - | W.Confirmed_block{lookup=Worker_pure_data.V2_present_block{value;_};_}->value.task_status=Some status - | _->false) "real canonical DTO lost committed status UUID"; - ignore(value(call(W.Acknowledge_intent(W.Mutation_intent status_intent)))); - require(Rrbvec.is_empty(recover()).items) "real status ACK retained pending recovery entry"; - let completed_appends=ref Rrbvec.empty in - List.iteri(fun index day-> - let root_id=uuid(Printf.sprintf "9f000000-0000-4000-8000-%012d" (2000+index*10)) in - let child_id=uuid(Printf.sprintf "9f000000-0000-4000-8000-%012d" (2001+index*10)) in - let append=W.append_intent session ~mutation_id:(uuid(Printf.sprintf "9f000000-0000-4000-8000-%012d" (2002+index*10))) - ~block_id:root_id ~day:(W.journal_day day |> Result.get_ok) ~title:"Atomic Service root 中文" - ~task_status:(Some status) ~children:(Rrbvec.singleton Worker_pure_data.{block_id=child_id;title="Atomic Service child"; - task_status=Some(Test_support.status_uuid 3);children=Rrbvec.empty}) |> Result.get_ok in - let receipt=match value(call(W.Append append)) with - | W.Submission_committed receipt->receipt|_->failwith "actual Append did not commit atomically" in - require(Rrbvec.length(W.receipt_targets receipt)=2) "Append receipt lost complete immutable targets"; - let confirmation=value(call(W.Confirm(presentation,receipt))) in - let summary=match W.confirmation_primary confirmation with W.Confirmed_block summary->summary - |_->failwith "Append canonical primary was not the admitted root" in - require(match summary.lookup,summary.page with - | Worker_pure_data.V2_present_block{value;_},Some(Worker_pure_data.V2_present_page{page;_})-> - value.block.uuid=root_id && value.block.title="Atomic Service root 中文" && value.task_status=Some status - && value.block.parent=page.uuid && value.block.page=page.uuid - && (if index=0 then page.title="2026-11-02" else - page.uuid=uuid "9f000000-0000-4000-8000-000000002100" && page.title="Existing Web day title") - |_->false) "Append lost settings format or adopted the candidate instead of the existing Web page"; - require(List.exists(fun(item:Worker_pure_data.child_member)->item.value.block.uuid=child_id - && item.value.task_status=Some(Test_support.status_uuid 3))summary.items) - "Append separated or lost its child status write"; - ignore(value(call(W.Acknowledge_intent(W.receipt_intent receipt)))); - require(Rrbvec.is_empty(recover()).items) "actual Append ACK did not settle its durable recovery item"; - require(match value(call((match W.restore_durable_intent session(W.encode_durable_intent(W.Append_intent append)) |> Result.get_ok with W.Append_intent intent->W.Append intent|Mutation_intent _->failwith "stored Append lost its family"))) with - | W.Submission_committed again->W.receipt_revisions again=W.receipt_revisions receipt - |_->false) "Append retry lost the original commit proof"; - require(value(call(W.Query(budget,W.Block_summary{block=root_id;after=None})))=summary) - "Append retry duplicated or changed the complete canonical tree"; - completed_appends:=Rrbvec.push_back !completed_appends(W.encode_durable_intent(W.Append_intent append),receipt,root_id,summary)) [20261102;20261103]; - let rejected_append=W.append_intent session - ~mutation_id:(uuid "9f000000-0000-4000-8000-000000002199") - ~block_id:Test_support.authoritative_block_uuid - ~day:(W.journal_day 20261102 |> Result.get_ok) ~title:"Conflicting existing block" - ~task_status:None ~children:Rrbvec.empty |> Result.get_ok in - require(match value(call(W.Append rejected_append)) with W.Submission_rejected _->true|_->false) - "existing block Append conflict was not a known non-commit"; - require(match C.intent_progress(Host.view host)(W.Append_intent rejected_append) with - |Some(W.Rejected _)->true|_->false) "Rejected Append lost its original lifecycle family"; - ignore(value(call(W.Acknowledge_intent(W.Append_intent rejected_append)))); - require(C.intent_progress(Host.view host)(W.Append_intent rejected_append)=None - && Rrbvec.is_empty(recover()).items) - "Rejected Append without a receipt could not settle its original proof"; - (* Child business ordering against the actual Service and persisted intents. - UI application here uses the production pure block projection; mounted - application tests separately verify the rendered composer barrier. *) - let module Child=Journal_graph_mutation_confirmation.Pure_insertion in - let parent_summary=value(call(W.Query(budget,W.Block_summary - {block=Test_support.authoritative_block_uuid;after=None}))) in - let parent_revision,page=match parent_summary.lookup,parent_summary.page with - | Worker_pure_data.V2_present_block{revision;_},Some(Worker_pure_data.V2_present_page{page;_})->revision,page - | _->failwith "real Child parent unavailable" in - let child=uuid "9f000000-0000-4000-8000-000000001003" in - let mutation_id=uuid "9f000000-0000-4000-8000-000000001004" in - let command:Journal_graph_projection.create_child= - {mutation_id=W.Uuid.to_string mutation_id;calendar_generation=1L; - block_id=W.Uuid.to_string child;parent_block_id=W.Uuid.to_string Test_support.authoritative_block_uuid; - expected_parent_revision=parent_revision;sibling_order="a";source="Typed real child"; - task_state=Journal_model.Status(W.Uuid.to_string status);status_confirmation=true;status_intent_revision=0L; - creation_time=Journal_time.create ~instant_unix_ms:1_788_192_000_000L ~local_day:20260901 - ~local_minute_of_day:0 |> Result.get_ok} in - let insert=W.Insert_blocks{parent=Test_support.authoritative_block_uuid; - roots=[{uuid=child;title=command.source;children=[]}]; - preconditions={blocks=[Test_support.authoritative_block_uuid,parent_revision];pages=[]; - scopes=[Worker_pure_data.V2_children_scope Test_support.authoritative_block_uuid,parent_summary.scope_revision]}} in - let initial,action=Child.start ~command ~page ~resolved_status:(Some status) ~insert |> Result.get_ok in - let writes=ref 0 and acks=ref 0 and applied=ref None and proofs=ref [] in - let rec drive fuel state action = - require(fuel>0) "actual Child lifecycle failed to settle"; - let next,action=match action with - | Child.Write(index,round) -> - incr writes; - let stage=Child.stage state index in - let durable=Child.stage_intent session stage |> Result.get_ok in - let intent=match durable with W.Mutation_intent intent->intent|_->assert false in - let state=Child.bind state ~index ~round ~intent:(W.Mutation_intent intent) ~presentation |> Result.get_ok in - let receipt=match value(call(W.Mutate intent)) with - | W.Submission_committed receipt->receipt|_->failwith "real Child lost durable commit" in - proofs:=(intent,receipt):: !proofs; - Child.receive state ~index ~round(W.Confirmed(value(call(W.Confirm(presentation,receipt))))) - | Read_child round -> - Child.read_child state ~round(value(call(W.Query(budget,W.Block_summary{block=child;after=None})))) - | Read_parent(round,summary) -> - ignore(value(call(W.Query(budget,W.Outline_block{uuid=Test_support.authoritative_block_uuid;revision=None})))); - let block,revision,page=match summary.lookup,summary.page with - | Worker_pure_data.V2_present_block{value;revision},Some(Worker_pure_data.V2_present_page{page;_})->value.block,revision,page - | _->failwith "real Child canonical DTO absent" in - applied:=Some(Journal_graph_projection.block_on_page ~page ~revision - ~child_count:(List.length summary.items) ~time_context:{localtime=Unix.gmtime} block |> Result.get_ok); - Child.projection_ready state ~round - | Present round -> - require(Option.exists(fun block->Journal_model.source block=command.source) !applied) - "Child ACK preceded production projection application"; - Child.applied state ~round presentation - | Acknowledge(index,round,intent,_) -> - require(!applied<>None) "actual Child intent acknowledged without a consumer"; - ignore(value(call(W.Acknowledge_intent intent)));incr acks; - require(Rrbvec.is_empty(recover()).items) "settled Child prefix retained recovery obligation"; - Child.acknowledged state ~index ~round(Ok()) - | Complete->state,Complete - | Wait message->failwith("actual Child stalled: "^message) - | Ignore->failwith "actual Child dropped its current continuation" in - match action with Child.Complete->next|_->drive(fuel-1)next action in - ignore(drive 24 initial action); - require(!writes=2 && !acks=2) "real Child bypassed Insert/Status or per-stage ACK"; - let final=value(call(W.Query(budget,W.Outline_block{uuid=child;revision=None}))) in - require(match final with Worker_pure_data.V2_present_block{value;_}->value.block.title=command.source && value.task_status=Some status|_->false) - "real Child chain did not preserve source/status"; - List.iter(fun(intent,receipt)-> - require(match value(call(W.Mutate intent)) with - | W.Submission_committed original->W.receipt_revisions original=W.receipt_revisions receipt - | _->false) "Child ACK removed original commit proof") !proofs; - require(value(call(W.Query(budget,W.Outline_block{uuid=child;revision=None})))=final) - "Child original-intent reconciliation wrote a duplicate"; - - let revision=match read() with Worker_pure_data.V2_present_block{revision;_}->revision|_->assert false in - let deletion=W.mutation_intent session ~mutation_id:(uuid "9f000000-0000-4000-8000-000000001002") - (W.Delete_blocks{root=Test_support.authoritative_block_uuid; - preconditions={blocks=[Test_support.authoritative_block_uuid,revision];pages=[];scopes=[]}}) |> Result.get_ok in - let deleted_receipt=match value(call(W.Mutate deletion)) with - | W.Submission_committed receipt->receipt|_->failwith "real typed deletion failed" in - let deleted_confirmation=value(call(W.Confirm(presentation,deleted_receipt))) in - require(match W.confirmation_primary deleted_confirmation with - | W.Confirmed_absence{uuid;_}->W.Uuid.equal uuid Test_support.authoritative_block_uuid - | _->false) "typed deletion did not deliver exact canonical absence"; - ignore(value(call(W.Acknowledge_intent(W.Mutation_intent deletion)))); - require(Rrbvec.is_empty(recover()).items) "real deletion ACK retained pending recovery entry"; - require(match value(call(W.Mutate deletion)) with - | W.Submission_committed original->W.receipt_revisions original=W.receipt_revisions deleted_receipt - | _->false) "deletion reconciliation lost its original idempotent proof"; - require(match read() with Worker_pure_data.V2_missing_block{uuid;_}->W.Uuid.equal uuid Test_support.authoritative_block_uuid|_->false) - "same-intent deletion reconciliation changed canonical absence"; - - - ignore(Mailbox.send client(Service.Client_command Return_to_graph_picker)); - wait(fun()->Option.exists(fun projection->W.selected projection=None)(C.projection(Host.view host))); - ignore(Mailbox.send client(Service.Client_command(Select_graph Test_support.graph_uuid))); - wait(fun()->Option.exists(fun projection-> - match W.selected projection with - | Some current->let fresh=W.presentation_session current in - not(W.equal_session fresh session) - && List.exists(fun resident->resident.W.phase=W.Open && W.equal_session resident.session fresh)(W.residents projection) - | None->false)(C.projection(Host.view host))); - let current=Option.get(Option.bind(C.projection(Host.view host))W.selected) in - let fresh=W.presentation_session current in - Rrbvec.iter(fun(bytes,original,root_id,summary)-> - let append=match W.restore_durable_intent fresh bytes |> Result.get_ok with W.Append_intent intent->intent|Mutation_intent _->failwith "recovered Append lost its family" in - let receipt=match value(call_in fresh(W.Append append)) with - | W.Submission_committed receipt->receipt|_->failwith "reopened Append did not reconcile" in - require(W.receipt_revisions receipt=W.receipt_revisions original) - "reopened Append replaced original evidence with the new read generation"; - let confirmation=value(call_in fresh(W.Confirm(current,receipt))) in - require(match W.confirmation_primary confirmation with - | W.Confirmed_block value->value.lookup=summary.W.lookup && value.page=summary.page && value.items=summary.items - | _->false) "reopened Append changed the original complete source/status tree"; - require(W.receipt_targets receipt=W.receipt_targets original - && Rrbvec.exists(W.Uuid.equal root_id)(W.receipt_targets receipt)) "reopened Append changed stable targets"; - ignore(value(call_in fresh(W.Acknowledge_intent(W.receipt_intent receipt))))) !completed_appends; - ignore(Host.shutdown host); - wait(fun()->C.shutdown_state(Host.view host)=C.Stopped)) -;; - -let test_host_preserves_query_origin_before_forget () = - let module Host = Journal_worker_coordination_host in - with_client (fun client -> - let queue = Queue.create () in - let host = - Host.create - ~client - ~client_id - ~managed_origin:"https://api.logseq.io" - ~user_id:"user-1" - ~enqueue:(fun f -> Queue.add f queue) - ~changed:(fun _ -> ()) - |> Result.get_ok - in - Mailbox.on_event client (fun event -> ignore (Host.on_event host event)); - let pump () = - Mailbox.Private.deliver client ~max_events:64; - let count = Queue.length queue in - for _ = 1 to count do - (Queue.take queue) () - done - in - let wait predicate = - let deadline = Unix.gettimeofday () +. 4. in - while (not (predicate ())) && Unix.gettimeofday () < deadline do - pump (); - Thread.delay 0.001 - done; - require (predicate ()) "host provenance integration did not settle" - in - wait (fun () -> - match C.bootstrap (Host.view host) with - | W.Bootstrap_not_ready _ -> true - | _ -> false); - let owner : Service.timeline_scope = - { startup = { account_generation = 1; presentation_generation = 1 } - ; graph_id = Test_support.graph_uuid - ; graph_generation = 1 - } - in - let ticket = - Host.plan_query host ~owner ~maximum_items:1 ~maximum_bytes:65536 W.Graph_info - |> Result.get_ok - in - let callback = ref None - and calls = ref 0 in - Host.watch_read host ticket ~f:(fun origin reply -> - incr calls; - require - (C.query_origin (Host.view host) ticket = None) - "business callback ran before ticket Forget"; - callback := Some (origin, reply)); - ignore - (Mailbox.send - client - (Service.Client_command (Restore_local_account { user_id = "user-1" }))); - wait (fun () -> Option.is_some !callback); - let session, budget = - match Option.get !callback with - | Some origin, W.Answered (Ok info) -> - require - (W.Uuid.equal info.graph_uuid Test_support.graph_uuid) - "real host query lost graph"; - origin - | _ -> failwith "real host callback lost exact execution origin" - in - require - (W.validate_request session (W.Query (budget, W.Graph_info)) = Ok ()) - "forwarded origin budget is unrelated to Send"; - require (Host.validate_session host session = Ok ()) "live read origin was rejected"; - ignore (Host.shutdown host); - wait (fun () -> C.shutdown_state (Host.view host) = C.Stopped); - require - (Host.validate_session host session = Error W.Revoked_session) - "stopped host allowed cached read"; - require (!calls = 1) "query origin callback was delivered more than once") -;; - -let test_visible_media_listing_uses_typed_reads () = - let module Host=Journal_worker_coordination_host in - let module Media=Journal_media_runtime in - with_client ~favorites:true ~assets:true (fun client-> - let jobs=Queue.create() in - let host=Host.create ~client ~client_id ~managed_origin:"https://api.logseq.io" ~user_id:"user-1" - ~enqueue:(fun f->Queue.add f jobs) ~changed:(fun _->()) |> Result.get_ok in - Mailbox.on_event client(fun event->ignore(Host.on_event host event)); - let pump()=Mailbox.Private.deliver client ~max_events:64; - let count=Queue.length jobs in for _=1 to count do (Queue.take jobs)() done in - let wait predicate=let deadline=Unix.gettimeofday()+.6. in - while not(predicate()) && Unix.gettimeofday()Option.exists(fun p->List.exists(fun r->r.W.phase=W.Open)(W.residents p))(C.projection(Host.view host))); - let owner:Service.timeline_scope={startup={account_generation=1;presentation_generation=1};graph_id=Test_support.graph_uuid;graph_generation=1} in - let runtime=ref None and typed_reads=ref 0 and seen=ref [] and finished=ref false and last=ref None and held=ref None and retiring=ref false and late_returned=ref false in - let root=W.Uuid.to_string(journal_page 1) in - let read(Media.Read request)= - let ticket=Host.plan_query_when_available host ~owner ~maximum_items:(min 1 request.maximum_items) ~maximum_bytes:65536 request.query |> Result.get_ok in - (match request.query with W.Assets _ -> incr typed_reads | _ -> ()); - let number= !typed_reads in - Host.watch_read host ticket ~f:(fun origin reply-> - require(Option.exists(fun(session,_)->Host.validate_session ~owner host session=Ok())origin) "media listing borrowed its response origin"; - match reply with - | W.Answered(Ok value)-> - let result=request.outcome value in - (match request.query with - | W.Assets _ -> - last:=Some result; - if !retiring then ( - require(not(Media.receive_read(Option.get !runtime)result)) "in-flight result revived reset scope"; - late_returned:=true) - else if number=1 then held:=Some result else ( - require(Media.receive_read(Option.get !runtime) result) "owned typed media page was lost"; - require(not(Media.receive_read(Option.get !runtime) result)) "media page consumed twice"; - if Option.is_some value.next then Queue.add(fun()->Media.next(Option.get !runtime) ~root)jobs - else finished:=true) - | _ -> ignore (Media.receive_read (Option.get !runtime) result)) - | _->failwith "real typed media listing failed");true in - let media=Media.create ~read - ~demand:(host_asset_demand host ~owner - ~receive:(fun ticket demand->ignore(Media.receive_demand(Option.get !runtime)ticket demand))) - ~release_consumer:(host_asset_release host) - ~acquire:(fun _ _->failwith "metadata listing unexpectedly acquired a rendered file") - ~release_file:(fun _->failwith "metadata listing unexpectedly released a rendered file") - ~retry_asset:(fun ~graph_generation:_ _->failwith "metadata listing unexpectedly retried a network asset") - ~changed:(fun _ (view:Media.view)-> - require(view.error=None) "typed media projection failed"; - seen:=List.rev_append(List.map(fun(item:Media.item)->item.asset.uuid)view.items)!seen) () in - runtime:=Some media; - Media.reset media ~graph_generation:(Some 1); - Media.root_visible media ~root true; - wait(fun()->Option.is_some !held); - Media.resync media; - require(Media.receive_read media(Option.get !held)) "superseded media read lost its owning terminal"; - require(!typed_reads=2) "typed completion left the replacement listing queued without pumping"; - require(not(Media.receive_read media(Option.get !held))) "superseded media terminal consumed twice"; - wait(fun()-> !finished); - require(!typed_reads>=3) "media fixture did not exercise sealed pagination"; - require(List.sort_uniq W.Uuid.compare !seen=List.init 3(fun n->uuid(Printf.sprintf "9c000000-0000-4000-8000-%012d"(901+n)))) - "visible media listing lost a real external descriptor"; - retiring:=true; - Media.resync media; - Media.reset media ~graph_generation:(Some 2); - wait(fun()-> !late_returned); - require(not(Media.receive_read media (Option.get !last))) "old media result revived reset scope"; - ignore(Host.shutdown host);wait(fun()->C.shutdown_state(Host.view host)=C.Stopped)) -;; - -let test_offline_asset_scans_use_typed_reads () = - let module Host=Journal_worker_coordination_host in - let module Assets=Journal_asset_runtime in - with_client ~favorites:true ~assets:true (fun client-> - let jobs=Queue.create() in - let host=Host.create ~client ~client_id ~managed_origin:"https://api.logseq.io" ~user_id:"user-1" - ~enqueue:(fun f->Queue.add f jobs) ~changed:(fun _->()) |> Result.get_ok in - Mailbox.on_event client(fun event->ignore(Host.on_event host event)); - let pump()=Mailbox.Private.deliver client ~max_events:64; - let count=Queue.length jobs in for _=1 to count do (Queue.take jobs)() done in - let wait predicate=let deadline=Unix.gettimeofday()+.6. in - while not(predicate()) && Unix.gettimeofday()Option.exists(fun p->List.exists(fun r->r.W.phase=W.Open)(W.residents p))(C.projection(Host.view host))); - let owner:Service.timeline_scope={startup={account_generation=1;presentation_generation=1};graph_id=Test_support.graph_uuid;graph_generation=1} in - let runtime=ref None and typed_reads=ref 0 and status=ref None and asset_continuations=ref 0 and asset_rows=ref [] in - let read(Assets.Read request)= - (match request.query with W.Assets {after=Some _;_}->incr asset_continuations|_->()); - let ticket=Host.plan_query_when_available host ~owner ~maximum_items:(min 1 request.maximum_items) ~maximum_bytes:65536 request.query |> Result.get_ok in - incr typed_reads; - Host.watch_read host ticket ~f:(fun origin reply-> - require(Option.exists(fun(session,_)->Host.validate_session ~owner host session=Ok())origin) "asset scan borrowed its response origin"; - match reply with - | W.Answered(Ok value)-> - (match request.query with - | W.Assets _->asset_rows:=List.rev_append (List.map(fun(a:Logseq_db_types.Asset_descriptor.t)->a.uuid)value.items) !asset_rows - | _->()); - let result=request.outcome value in - require(Assets.receive_read(Option.get !runtime)~request_id:request.request_id result) - "owned typed scan result was lost"; - require(not(Assets.receive_read(Option.get !runtime)~request_id:request.request_id result)) - "typed scan consumed the same result twice" - | _->failwith "real typed offline asset read failed");true in - let assets=Assets.create ~read - ~demand:(host_asset_demand host ~owner - ~receive:(fun ticket demand->ignore(Assets.receive_demand(Option.get !runtime)ticket demand))) - ~release_consumer:(host_asset_release host) - ~changed:(fun _ recent favorites->status:=Some(recent,favorites)) () in - runtime:=Some assets; - Assets.refresh assets ~graph_generation:1 ~today:20260901 ~settings:Journal_asset_policy.default_settings; - wait(fun()->match !status with Some(recent,favorites)-> - recent.Journal_asset_policy.enumeration=Complete && favorites.enumeration=Complete|_->false); - require(!typed_reads>=3) "offline scan did not execute real catalog and asset query families"; - require(!asset_continuations>=2) (Printf.sprintf "real asset More did not retain its sealed continuation (after=%d)" !asset_continuations); - require(List.sort_uniq W.Uuid.compare !asset_rows = List.init 3(fun n->uuid(Printf.sprintf "9c000000-0000-4000-8000-%012d" (901+n)))) - "asset pagination lost a real external descriptor"; - Assets.shutdown assets;ignore(Host.shutdown host);wait(fun()->C.shutdown_state(Host.view host)=C.Stopped)) -;; - -let test_dependency_point_reads_use_typed_owner () = let module Host = Journal_worker_coordination_host in - with_client (fun client -> + with_client ~statuses:true ~prepare:prepare_atomic_append_fixture (fun client -> let jobs = Queue.create () in let host = Host.create @@ -2668,13 +3338,14 @@ let test_dependency_point_reads_use_typed_owner () = ~client_id ~managed_origin:"https://api.logseq.io" ~user_id:"user-1" - ~enqueue:(fun f -> Queue.add f jobs) + ~enqueue:(fun job -> Queue.add job jobs) ~changed:(fun _ -> ()) |> Result.get_ok in let consumed = ref 0 in Mailbox.on_event client (fun event -> - if (Host.on_event host event).R.disposition = R.Consumed then incr consumed); + let delivery = Host.on_event host event in + if delivery.R.disposition = R.Consumed then incr consumed); let pump () = Mailbox.Private.deliver client ~max_events:64; let count = Queue.length jobs in @@ -2688,7 +3359,7 @@ let test_dependency_point_reads_use_typed_owner () = pump (); Thread.delay 0.001 done; - require (predicate ()) "typed dependency point query did not settle" + require (predicate ()) "typed durable lifecycle did not settle" in ignore (Mailbox.send @@ -2696,1993 +3367,5226 @@ let test_dependency_point_reads_use_typed_owner () = (Service.Client_command (Restore_local_account { user_id = "user-1" }))); wait (fun () -> Option.exists - (fun p -> List.exists (fun r -> r.W.phase = W.Open) (W.residents p)) + (fun projection -> + Option.is_some (W.selected projection) + && List.exists + (fun resident -> resident.W.phase = W.Open) + (W.residents projection)) (C.projection (Host.view host))); - let owner : Service.timeline_scope = - { startup = { account_generation = 1; presentation_generation = 1 } - ; graph_id = Test_support.graph_uuid - ; graph_generation = 1 - } + let presentation = + Option.get (Option.bind (C.projection (Host.view host)) W.selected) in - let query : type a. int -> a W.query -> (a, W.error) result = - fun maximum_bytes query -> - let result = ref None in + let session = W.presentation_session presentation in + let call_in : type a. W.session -> a W.request -> a W.reply = + fun session operation -> let before = !consumed in - let ticket = Host.plan_query_when_available host ~owner ~maximum_items:1 - ~maximum_bytes query |> Result.get_ok in - Host.watch_read host ticket ~f:(fun origin reply -> result := Some (origin, reply)); - require (!result = None) "point callback ran synchronously before business admission"; + let ticket = Host.request host session operation |> Result.get_ok in + let result = ref None in + Host.watch host ticket ~f:(fun reply -> result := Some reply); + require (!result = None) "typed durable callback recursively completed admission"; wait (fun () -> !result <> None); - require (!consumed = before + 1) - "dependency point response was not consumed exactly once"; - let origin, reply = Option.get !result in require - (Option.exists - (fun (session, _) -> Host.validate_session ~owner host session = Ok ()) origin) - "point dependency inferred its origin after delivery"; - match reply with - | W.Answered result -> result - | _ -> failwith "dependency read was not answered" + (!consumed = before + 1) + "typed lifecycle response has duplicate or missing owner"; + Option.get !result + in + let call : type a. a W.request -> a W.reply = + fun operation -> call_in session operation + in + let value : type a. a W.reply -> a = function + | W.Answered (Ok value) -> value + | _ -> failwith "typed lifecycle operation failed" + in + let budget = + W.read_budget session ~maximum_items:8 ~maximum_bytes:16384 |> Result.get_ok + in + let read () = + value + (call + (W.Query + ( budget + , W.Outline_block + { uuid = Test_support.authoritative_block_uuid; revision = None } ))) + in + let revision = + match read () with + | Worker_pure_data.V2_present_block { revision; _ } -> revision + | _ -> assert false + in + let intent = + W.mutation_intent + session + ~mutation_id:(uuid "9f000000-0000-4000-8000-000000001000") + (W.Save_block + { block = Test_support.authoritative_block_uuid + ; title = "Typed applied readback" + ; preconditions = + { blocks = [ Test_support.authoritative_block_uuid, revision ] + ; pages = [] + ; scopes = [] + } + }) + |> Result.get_ok + in + let receipt = + match value (call (W.Mutate intent)) with + | W.Submission_committed receipt -> receipt + | _ -> failwith "typed write lost proof" in require - (match - query - 65536 - (W.Outline_block - { uuid = Test_support.authoritative_block_uuid; revision = None }) - with - | Ok (Worker_pure_data.V2_present_block { value; revision }) -> - value.block.uuid = Test_support.authoritative_block_uuid && revision <> "" + (Result.is_error + (Host.request host session (W.Acknowledge_intent (W.Mutation_intent intent)))) + "typed unconfirmed commit admitted ACK"; + let recover () = + value + (call + (W.Recover_mutations + { after = None + ; maximum_items = 8 + ; maximum_bytes = W.maximum_mutation_recovery_bytes + })) + in + let page = recover () in + require (Rrbvec.length page.items = 1) "typed recovery lost the persisted intent"; + require + (match (Rrbvec.nth page.items 0).progress with + | W.Committed_pending_presentation actual -> + W.receipt_revisions actual = W.receipt_revisions receipt | _ -> false) - "typed dependency lost full block identity or revision"; + "typed recovery replaced original proof"; + let confirmation = value (call (W.Confirm (presentation, receipt))) in + let ui_applied = + match W.confirmation_primary confirmation with + | W.Confirmed_block { lookup = Worker_pure_data.V2_present_block { value; _ }; _ } + -> value.block.title = "Typed applied readback" + | _ -> false + in + require ui_applied "canonical DTO was not applied to the consuming projection"; + ignore (value (call (W.Acknowledge_intent (W.Mutation_intent intent)))); require - (match - query 65536 (W.Page { uuid = Test_support.page_uuid; revision = None }) - with - | Ok (Worker_pure_data.V2_present_page { page; revision }) -> - page.uuid = Test_support.page_uuid && revision <> "" + (C.intent_progress (Host.view host) (W.Mutation_intent intent) = None) + "actual typed ACK did not retire pure ownership"; + require + (Rrbvec.is_empty (recover ()).items) + "actual typed ACK did not settle database recovery entry"; + let before = read () in + require + (match value (call (W.Mutate intent)) with + | W.Submission_committed original -> + W.receipt_revisions original = W.receipt_revisions receipt | _ -> false) - "typed dependency lost its full page DTO"; + "typed ACK removed original idempotency proof"; + require (read () = before) "typed same-intent reconciliation executed a second write"; + let revision = + match read () with + | Worker_pure_data.V2_present_block { revision; _ } -> revision + | _ -> assert false + in + let status = Test_support.status_uuid 2 in + let status_intent = + W.mutation_intent + session + ~mutation_id:(uuid "9f000000-0000-4000-8000-000000001001") + (W.Set_task_status + { block = Test_support.authoritative_block_uuid + ; status + ; preconditions = + { blocks = [ Test_support.authoritative_block_uuid, revision ] + ; pages = [] + ; scopes = [] + } + }) + |> Result.get_ok + in + let status_receipt = + match value (call (W.Mutate status_intent)) with + | W.Submission_committed receipt -> receipt + | _ -> failwith "real typed status commit failed" + in + let status_confirmation = value (call (W.Confirm (presentation, status_receipt))) in require - (match - query - 64 - (W.Outline_block - { uuid = Test_support.authoritative_block_uuid; revision = None }) - with - | Error W.Response_budget_exceeded -> true + (match W.confirmation_primary status_confirmation with + | W.Confirmed_block { lookup = Worker_pure_data.V2_present_block { value; _ }; _ } + -> value.task_status = Some status | _ -> false) - "point read bypassed encoded budget validation"; + "real canonical DTO lost committed status UUID"; + ignore (value (call (W.Acknowledge_intent (W.Mutation_intent status_intent)))); + require + (Rrbvec.is_empty (recover ()).items) + "real status ACK retained pending recovery entry"; + let completed_appends = ref Rrbvec.empty in + List.iteri + (fun index day -> + let root_id = + uuid (Printf.sprintf "9f000000-0000-4000-8000-%012d" (2000 + (index * 10))) + in + let child_id = + uuid (Printf.sprintf "9f000000-0000-4000-8000-%012d" (2001 + (index * 10))) + in + let append = + W.append_intent + session + ~mutation_id: + (uuid + (Printf.sprintf "9f000000-0000-4000-8000-%012d" (2002 + (index * 10)))) + ~block_id:root_id + ~day:(W.journal_day day |> Result.get_ok) + ~title:"Atomic Service root 中文" + ~task_status:(Some status) + ~children: + (Rrbvec.singleton + Worker_pure_data. + { block_id = child_id + ; title = "Atomic Service child" + ; task_status = Some (Test_support.status_uuid 3) + ; children = Rrbvec.empty + }) + |> Result.get_ok + in + let receipt = + match value (call (W.Append append)) with + | W.Submission_committed receipt -> receipt + | _ -> failwith "actual Append did not commit atomically" + in + require + (Rrbvec.length (W.receipt_targets receipt) = 2) + "Append receipt lost complete immutable targets"; + let confirmation = value (call (W.Confirm (presentation, receipt))) in + let summary = + match W.confirmation_primary confirmation with + | W.Confirmed_block summary -> summary + | _ -> failwith "Append canonical primary was not the admitted root" + in + require + (match summary.lookup, summary.page with + | ( Worker_pure_data.V2_present_block { value; _ } + , Some (Worker_pure_data.V2_present_page { page; _ }) ) -> + value.block.uuid = root_id + && value.block.title = "Atomic Service root 中文" + && value.task_status = Some status + && value.block.parent = page.uuid + && value.block.page = page.uuid + && + if index = 0 + then page.title = "2026-11-02" + else + page.uuid = uuid "9f000000-0000-4000-8000-000000002100" + && page.title = "Existing Web day title" + | _ -> false) + "Append lost settings format or adopted the candidate instead of the existing \ + Web page"; + require + (List.exists + (fun (item : Worker_pure_data.child_member) -> + item.value.block.uuid = child_id + && item.value.task_status = Some (Test_support.status_uuid 3)) + summary.items) + "Append separated or lost its child status write"; + ignore (value (call (W.Acknowledge_intent (W.receipt_intent receipt)))); + require + (Rrbvec.is_empty (recover ()).items) + "actual Append ACK did not settle its durable recovery item"; + require + (match + value + (call + (match + W.restore_durable_intent + session + (W.encode_durable_intent (W.Append_intent append)) + |> Result.get_ok + with + | W.Append_intent intent -> W.Append intent + | Mutation_intent _ -> failwith "stored Append lost its family")) + with + | W.Submission_committed again -> + W.receipt_revisions again = W.receipt_revisions receipt + | _ -> false) + "Append retry lost the original commit proof"; + require + (value + (call (W.Query (budget, W.Block_summary { block = root_id; after = None }))) + = summary) + "Append retry duplicated or changed the complete canonical tree"; + completed_appends + := Rrbvec.push_back + !completed_appends + (W.encode_durable_intent (W.Append_intent append), receipt, root_id, summary)) + [ 20261102; 20261103 ]; + let rejected_append = + W.append_intent + session + ~mutation_id:(uuid "9f000000-0000-4000-8000-000000002199") + ~block_id:Test_support.authoritative_block_uuid + ~day:(W.journal_day 20261102 |> Result.get_ok) + ~title:"Conflicting existing block" + ~task_status:None + ~children:Rrbvec.empty + |> Result.get_ok + in + require + (match value (call (W.Append rejected_append)) with + | W.Submission_rejected _ -> true + | _ -> false) + "existing block Append conflict was not a known non-commit"; + require + (match C.intent_progress (Host.view host) (W.Append_intent rejected_append) with + | Some (W.Rejected _) -> true + | _ -> false) + "Rejected Append lost its original lifecycle family"; + ignore (value (call (W.Acknowledge_intent (W.Append_intent rejected_append)))); + require + (C.intent_progress (Host.view host) (W.Append_intent rejected_append) = None + && Rrbvec.is_empty (recover ()).items) + "Rejected Append without a receipt could not settle its original proof"; + (* Child business ordering against the actual Service and persisted intents. + UI application here uses the production pure block projection; mounted + application tests separately verify the rendered composer barrier. *) + let module Child = Journal_graph_mutation_confirmation.Pure_insertion in + let parent_summary = + value + (call + (W.Query + ( budget + , W.Block_summary + { block = Test_support.authoritative_block_uuid; after = None } ))) + in + let parent_revision, page = + match parent_summary.lookup, parent_summary.page with + | ( Worker_pure_data.V2_present_block { revision; _ } + , Some (Worker_pure_data.V2_present_page { page; _ }) ) -> revision, page + | _ -> failwith "real Child parent unavailable" + in + let child = uuid "9f000000-0000-4000-8000-000000001003" in + let mutation_id = uuid "9f000000-0000-4000-8000-000000001004" in + let command : Journal_graph_projection.create_child = + { mutation_id = W.Uuid.to_string mutation_id + ; calendar_generation = 1L + ; block_id = W.Uuid.to_string child + ; parent_block_id = W.Uuid.to_string Test_support.authoritative_block_uuid + ; expected_parent_revision = parent_revision + ; sibling_order = "a" + ; source = "Typed real child" + ; task_state = Journal_model.Status (W.Uuid.to_string status) + ; status_confirmation = true + ; status_intent_revision = 0L + ; creation_time = + Journal_time.create + ~instant_unix_ms:1_788_192_000_000L + ~local_day:20260901 + ~local_minute_of_day:0 + |> Result.get_ok + } + in + let insert = + W.Insert_blocks + { parent = Test_support.authoritative_block_uuid + ; roots = [ { uuid = child; title = command.source; children = [] } ] + ; preconditions = + { blocks = [ Test_support.authoritative_block_uuid, parent_revision ] + ; pages = [] + ; scopes = + [ ( Worker_pure_data.V2_children_scope + Test_support.authoritative_block_uuid + , parent_summary.scope_revision ) + ] + } + } + in + let initial, action = + Child.start ~command ~page ~resolved_status:(Some status) ~insert |> Result.get_ok + in + let writes = ref 0 + and acks = ref 0 + and applied = ref None + and proofs = ref [] in + let rec drive fuel state action = + require (fuel > 0) "actual Child lifecycle failed to settle"; + let next, action = + match action with + | Child.Write (index, round) -> + incr writes; + let stage = Child.stage state index in + let durable = Child.stage_intent session stage |> Result.get_ok in + let intent = + match durable with + | W.Mutation_intent intent -> intent + | _ -> assert false + in + let state = + Child.bind + state + ~index + ~round + ~intent:(W.Mutation_intent intent) + ~presentation + |> Result.get_ok + in + let receipt = + match value (call (W.Mutate intent)) with + | W.Submission_committed receipt -> receipt + | _ -> failwith "real Child lost durable commit" + in + proofs := (intent, receipt) :: !proofs; + Child.receive + state + ~index + ~round + (W.Confirmed (value (call (W.Confirm (presentation, receipt))))) + | Read_child round -> + Child.read_child + state + ~round + (value + (call (W.Query (budget, W.Block_summary { block = child; after = None })))) + | Read_parent (round, summary) -> + ignore + (value + (call + (W.Query + ( budget + , W.Outline_block + { uuid = Test_support.authoritative_block_uuid; revision = None } + )))); + let block, revision, page = + match summary.lookup, summary.page with + | ( Worker_pure_data.V2_present_block { value; revision } + , Some (Worker_pure_data.V2_present_page { page; _ }) ) -> + value.block, revision, page + | _ -> failwith "real Child canonical DTO absent" + in + applied + := Some + (Journal_graph_projection.block_on_page + ~page + ~revision + ~child_count:(List.length summary.items) + ~time_context:{ localtime = Unix.gmtime } + block + |> Result.get_ok); + Child.projection_ready state ~round + | Present round -> + require + (Option.exists + (fun block -> Journal_model.source block = command.source) + !applied) + "Child ACK preceded production projection application"; + Child.applied state ~round presentation + | Acknowledge (index, round, intent, _) -> + require (!applied <> None) "actual Child intent acknowledged without a consumer"; + ignore (value (call (W.Acknowledge_intent intent))); + incr acks; + require + (Rrbvec.is_empty (recover ()).items) + "settled Child prefix retained recovery obligation"; + Child.acknowledged state ~index ~round (Ok ()) + | Complete -> state, Complete + | Wait message -> failwith ("actual Child stalled: " ^ message) + | Ignore -> failwith "actual Child dropped its current continuation" + in + match action with + | Child.Complete -> next + | _ -> drive (fuel - 1) next action + in + ignore (drive 24 initial action); + require + (!writes = 2 && !acks = 2) + "real Child bypassed Insert/Status or per-stage ACK"; + let final = + value (call (W.Query (budget, W.Outline_block { uuid = child; revision = None }))) + in + require + (match final with + | Worker_pure_data.V2_present_block { value; _ } -> + value.block.title = command.source && value.task_status = Some status + | _ -> false) + "real Child chain did not preserve source/status"; + List.iter + (fun (intent, receipt) -> + require + (match value (call (W.Mutate intent)) with + | W.Submission_committed original -> + W.receipt_revisions original = W.receipt_revisions receipt + | _ -> false) + "Child ACK removed original commit proof") + !proofs; + require + (value (call (W.Query (budget, W.Outline_block { uuid = child; revision = None }))) + = final) + "Child original-intent reconciliation wrote a duplicate"; + let revision = + match read () with + | Worker_pure_data.V2_present_block { revision; _ } -> revision + | _ -> assert false + in + let deletion = + W.mutation_intent + session + ~mutation_id:(uuid "9f000000-0000-4000-8000-000000001002") + (W.Delete_blocks + { root = Test_support.authoritative_block_uuid + ; preconditions = + { blocks = [ Test_support.authoritative_block_uuid, revision ] + ; pages = [] + ; scopes = [] + } + }) + |> Result.get_ok + in + let deleted_receipt = + match value (call (W.Mutate deletion)) with + | W.Submission_committed receipt -> receipt + | _ -> failwith "real typed deletion failed" + in + let deleted_confirmation = value (call (W.Confirm (presentation, deleted_receipt))) in + require + (match W.confirmation_primary deleted_confirmation with + | W.Confirmed_absence { uuid; _ } -> + W.Uuid.equal uuid Test_support.authoritative_block_uuid + | _ -> false) + "typed deletion did not deliver exact canonical absence"; + ignore (value (call (W.Acknowledge_intent (W.Mutation_intent deletion)))); + require + (Rrbvec.is_empty (recover ()).items) + "real deletion ACK retained pending recovery entry"; + require + (match value (call (W.Mutate deletion)) with + | W.Submission_committed original -> + W.receipt_revisions original = W.receipt_revisions deleted_receipt + | _ -> false) + "deletion reconciliation lost its original idempotent proof"; + require + (match read () with + | Worker_pure_data.V2_missing_block { uuid; _ } -> + W.Uuid.equal uuid Test_support.authoritative_block_uuid + | _ -> false) + "same-intent deletion reconciliation changed canonical absence"; + ignore (Mailbox.send client (Service.Client_command Return_to_graph_picker)); + wait (fun () -> + Option.exists + (fun projection -> W.selected projection = None) + (C.projection (Host.view host))); + ignore + (Mailbox.send + client + (Service.Client_command (Select_graph Test_support.graph_uuid))); + wait (fun () -> + Option.exists + (fun projection -> + match W.selected projection with + | Some current -> + let fresh = W.presentation_session current in + (not (W.equal_session fresh session)) + && List.exists + (fun resident -> + resident.W.phase = W.Open && W.equal_session resident.session fresh) + (W.residents projection) + | None -> false) + (C.projection (Host.view host))); + let current = Option.get (Option.bind (C.projection (Host.view host)) W.selected) in + let fresh = W.presentation_session current in + Rrbvec.iter + (fun (bytes, original, root_id, summary) -> + let append = + match W.restore_durable_intent fresh bytes |> Result.get_ok with + | W.Append_intent intent -> intent + | Mutation_intent _ -> failwith "recovered Append lost its family" + in + let receipt = + match value (call_in fresh (W.Append append)) with + | W.Submission_committed receipt -> receipt + | _ -> failwith "reopened Append did not reconcile" + in + require + (W.receipt_revisions receipt = W.receipt_revisions original) + "reopened Append replaced original evidence with the new read generation"; + let confirmation = value (call_in fresh (W.Confirm (current, receipt))) in + require + (match W.confirmation_primary confirmation with + | W.Confirmed_block value -> + value.lookup = summary.W.lookup + && value.page = summary.page + && value.items = summary.items + | _ -> false) + "reopened Append changed the original complete source/status tree"; + require + (W.receipt_targets receipt = W.receipt_targets original + && Rrbvec.exists (W.Uuid.equal root_id) (W.receipt_targets receipt)) + "reopened Append changed stable targets"; + ignore (value (call_in fresh (W.Acknowledge_intent (W.receipt_intent receipt))))) + !completed_appends; ignore (Host.shutdown host); wait (fun () -> C.shutdown_state (Host.view host) = C.Stopped)) ;; -(* The pure Sync reducer already retains same-user generations. Registration - invalidation and coalesced SDK controls belong to the real Service boundary. *) -let test_restored_authentication_session mode () = - with_service_fixture (fun dependencies config -> - let client, startup = Mailbox.Private.prepare - ~runtime_epoch:(Id.Runtime.Epoch.of_int64 9200L) - ~worker_generation:(Id.Worker.Generation.of_int64 1L) - (Service.create ~dependencies) config in - let ready = Atomic.make false and pause = Atomic.make false - and paused = Atomic.make false in - let worker = Domain.spawn (fun () -> - Logseq_db_worker_lui.Journal_worker_eio_backend.run (fun environment -> - Eio.Switch.run (fun session_switch -> - Mailbox.Private.run_session startup ~environment ~session_switch - ~on_startup:(fun result -> require (Result.is_ok result) "auth Service startup"; Atomic.set ready true) - ~on_idle_wait:(fun () -> if Atomic.get pause then ( - Atomic.set paused true; - while Atomic.get pause do Thread.delay 0.001 done; - Atomic.set paused false)) - ~on_yield:(fun () -> ())))) in - Fun.protect - ~finally:(fun () -> Atomic.set pause false; Mailbox.Private.request_stop client; ignore(Domain.join worker)) - (fun () -> - let responses = Hashtbl.create 32 and baseline = ref None and revoked = ref [] in - let pump () = Mailbox.For_testing.drain_events client ~max_events:128 |> List.iter (function - | Mailbox.Response {request_id;outcome;_} -> - require (not(Hashtbl.mem responses request_id)) "auth request completed twice"; - Hashtbl.add responses request_id outcome - | Push {payload=Service.Coordination_updates updates;_} -> - List.iter (function - | Service.Coordination_snapshot value -> remember_baseline baseline value - | Coordination_revoked(id,_) -> revoked := id :: !revoked) updates - | _ -> ()) in - let wait stage predicate = - let deadline = Unix.gettimeofday () +. 6. in - while not(predicate()) && Unix.gettimeofday() id | _ -> failwith "auth request admission" in - let result id = wait "request completion" (fun () -> Hashtbl.mem responses id); Hashtbl.find responses id in - let call request = result(send request) in - let auth user_id = Service.Client_command(Reconcile_authenticated_user{user_id}) in - wait "ready" (fun () -> Atomic.get ready); - (* Register before restoration to require tracking the live baseline, - rather than only its original empty registration snapshot. *) - let registration = 9200L in - (match call(Service.Register_coordination registration) with - | Completed(Service.Coordination_registered value) -> remember_baseline baseline value - | _ -> failwith "auth registration"); - ignore(call(Service.Client_command(Restore_local_account{user_id="user-1"}))); - wait "restored graph" (fun () -> Option.exists (fun value -> - List.exists(fun(_,state)->state.Logseq_db_worker.phase=Graph_open) - value.Service.baseline.state.residents) !baseline); - let old = Option.get !baseline in - let owner = Option.get old.baseline.state.sync.selected in - let source = observation_read client !baseline owner registration Read.Graph_info in - let id n=uuid(Printf.sprintf "9c000000-0000-4000-8000-%012d" (9200+n)) in - let query n=Service.Coordination_query{registration;owner;request_id=id n;read=source} in - let observe=Service.Coordination_observe{registration;owner;observation_id=id 10;request_id=id 11;read=source} in - let successful = function - | Mailbox.Completed(Service.Coordination_queried{answer=Read.Query_answer(_,Read.Graph_info,Ok info);_}) -> - info.graph_uuid=Test_support.graph_uuid - | _ -> false in - require(successful(call(query 1))) "restored session cannot read"; - require(match call observe with Completed(Service.Coordination_observed{epoch=0L;_})->true|_->false) - "restored observation not established"; - if mode="same-online" then - ignore(call(Service.Client_command(Request_sign_in owner.startup))); - if mode="verified-aba" then ignore(call(auth(Some "user-1"))); - Atomic.set pause true; - ignore(send Service.Get_graph_state); - wait "scheduler paused" (fun () -> Atomic.get paused); - let pending=send(query 2) in - let keys=match mode with - | "same" | "same-online" | "full-same" -> [Some "user-1"] - | "duplicate" -> [Some "user-1";Some "user-1"] - | "changed" -> [Some "other-user"] - | "signout" | "full-signout" | "cancelled-signout" -> [None] - | "first-aba" | "verified-aba" -> [Some "other-user";Some "user-1"] - | "first-signout-a" -> [None;Some "user-1"] - | _ -> failwith "unknown auth session case" in - if String.starts_with ~prefix:"full-" mode then ( - let full=ref false in - for _=1 to 128 do if not !full then - match Mailbox.send client Service.Get_graph_state with - | Full -> full:=true | Accepted _ -> () | _ -> failwith "queue stopped" done; - require !full "auth fixture did not fill response reservations"); - let confirmations=List.filter_map(fun user -> match Mailbox.send client(auth user) with - | Accepted id -> - if mode="cancelled-signout" then Mailbox.cancel client ~request_id:id; - Some id - | Full when String.starts_with ~prefix:"full-" mode -> None - | _ -> failwith "auth control admission") keys in - Atomic.set pause false; - let outcome=result pending in - List.iter(fun id -> ignore(result id)) confirmations; - (* Data completion is after the pending Latest control even on Full. *) - let preserves=List.mem mode ["same";"same-online";"duplicate";"full-same"] in - if preserves then ( - require(successful outcome) "same-user confirmation revoked an outstanding read"; - require(not(List.mem registration !revoked)) "same-user confirmation published revocation"; - (match call(Service.Register_coordination registration) with - | Completed(Service.Coordination_registered current) -> - require(current.baseline.state.sync.account=old.baseline.state.sync.account) - "same-user confirmation replaced account generations"; - require(current.baseline.state.sync.selected=old.baseline.state.sync.selected) - "same-user confirmation replaced graph/presentation ownership" - | _ -> failwith "same-user confirmation retired the original registration"); - require(successful(call(query 3))) "original session cannot read after confirmation"; - require(match call observe with Completed(Service.Coordination_observed{epoch=0L;_})->true|_->false) - "same-user confirmation retired the original observation"; - (* A genuine new graph lifetime still rejects the old capability. *) - ignore(call(Service.Client_command Return_to_graph_picker)); - wait "graph closed" (fun () -> Option.exists(fun value->value.Service.baseline.state.sync.selected=None)!baseline); - require(not(successful(call(query 4)))) "same-user exemption revived a closed graph"; - ignore(call(Service.Client_command(Select_graph Test_support.graph_uuid))); - wait "graph reopened" (fun () -> Option.exists(fun value -> - List.exists(fun((scope:Sync.graph_scope),state)->scope.graph_generation<>owner.graph_generation - && state.Logseq_db_worker.phase=Graph_open)value.Service.baseline.state.residents)!baseline); - require(not(successful(call(query 5)))) "same-user exemption crossed a new graph lifetime") - else ( - require(not(successful outcome)) "changed authority completed an old outstanding read"; - wait "binding revoked" (fun () -> List.mem registration !revoked); - require(not(successful(call(query 3)))) "changed authority revived old query admission"; - require(match call(Service.Register_coordination registration) with Failed _->true|_->false) - "retired registration became live again"))) -;; - -let test_restored_authentication_sessions () = - List.iter (fun mode -> test_restored_authentication_session mode ()) - ["same";"same-online";"duplicate";"changed";"signout";"first-aba";"first-signout-a"; - "verified-aba";"full-same";"full-signout";"cancelled-signout"]; - List.iter (fun change -> test_read_completion_rechecks_graph_owner ~change ()) - [`Same;`Changed;`Signout] -;; - -let test_changefeed_business_waits_for_ui_application () = +let test_host_preserves_query_origin_before_forget () = let module Host = Journal_worker_coordination_host in - let module Business = Journal_graph_runtime in - with_client ~statuses:true (fun client -> - let jobs = Queue.create () in - let process_notice=ref(fun (_:W.changefeed_notice)->()) in - let previous_notice_view=ref None in + with_client (fun client -> + let queue = Queue.create () in let host = Host.create ~client ~client_id ~managed_origin:"https://api.logseq.io" ~user_id:"user-1" - ~enqueue:(fun f -> Queue.add f jobs) - ~changed:(fun view -> - let notices=C.changefeed_updates ~previous:!previous_notice_view view in - previous_notice_view:=Some view; - Rrbvec.iter (fun notice-> !process_notice notice) notices) + ~enqueue:(fun f -> Queue.add f queue) + ~changed:(fun _ -> ()) |> Result.get_ok in - let acks = ref 0 and acknowledged_batches=Hashtbl.create 16 in - Mailbox.on_event client (fun event -> - let delivery = Host.on_event host event in - match event, delivery.R.disposition with - | ( Mailbox.Response - { outcome = - Completed - (Service.Coordination_changes_acknowledged{result=Ok value;_}) - ; _ - } - , R.Consumed ) -> incr acks;Hashtbl.replace acknowledged_batches (W.change_ack_through value) () - | _ -> ()); + Mailbox.on_event client (fun event -> ignore (Host.on_event host event)); let pump () = Mailbox.Private.deliver client ~max_events:64; - let count = Queue.length jobs in + let count = Queue.length queue in for _ = 1 to count do - (Queue.take jobs) () + (Queue.take queue) () done in let wait predicate = - let deadline = Unix.gettimeofday () +. 6. in + let deadline = Unix.gettimeofday () +. 4. in while (not (predicate ())) && Unix.gettimeofday () < deadline do pump (); Thread.delay 0.001 done; - require (predicate ()) "typed Changes business/client did not settle" - in - ignore - (Mailbox.send - client - (Service.Client_command (Restore_local_account { user_id = "user-1" }))); - let session () = - Option.bind - (C.projection (Host.view host)) - (fun p -> - List.find_map - (fun r -> if r.W.phase = W.Open then Some r.session else None) - (W.residents p)) + require (predicate ()) "host provenance integration did not settle" in - wait (fun () -> Option.is_some (session ())); - let session = Option.get (session ()) in + wait (fun () -> + match C.bootstrap (Host.view host) with + | W.Bootstrap_not_ready _ -> true + | _ -> false); let owner : Service.timeline_scope = { startup = { account_generation = 1; presentation_generation = 1 } ; graph_id = Test_support.graph_uuid ; graph_generation = 1 } in - let business = - Business.create - ~validate_session:(Host.validate_session ~owner host) - ~on_change:(Host.post host) - () - in - let calendar = - Journal_calendar.Sampler.create - ~clock:(fun () -> 1788307200.) - ~localtime:Unix.gmtime - () + let ticket = + Host.plan_query host ~owner ~maximum_items:1 ~maximum_bytes:65536 W.Graph_info + |> Result.get_ok in - Business.set_calendar - business - (Journal_calendar.Sampler.sample calendar |> Result.get_ok); - let ui_title = ref None - and detail_loaded = ref false - and held = ref None - and hold_once = ref true in - let expected_failure = ref false - and fail_budget_once = ref false - and saw_failure = ref false in - let more_seen = ref false in - let active = ref false - and notices = ref 0 - and generation = ref 1000L - and batch : W.change_batch option ref = ref None in - let latest_blocks=Hashtbl.create 4 and delete_conflict=ref None and deleted=ref false in - let rec ui responses = - List.iter - (fun (response : Business.response) -> - match response.payload with - | Business.Mutation_settled(intent,_,payload) -> - ui [{Business.payload}]; - let ticket=Host.request host session(W.Acknowledge_intent intent) |> Result.get_ok in - Host.watch host ticket ~f:(function - | W.Answered(Ok())->Business.acknowledge_mutation business intent - | _->failwith "actual business mutation ACK failed") - | Business.Detail_loaded { detail; _ } -> - Hashtbl.replace latest_blocks (Journal_model.id detail.root) detail.root; - detail_loaded := true; - ui_title := Some (Journal_model.source detail.root) - | Children_reconciled detail -> - Hashtbl.replace latest_blocks (Journal_model.id detail.root) detail.root; - ui_title := Some (Journal_model.source detail.root) - | Block_updated { block; _ } -> - Hashtbl.replace latest_blocks (Journal_model.id block) block; - ui_title := Some (Journal_model.source block) - | Delete_conflict{block;_}->delete_conflict:=Some block - | Subtree_deleted _->deleted:=true - | Rejected _ | Feed_failed _ | Open_failed _ - | Task_statuses_changed { error = Some _; _ } -> - if !expected_failure - then saw_failure := true - else failwith "typed business change application rejected valid data" - | _ -> ()) - responses + let callback = ref None + and calls = ref 0 in + Host.watch_read host ticket ~f:(fun origin reply -> + incr calls; + require + (C.query_origin (Host.view host) ticket = None) + "business callback ran before ticket Forget"; + callback := Some (origin, reply)); + ignore + (Mailbox.send + client + (Service.Client_command (Restore_local_account { user_id = "user-1" }))); + wait (fun () -> Option.is_some !callback); + let session, budget = + match Option.get !callback with + | Some origin, W.Answered (Ok info) -> + require + (W.Uuid.equal info.graph_uuid Test_support.graph_uuid) + "real host query lost graph"; + origin + | _ -> failwith "real host callback lost exact execution origin" in - let rec deliver output = - List.iter send output.Business.requests; - ui output.responses - and send request = - let application = - Business.application_for_request business ~request_id:request - in - match Business.worker_read business request with - | Some (Business.Read read) -> - let maximum_bytes = - if !fail_budget_once && application <> None - then ( - fail_budget_once := false; - 1) - else 65536 - in - let maximum_items = - match read.query with - | W.Changes _ -> 1 - | _ -> read.maximum_items - in - let ticket = - Host.plan_query host ~owner ~maximum_items ~maximum_bytes read.query - |> Result.get_ok - in - Host.watch_read host ticket ~f:(fun origin reply -> - match reply with - | W.Answered (Ok value) -> - finished request application origin (Journal_graph_runtime.Query_result(read.query,value)) - | W.Answered (Error (W.Stale_read _)) -> - finished request application origin Business.Changes_retired - | W.Answered (Error W.Response_budget_exceeded) when !expected_failure -> - finished - request - application - origin - (Business.Read_failed - { code = Logseq_db_worker.Error.Response_too_large - ; message = "Actual encoded read budget failure" }) - | _ -> failwith "real typed Changes business query failed") - | Some (Business.Acknowledge read) -> - let ticket = - Host.request - host - (W.change_batch_session read.batch) - (W.Acknowledge_changes read.batch) - |> Result.get_ok - in - Host.watch_read host ticket ~f:(fun origin reply -> - match reply with - | W.Answered (Ok value) -> finished request None origin (Business.Changes_acknowledged value) - | _ -> failwith "real typed Changes business ACK failed") - | Some (Business.Confirm_readback read) -> - (match Host.reconcile_mutation host ~presentation:read.presentation read.intent |> Result.get_ok with - |C.Submission_ticket _->failwith "business readback unexpectedly resubmitted its committed intent" - |C.Confirmation_ticket ticket-> - Host.watch_read host ticket ~f:(fun origin->function - |W.Answered(Ok confirmation)-> - (match W.confirmation_primary confirmation with - |W.Confirmed_block summary->finished request application origin(Business.Query_result(read.query,summary)) - |_->failwith "business confirmation target missing") - |_->failwith "business canonical read failed")) - | None -> - let presentation=Option.get(Option.bind(C.projection(Host.view host))W.selected) in - let intent=Business.worker_intent business request session |> Result.get_ok in - require(Result.is_ok (Business.bind_mutation business request ~presentation ~intent)) "business intent did not bind"; - let rec watch = function - |C.Submission_ticket ticket->Host.watch host ticket ~f:(function - |W.Answered(Ok(W.Submission_committed _))-> - watch(Host.reconcile_mutation host ~presentation intent |> Result.get_ok) - |W.Answered(Ok(W.Submission_rejected error))-> - deliver(Business.receive_mutation business request(W.Rejected error)) - |W.Answered(Ok W.Submission_unknown)-> - deliver(Business.receive_mutation business request W.Outcome_unknown) - |_->failwith "actual business mutation did not settle") - |C.Confirmation_ticket ticket->Host.watch host ticket ~f:(function - |W.Answered(Ok confirmation)->deliver(Business.receive_mutation business request(W.Confirmed confirmation)) - |_->failwith "actual business mutation did not confirm") in - watch(Host.reconcile_mutation host ~presentation intent |> Result.get_ok) - and finished request application origin result = - let application, discard = - match result with - | Business.Query_result(W.Changes _,value) -> - if value.next <> None then more_seen := true; - let view = Host.view host in - if - C.change_application_started view value - && C.change_application_error view value = None - then None, true - else ( - let token = - Host.begin_change_application host value ~root:request - |> Result.get_ok - in - require - (Business.set_change_application - business - ~request_id:request - token) - "business application root was not owned"; - batch := Some value; - Some token, false) - | _ -> application, false - in - if discard - then Business.abandon business request - else ( - let child = - match Business.worker_read business request with - | Some(Business.Read{query=W.Changes _;_}) | Some(Business.Acknowledge _) -> false - | _ -> true - in - let output = - Business.receive_worker_read - ?origin - business - ~request_id:request - result - in - List.iter send output.requests; - let apply () = - ui output.responses; - Option.iter - (fun token -> - Host.post - host - (C.Change_consumer_applied (token, request, Ok ()))) - application - in + require + (W.validate_request session (W.Query (budget, W.Graph_info)) = Ok ()) + "forwarded origin budget is unrelated to Send"; + require (Host.validate_session host session = Ok ()) "live read origin was rejected"; + ignore (Host.shutdown host); + wait (fun () -> C.shutdown_state (Host.view host) = C.Stopped); + require + (Host.validate_session host session = Error W.Revoked_session) + "stopped host allowed cached read"; + require (!calls = 1) "query origin callback was delivered more than once") +;; - if !active && !hold_once && child && application <> None - then ( - hold_once := false; - held := Some apply) - else apply ()) +let test_visible_media_listing_uses_typed_reads () = + let module Host = Journal_worker_coordination_host in + let module Media = Journal_media_runtime in + with_client ~favorites:true ~assets:true (fun client -> + let jobs = Queue.create () in + let host = + Host.create + ~client + ~client_id + ~managed_origin:"https://api.logseq.io" + ~user_id:"user-1" + ~enqueue:(fun f -> Queue.add f jobs) + ~changed:(fun _ -> ()) + |> Result.get_ok in - send (Business.start business); - deliver - (Business.submit - business - (Load_detail - { block_id = W.Uuid.to_string Test_support.authoritative_block_uuid - ; after = None - ; limit = 1 - ; request_generation = 900L - })); - wait (fun () -> !detail_loaded); - active := true; - (process_notice - := fun notice -> - if Result.is_ok(Host.validate_session host ~owner notice.W.notice_session) then ( - incr notices; - generation := Int64.succ !generation; - deliver (Business.reconcile_signal business ~request_generation:!generation notice.notice))); - let write n title = - let revision = ref None in - let budget = - W.read_budget session ~maximum_items:1 ~maximum_bytes:65536 |> Result.get_ok - in + Mailbox.on_event client (fun event -> ignore (Host.on_event host event)); + let pump () = + Mailbox.Private.deliver client ~max_events:64; + let count = Queue.length jobs in + for _ = 1 to count do + (Queue.take jobs) () + done + in + let wait predicate = + let deadline = Unix.gettimeofday () +. 6. in + while (not (predicate ())) && Unix.gettimeofday () < deadline do + pump (); + Thread.delay 0.001 + done; + require (predicate ()) "typed visible media listing did not settle" + in + ignore + (Mailbox.send + client + (Service.Client_command (Restore_local_account { user_id = "user-1" }))); + wait (fun () -> + Option.exists + (fun p -> List.exists (fun r -> r.W.phase = W.Open) (W.residents p)) + (C.projection (Host.view host))); + let owner : Service.timeline_scope = + { startup = { account_generation = 1; presentation_generation = 1 } + ; graph_id = Test_support.graph_uuid + ; graph_generation = 1 + } + in + let runtime = ref None + and typed_reads = ref 0 + and seen = ref [] + and finished = ref false + and last = ref None + and held = ref None + and retiring = ref false + and late_returned = ref false in + let root = W.Uuid.to_string (journal_page 1) in + let read (Media.Read request) = let ticket = - Host.request + Host.plan_query_when_available host - session - (W.Query - ( budget - , W.Outline_block - { uuid = Test_support.authoritative_block_uuid; revision = None } )) + ~owner + ~maximum_items:(min 1 request.maximum_items) + ~maximum_bytes:65536 + request.query |> Result.get_ok in - Host.watch host ticket ~f:(function - | W.Answered (Ok (Worker_pure_data.V2_present_block value)) -> revision := Some value.revision - | _ -> failwith "write revision query failed"); - wait (fun () -> !revision <> None); - let intent = - W.mutation_intent - session - ~mutation_id:(uuid (Printf.sprintf "9d000000-0000-4000-8000-%012d" (7000 + n))) - (W.Save_block - { block = Test_support.authoritative_block_uuid - ; title - ; preconditions = - { blocks = - [ Test_support.authoritative_block_uuid, Option.get !revision ] - ; pages = [] - ; scopes = [] - } - }) - |> Result.get_ok - in - let done_ = ref false in - let ticket = Host.request host session (W.Mutate intent) |> Result.get_ok in - Host.watch host ticket ~f:(function - | W.Answered (Ok (W.Submission_committed _)) -> done_ := true - | _ -> failwith "business window write failed"); - wait (fun () -> !done_) + (match request.query with + | W.Assets _ -> incr typed_reads + | _ -> ()); + let number = !typed_reads in + Host.watch_read host ticket ~f:(fun origin reply -> + require + (Option.exists + (fun (session, _) -> Host.validate_session ~owner host session = Ok ()) + origin) + "media listing borrowed its response origin"; + match reply with + | W.Answered (Ok value) -> + let result = request.outcome value in + (match request.query with + | W.Assets _ -> + last := Some result; + if !retiring + then ( + require + (not (Media.receive_read (Option.get !runtime) result)) + "in-flight result revived reset scope"; + late_returned := true) + else if number = 1 + then held := Some result + else ( + require + (Media.receive_read (Option.get !runtime) result) + "owned typed media page was lost"; + require + (not (Media.receive_read (Option.get !runtime) result)) + "media page consumed twice"; + if Option.is_some value.next + then Queue.add (fun () -> Media.next (Option.get !runtime) ~root) jobs + else finished := true) + | _ -> ignore (Media.receive_read (Option.get !runtime) result)) + | _ -> failwith "real typed media listing failed"); + true in - write 1 "Business change A"; - wait (fun () -> !held <> None); - require (!acks = 0) "real ACK preceded the held UI application"; + let media = + Media.create + ~read + ~demand: + (host_asset_demand host ~owner ~receive:(fun ticket demand -> + ignore (Media.receive_demand (Option.get !runtime) ticket demand))) + ~release_consumer:(host_asset_release host) + ~acquire:(fun _ _ -> + failwith "metadata listing unexpectedly acquired a rendered file") + ~release_file:(fun _ -> + failwith "metadata listing unexpectedly released a rendered file") + ~retry_asset:(fun ~graph_generation:_ _ -> + failwith "metadata listing unexpectedly retried a network asset") + ~changed:(fun _ (view : Media.view) -> + require (view.error = None) "typed media projection failed"; + seen + := List.rev_append + (List.map (fun (item : Media.item) -> item.asset.uuid) view.items) + !seen) + () + in + runtime := Some media; + Media.reset media ~graph_generation:(Some 1); + Media.root_visible media ~root true; + wait (fun () -> Option.is_some !held); + Media.resync media; require - (not (C.change_application_ready (Host.view host) (Option.get !batch))) - "held UI consumer became ready"; - write 2 "Business change B"; - wait (fun () -> !notices >= 2); - require (!acks = 0) "a newer push ACKed an unapplied older consumer"; - write 3 "Business change C"; - wait (fun () -> !notices >= 3); - Queue.add (Option.get !held) jobs; - held := None; - wait (fun () -> !acks >= 3 && !ui_title = Some "Business change C"); - require !more_seen "real byte/item-bounded Changes did not exercise More"; + (Media.receive_read media (Option.get !held)) + "superseded media read lost its owning terminal"; require - (C.change_application_error (Host.view host) (Option.get !batch) = None) - "real applied consumers retained a failure"; - (* Original Capture/reconcile/status and stale-delete combinations, using - the production business reducer and the actual typed Service throughout. *) - let operation : type a. a W.request -> a = fun request-> - let ticket=Host.request host session request |> Result.get_ok in - let result=ref None in Host.watch host ticket ~f:(fun reply->result:=Some reply); - wait(fun()-> !result<>None); - match Option.get !result with W.Answered(Ok value)->value|_->failwith "combination fixture failed" in - let target=uuid "9d000000-0000-4000-8000-000000008000" in - let target_id=W.Uuid.to_string target in - let append=W.append_intent session ~mutation_id:(uuid "9d000000-0000-4000-8000-000000008001") - ~block_id:target ~day:(W.journal_day 20260902 |> Result.get_ok) ~title:"Captured regression block" - ~task_status:None ~children:Rrbvec.empty |> Result.get_ok in - let receipt=match operation(W.Append append)with W.Submission_committed value->value|_->failwith "capture seed failed" in - let presentation=Option.get(Option.bind(C.projection(Host.view host))W.selected)in - ignore(operation(W.Confirm(presentation,receipt))); - ignore(operation(W.Acknowledge_intent(W.Append_intent append))); - deliver(Business.submit business(Load_detail{block_id=target_id;after=None;limit=32;request_generation=910L})); - wait(fun()->Hashtbl.mem latest_blocks target_id); - let save_target n title= - let budget=W.read_budget session ~maximum_items:1 ~maximum_bytes:65536 |> Result.get_ok in - let revision=match operation(W.Query(budget,W.Outline_block{uuid=target;revision=None}))with - |Worker_pure_data.V2_present_block value->value.revision|_->failwith "captured target missing" in - let intent=W.mutation_intent session ~mutation_id:(uuid(Printf.sprintf "9d000000-0000-4000-8000-%012d" n)) - (W.Save_block{block=target;title;preconditions={blocks=[target,revision];pages=[];scopes=[]}}) |> Result.get_ok in - match operation(W.Mutate intent)with W.Submission_committed _->()|_->failwith "concurrent target save failed" in - save_target 8002 "Reconciled capture"; - wait(fun()->Journal_model.source(Hashtbl.find latest_blocks target_id)="Reconciled capture"); - let reconciled=Hashtbl.find latest_blocks target_id in - let actual=operation(W.Query((W.read_budget session ~maximum_items:1 ~maximum_bytes:65536 |> Result.get_ok), - W.Outline_block{uuid=target;revision=None}))in - require(match actual with Worker_pure_data.V2_present_block value->value.revision=Journal_model.revision reconciled|_->false) - "changefeed reconciliation emitted an obsolete target revision"; - write 5 "Unrelated target change"; - let status=Test_support.status_uuid 3 in - deliver(Business.submit business(Set_task_state{mutation_id="9d000000-0000-4000-8000-000000008003"; - block_id=target_id;expected_revision=Journal_model.revision reconciled;task_state=Journal_model.Status(W.Uuid.to_string status)})); - wait(fun()->Journal_model.task_state(Hashtbl.find latest_blocks target_id)=Journal_model.Status(W.Uuid.to_string status)); - let stale=Hashtbl.find latest_blocks target_id in - save_target 8004 "Concurrent edit before delete deadline"; - let deletion revision id=Business.submit business(Delete_subtree{mutation_id=id;block_id=target_id;expected_revision=revision})in - deliver(deletion(Journal_model.revision stale)"9d000000-0000-4000-8000-000000008005"); - wait(fun()-> !delete_conflict<>None); - let refreshed=Option.get !delete_conflict in - require(Journal_model.source refreshed="Concurrent edit before delete deadline") - "delete conflict retained stale canonical source"; - deliver(deletion(Journal_model.revision refreshed)"9d000000-0000-4000-8000-000000008006"); - wait(fun()-> !deleted); - deliver(Business.submit business(Load_detail{block_id=W.Uuid.to_string Test_support.authoritative_block_uuid; - after=None;limit=1;request_generation=911L})); - wait(fun()-> !ui_title=Some "Unrelated target change"); - expected_failure := true; - fail_budget_once := true; - write 4 "Business change D"; + (!typed_reads = 2) + "typed completion left the replacement listing queued without pumping"; + require + (not (Media.receive_read media (Option.get !held))) + "superseded media terminal consumed twice"; + wait (fun () -> !finished); + require (!typed_reads >= 3) "media fixture did not exercise sealed pagination"; + require + (List.sort_uniq W.Uuid.compare !seen + = List.init 3 (fun n -> + uuid (Printf.sprintf "9c000000-0000-4000-8000-%012d" (901 + n)))) + "visible media listing lost a real external descriptor"; + retiring := true; + Media.resync media; + Media.reset media ~graph_generation:(Some 2); + wait (fun () -> !late_returned); + require + (not (Media.receive_read media (Option.get !last))) + "old media result revived reset scope"; + ignore (Host.shutdown host); + wait (fun () -> C.shutdown_state (Host.view host) = C.Stopped)) +;; + +let test_offline_asset_scans_use_typed_reads () = + let module Host = Journal_worker_coordination_host in + let module Assets = Journal_asset_runtime in + with_client ~favorites:true ~assets:true (fun client -> + let jobs = Queue.create () in + let host = + Host.create + ~client + ~client_id + ~managed_origin:"https://api.logseq.io" + ~user_id:"user-1" + ~enqueue:(fun f -> Queue.add f jobs) + ~changed:(fun _ -> ()) + |> Result.get_ok + in + Mailbox.on_event client (fun event -> ignore (Host.on_event host event)); + let pump () = + Mailbox.Private.deliver client ~max_events:64; + let count = Queue.length jobs in + for _ = 1 to count do + (Queue.take jobs) () + done + in + let wait predicate = + let deadline = Unix.gettimeofday () +. 6. in + while (not (predicate ())) && Unix.gettimeofday () < deadline do + pump (); + Thread.delay 0.001 + done; + require (predicate ()) "typed offline asset scan did not settle" + in + ignore + (Mailbox.send + client + (Service.Client_command (Restore_local_account { user_id = "user-1" }))); wait (fun () -> - !saw_failure && Option.exists(fun value->C.change_application_error(Host.view host)value<>None)!batch); - let failed_batch=Option.get !batch in - require(not(Hashtbl.mem acknowledged_batches failed_batch.through)) - "an actual failed read acknowledged its unapplied window"; - active := false; - ui_title := None; - detail_loaded := false; - Business.reset business; - wait (fun () -> Hashtbl.mem acknowledged_batches failed_batch.through); + Option.exists + (fun p -> List.exists (fun r -> r.W.phase = W.Open) (W.residents p)) + (C.projection (Host.view host))); + let owner : Service.timeline_scope = + { startup = { account_generation = 1; presentation_generation = 1 } + ; graph_id = Test_support.graph_uuid + ; graph_generation = 1 + } + in + let runtime = ref None + and typed_reads = ref 0 + and status = ref None + and asset_continuations = ref 0 + and asset_rows = ref [] in + let read (Assets.Read request) = + (match request.query with + | W.Assets { after = Some _; _ } -> incr asset_continuations + | _ -> ()); + let ticket = + Host.plan_query_when_available + host + ~owner + ~maximum_items:(min 1 request.maximum_items) + ~maximum_bytes:65536 + request.query + |> Result.get_ok + in + incr typed_reads; + Host.watch_read host ticket ~f:(fun origin reply -> + require + (Option.exists + (fun (session, _) -> Host.validate_session ~owner host session = Ok ()) + origin) + "asset scan borrowed its response origin"; + match reply with + | W.Answered (Ok value) -> + (match request.query with + | W.Assets _ -> + asset_rows + := List.rev_append + (List.map + (fun (a : Logseq_db_types.Asset_descriptor.t) -> a.uuid) + value.items) + !asset_rows + | _ -> ()); + let result = request.outcome value in + require + (Assets.receive_read + (Option.get !runtime) + ~request_id:request.request_id + result) + "owned typed scan result was lost"; + require + (not + (Assets.receive_read + (Option.get !runtime) + ~request_id:request.request_id + result)) + "typed scan consumed the same result twice" + | _ -> failwith "real typed offline asset read failed"); + true + in + let assets = + Assets.create + ~read + ~demand: + (host_asset_demand host ~owner ~receive:(fun ticket demand -> + ignore (Assets.receive_demand (Option.get !runtime) ticket demand))) + ~release_consumer:(host_asset_release host) + ~changed:(fun _ recent favorites -> status := Some (recent, favorites)) + () + in + runtime := Some assets; + Assets.refresh + assets + ~graph_generation:1 + ~today:20260901 + ~settings:Journal_asset_policy.default_settings; + wait (fun () -> + match !status with + | Some (recent, favorites) -> + recent.Journal_asset_policy.enumeration = Complete + && favorites.enumeration = Complete + | _ -> false); require - (!ui_title = None) - "retired consumer callbacks revived the discarded business scope"; + (!typed_reads >= 3) + "offline scan did not execute real catalog and asset query families"; + require + (!asset_continuations >= 2) + (Printf.sprintf + "real asset More did not retain its sealed continuation (after=%d)" + !asset_continuations); + require + (List.sort_uniq W.Uuid.compare !asset_rows + = List.init 3 (fun n -> + uuid (Printf.sprintf "9c000000-0000-4000-8000-%012d" (901 + n)))) + "asset pagination lost a real external descriptor"; + Assets.shutdown assets; ignore (Host.shutdown host); wait (fun () -> C.shutdown_state (Host.view host) = C.Stopped)) ;; -let with_observation_service ?epoch_ms ?prepare ?change_max_items ?(observe_push=fun _->()) f = - with_client_output_wait ?epoch_ms ?prepare ?change_max_items (fun client ~wait:wait_output -> - let replies = Hashtbl.create 16 - and baseline = ref None - and notices = ref Rrbvec.empty in +let test_dependency_point_reads_use_typed_owner () = + let module Host = Journal_worker_coordination_host in + with_client (fun client -> + let jobs = Queue.create () in + let host = + Host.create + ~client + ~client_id + ~managed_origin:"https://api.logseq.io" + ~user_id:"user-1" + ~enqueue:(fun f -> Queue.add f jobs) + ~changed:(fun _ -> ()) + |> Result.get_ok + in + let consumed = ref 0 in Mailbox.on_event client (fun event -> - (match event with Mailbox.Push {payload;_}->observe_push payload|_->()); - match event with - | Response { request_id; outcome; _ } -> Hashtbl.replace replies request_id outcome - | Push { payload = Service.Coordination_updates updates; _ } -> - List.iter - (function - | Service.Coordination_snapshot value -> remember_baseline baseline value - | _ -> ()) - updates - | Push { payload = Service.Coordination_observations updates; _ } -> - notices := Rrbvec.append !notices updates - | _ -> ()); + if (Host.on_event host event).R.disposition = R.Consumed then incr consumed); + let pump () = + Mailbox.Private.deliver client ~max_events:64; + let count = Queue.length jobs in + for _ = 1 to count do + (Queue.take jobs) () + done + in let wait predicate = - wait_output ~pump:(fun () -> Mailbox.Private.deliver client ~max_events:64) - ~timeout:5. ~label:"real observation Service did not settle" predicate + let deadline = Unix.gettimeofday () +. 6. in + while (not (predicate ())) && Unix.gettimeofday () < deadline do + pump (); + Thread.delay 0.001 + done; + require (predicate ()) "typed dependency point query did not settle" in - let send value = - match Mailbox.send client value with - | Accepted id -> id - | _ -> failwith "observation Service request not admitted" + ignore + (Mailbox.send + client + (Service.Client_command (Restore_local_account { user_id = "user-1" }))); + wait (fun () -> + Option.exists + (fun p -> List.exists (fun r -> r.W.phase = W.Open) (W.residents p)) + (C.projection (Host.view host))); + let owner : Service.timeline_scope = + { startup = { account_generation = 1; presentation_generation = 1 } + ; graph_id = Test_support.graph_uuid + ; graph_generation = 1 + } in - let call value = - let id = send value in - wait (fun () -> Hashtbl.mem replies id); - Hashtbl.find replies id + let query : type a. int -> a W.query -> (a, W.error) result = + fun maximum_bytes query -> + let result = ref None in + let before = !consumed in + let ticket = + Host.plan_query_when_available host ~owner ~maximum_items:1 ~maximum_bytes query + |> Result.get_ok + in + Host.watch_read host ticket ~f:(fun origin reply -> result := Some (origin, reply)); + require + (!result = None) + "point callback ran synchronously before business admission"; + wait (fun () -> !result <> None); + require + (!consumed = before + 1) + "dependency point response was not consumed exactly once"; + let origin, reply = Option.get !result in + require + (Option.exists + (fun (session, _) -> Host.validate_session ~owner host session = Ok ()) + origin) + "point dependency inferred its origin after delivery"; + match reply with + | W.Answered result -> result + | _ -> failwith "dependency read was not answered" in - ignore (call (Service.Client_command (Restore_local_account { user_id = "user-1" }))); - let registration = 910000L in - (match call (Service.Register_coordination registration) with - | Completed (Service.Coordination_registered value) -> remember_baseline baseline value - | _ -> failwith "observation baseline registration failed"); - wait (fun () -> - match !baseline with - | Some value -> - List.exists - (fun (_, graph) -> graph.Logseq_db_worker.phase = Logseq_db_worker.Graph_open) - value.baseline.state.residents - | None -> false); - let owner = Option.get (Option.get !baseline).baseline.state.sync.selected in - f client ~call ~send ~wait ~replies ~notices ~baseline ~owner ~registration) + require + (match + query + 65536 + (W.Outline_block + { uuid = Test_support.authoritative_block_uuid; revision = None }) + with + | Ok (Worker_pure_data.V2_present_block { value; revision }) -> + value.block.uuid = Test_support.authoritative_block_uuid && revision <> "" + | _ -> false) + "typed dependency lost full block identity or revision"; + require + (match query 65536 (W.Page { uuid = Test_support.page_uuid; revision = None }) with + | Ok (Worker_pure_data.V2_present_page { page; revision }) -> + page.uuid = Test_support.page_uuid && revision <> "" + | _ -> false) + "typed dependency lost its full page DTO"; + require + (match + query + 64 + (W.Outline_block + { uuid = Test_support.authoritative_block_uuid; revision = None }) + with + | Error W.Response_budget_exceeded -> true + | _ -> false) + "point read bypassed encoded budget validation"; + ignore (Host.shutdown host); + wait (fun () -> C.shutdown_state (Host.view host) = C.Stopped)) ;; -let prepare_enriched_read_data ~count ~title support = - let module Storage = Logseq_db_storage.Logseq_sqlite_storage in - let module Session = Logseq_db_storage.Storage_session in - let path = - Filename.concat - support - ("logseq-db-worker/synced-graphs/" - ^ W.Uuid.to_string Test_support.graph_uuid - ^ "/db.sqlite") - in - let connection = Storage.open_database path |> Result.get_ok in - let storage = Storage.datascript_storage connection in - let database = Storage.restore_database connection |> Result.get_ok in - let session = - Session.create - ~tail:(Datascript.Storage.restore_tail_groups storage) - ~callbacks:(Storage.connection_callbacks connection) - in - Fun.protect - ~finally:(fun () -> ignore (Session.close session)) - (fun () -> - let open Datascript in - let tag = Temp_id "budget-tag" in - let page = - Lookup_ref ("block/uuid", Uuid (W.Uuid.to_string Test_support.page_uuid)) +(* The pure Sync reducer already retains same-user generations. Registration + invalidation and coalesced SDK controls belong to the real Service boundary. *) +let test_restored_authentication_session mode () = + with_service_fixture (fun dependencies config -> + let client, startup = + Mailbox.Private.prepare + ~runtime_epoch:(Id.Runtime.Epoch.of_int64 9200L) + ~worker_generation:(Id.Worker.Generation.of_int64 1L) + (Service.create ~dependencies) + config + in + let ready = Atomic.make false + and pause = Atomic.make false + and paused = Atomic.make false in + let worker = + Domain.spawn (fun () -> + Logseq_db_worker_lui.Journal_worker_eio_backend.run (fun environment -> + Eio.Switch.run (fun session_switch -> + Mailbox.Private.run_session + startup + ~environment + ~session_switch + ~on_startup:(fun result -> + require (Result.is_ok result) "auth Service startup"; + Atomic.set ready true) + ~on_idle_wait:(fun () -> + if Atomic.get pause + then ( + Atomic.set paused true; + while Atomic.get pause do + Thread.delay 0.001 + done; + Atomic.set paused false)) + ~on_yield:(fun () -> ())))) + in + Fun.protect + ~finally:(fun () -> + Atomic.set pause false; + Mailbox.Private.request_stop client; + ignore (Domain.join worker)) + (fun () -> + let responses = Hashtbl.create 32 + and baseline = ref None + and revoked = ref [] in + let pump () = + Mailbox.For_testing.drain_events client ~max_events:128 + |> List.iter (function + | Mailbox.Response { request_id; outcome; _ } -> + require + (not (Hashtbl.mem responses request_id)) + "auth request completed twice"; + Hashtbl.add responses request_id outcome + | Push { payload = Service.Coordination_updates updates; _ } -> + List.iter + (function + | Service.Coordination_snapshot value -> + remember_baseline baseline value + | Coordination_revoked (id, _) -> revoked := id :: !revoked) + updates + | _ -> ()) + in + let wait stage predicate = + let deadline = Unix.gettimeofday () +. 6. in + while (not (predicate ())) && Unix.gettimeofday () < deadline do + pump (); + Thread.delay 0.001 + done; + require (predicate ()) ("auth session " ^ mode ^ ": " ^ stage) + in + let send request = + match Mailbox.send client request with + | Accepted id -> id + | _ -> failwith "auth request admission" + in + let result id = + wait "request completion" (fun () -> Hashtbl.mem responses id); + Hashtbl.find responses id + in + let call request = result (send request) in + let auth user_id = + Service.Client_command (Reconcile_authenticated_user { user_id }) + in + wait "ready" (fun () -> Atomic.get ready); + (* Register before restoration to require tracking the live baseline, + rather than only its original empty registration snapshot. *) + let registration = 9200L in + (match call (Service.Register_coordination registration) with + | Completed (Service.Coordination_registered value) -> + remember_baseline baseline value + | _ -> failwith "auth registration"); + ignore + (call (Service.Client_command (Restore_local_account { user_id = "user-1" }))); + wait "restored graph" (fun () -> + Option.exists + (fun value -> + List.exists + (fun (_, state) -> state.Logseq_db_worker.phase = Graph_open) + value.Service.baseline.state.residents) + !baseline); + let old = Option.get !baseline in + let owner = Option.get old.baseline.state.sync.selected in + let source = + observation_read client !baseline owner registration Read.Graph_info + in + let id n = uuid (Printf.sprintf "9c000000-0000-4000-8000-%012d" (9200 + n)) in + let query n = + Service.Coordination_query + { registration; owner; request_id = id n; read = source } + in + let observe = + Service.Coordination_observe + { registration + ; owner + ; observation_id = id 10 + ; request_id = id 11 + ; read = source + } + in + let successful = function + | Mailbox.Completed + (Service.Coordination_queried + { answer = Read.Query_answer (_, Read.Graph_info, Ok info); _ }) -> + info.graph_uuid = Test_support.graph_uuid + | _ -> false + in + require (successful (call (query 1))) "restored session cannot read"; + require + (match call observe with + | Completed (Service.Coordination_observed { epoch = 0L; _ }) -> true + | _ -> false) + "restored observation not established"; + if mode = "same-online" + then ignore (call (Service.Client_command (Request_sign_in owner.startup))); + if mode = "verified-aba" then ignore (call (auth (Some "user-1"))); + Atomic.set pause true; + ignore (send Service.Get_graph_state); + wait "scheduler paused" (fun () -> Atomic.get paused); + let pending = send (query 2) in + let keys = + match mode with + | "same" | "same-online" | "full-same" -> [ Some "user-1" ] + | "duplicate" -> [ Some "user-1"; Some "user-1" ] + | "changed" -> [ Some "other-user" ] + | "signout" | "full-signout" | "cancelled-signout" -> [ None ] + | "first-aba" | "verified-aba" -> [ Some "other-user"; Some "user-1" ] + | "first-signout-a" -> [ None; Some "user-1" ] + | _ -> failwith "unknown auth session case" + in + if String.starts_with ~prefix:"full-" mode + then ( + let full = ref false in + for _ = 1 to 128 do + if not !full + then ( + match Mailbox.send client Service.Get_graph_state with + | Full -> full := true + | Accepted _ -> () + | _ -> failwith "queue stopped") + done; + require !full "auth fixture did not fill response reservations"); + let confirmations = + List.filter_map + (fun user -> + match Mailbox.send client (auth user) with + | Accepted id -> + if mode = "cancelled-signout" then Mailbox.cancel client ~request_id:id; + Some id + | Full when String.starts_with ~prefix:"full-" mode -> None + | _ -> failwith "auth control admission") + keys + in + Atomic.set pause false; + let outcome = result pending in + List.iter (fun id -> ignore (result id)) confirmations; + (* Data completion is after the pending Latest control even on Full. *) + let preserves = + List.mem mode [ "same"; "same-online"; "duplicate"; "full-same" ] + in + if preserves + then ( + require + (successful outcome) + "same-user confirmation revoked an outstanding read"; + require + (not (List.mem registration !revoked)) + "same-user confirmation published revocation"; + (match call (Service.Register_coordination registration) with + | Completed (Service.Coordination_registered current) -> + require + (current.baseline.state.sync.account = old.baseline.state.sync.account) + "same-user confirmation replaced account generations"; + require + (current.baseline.state.sync.selected = old.baseline.state.sync.selected) + "same-user confirmation replaced graph/presentation ownership" + | _ -> failwith "same-user confirmation retired the original registration"); + require + (successful (call (query 3))) + "original session cannot read after confirmation"; + require + (match call observe with + | Completed (Service.Coordination_observed { epoch = 0L; _ }) -> true + | _ -> false) + "same-user confirmation retired the original observation"; + (* A genuine new graph lifetime still rejects the old capability. *) + ignore (call (Service.Client_command Return_to_graph_picker)); + wait "graph closed" (fun () -> + Option.exists + (fun value -> value.Service.baseline.state.sync.selected = None) + !baseline); + require + (not (successful (call (query 4)))) + "same-user exemption revived a closed graph"; + ignore (call (Service.Client_command (Select_graph Test_support.graph_uuid))); + wait "graph reopened" (fun () -> + Option.exists + (fun value -> + List.exists + (fun ((scope : Sync.graph_scope), state) -> + scope.graph_generation <> owner.graph_generation + && state.Logseq_db_worker.phase = Graph_open) + value.Service.baseline.state.residents) + !baseline); + require + (not (successful (call (query 5)))) + "same-user exemption crossed a new graph lifetime") + else ( + require + (not (successful outcome)) + "changed authority completed an old outstanding read"; + wait "binding revoked" (fun () -> List.mem registration !revoked); + require + (not (successful (call (query 3)))) + "changed authority revived old query admission"; + require + (match call (Service.Register_coordination registration) with + | Failed _ -> true + | _ -> false) + "retired registration became live again"))) +;; + +let test_restored_authentication_sessions () = + List.iter + (fun mode -> test_restored_authentication_session mode ()) + [ "same" + ; "same-online" + ; "duplicate" + ; "changed" + ; "signout" + ; "first-aba" + ; "first-signout-a" + ; "verified-aba" + ; "full-same" + ; "full-signout" + ; "cancelled-signout" + ]; + List.iter + (fun change -> test_read_completion_rechecks_graph_owner ~change ()) + [ `Same; `Changed; `Signout ] +;; + +let test_changefeed_business_waits_for_ui_application () = + let module Host = Journal_worker_coordination_host in + let module Business = Journal_graph_runtime in + with_client ~statuses:true (fun client -> + let jobs = Queue.create () in + let process_notice = ref (fun (_ : W.changefeed_notice) -> ()) in + let previous_notice_view = ref None in + let host = + Host.create + ~client + ~client_id + ~managed_origin:"https://api.logseq.io" + ~user_id:"user-1" + ~enqueue:(fun f -> Queue.add f jobs) + ~changed:(fun view -> + let notices = C.changefeed_updates ~previous:!previous_notice_view view in + previous_notice_view := Some view; + Rrbvec.iter (fun notice -> !process_notice notice) notices) + |> Result.get_ok + in + let acks = ref 0 + and acknowledged_batches = Hashtbl.create 16 in + Mailbox.on_event client (fun event -> + let delivery = Host.on_event host event in + match event, delivery.R.disposition with + | ( Mailbox.Response + { outcome = + Completed + (Service.Coordination_changes_acknowledged { result = Ok value; _ }) + ; _ + } + , R.Consumed ) -> + incr acks; + Hashtbl.replace acknowledged_batches (W.change_ack_through value) () + | _ -> ()); + let pump () = + Mailbox.Private.deliver client ~max_events:64; + let count = Queue.length jobs in + for _ = 1 to count do + (Queue.take jobs) () + done + in + let wait predicate = + let deadline = Unix.gettimeofday () +. 6. in + while (not (predicate ())) && Unix.gettimeofday () < deadline do + pump (); + Thread.delay 0.001 + done; + require (predicate ()) "typed Changes business/client did not settle" + in + ignore + (Mailbox.send + client + (Service.Client_command (Restore_local_account { user_id = "user-1" }))); + let session () = + Option.bind + (C.projection (Host.view host)) + (fun p -> + List.find_map + (fun r -> if r.W.phase = W.Open then Some r.session else None) + (W.residents p)) + in + wait (fun () -> Option.is_some (session ())); + let session = Option.get (session ()) in + let owner : Service.timeline_scope = + { startup = { account_generation = 1; presentation_generation = 1 } + ; graph_id = Test_support.graph_uuid + ; graph_generation = 1 + } + in + let business = + Business.create + ~validate_session:(Host.validate_session ~owner host) + ~on_change:(Host.post host) + () + in + let calendar = + Journal_calendar.Sampler.create + ~clock:(fun () -> 1788307200.) + ~localtime:Unix.gmtime + () + in + Business.set_calendar + business + (Journal_calendar.Sampler.sample calendar |> Result.get_ok); + let ui_title = ref None + and detail_loaded = ref false + and held = ref None + and hold_once = ref true in + let expected_failure = ref false + and fail_budget_once = ref false + and saw_failure = ref false in + let more_seen = ref false in + let latest_notice = ref None in + let hold_change_root = ref false in + let held_change_root = ref None in + let accepted_change_roots = ref 0 in + let pending_change_roots = ref 0 in + let active = ref false + and notices = ref 0 + and generation = ref 1000L + and batch : W.change_batch option ref = ref None in + let latest_blocks = Hashtbl.create 4 + and delete_conflict = ref None + and deleted = ref false in + let rec ui responses = + List.iter + (fun (response : Business.response) -> + match response.payload with + | Business.Mutation_settled (intent, _, payload) -> + ui [ { Business.payload } ]; + let ticket = + Host.request host session (W.Acknowledge_intent intent) |> Result.get_ok + in + Host.watch host ticket ~f:(function + | W.Answered (Ok ()) -> Business.acknowledge_mutation business intent + | _ -> failwith "actual business mutation ACK failed") + | Business.Detail_loaded { detail; _ } -> + Hashtbl.replace latest_blocks (Journal_model.id detail.root) detail.root; + detail_loaded := true; + ui_title := Some (Journal_model.source detail.root) + | Children_reconciled detail -> + Hashtbl.replace latest_blocks (Journal_model.id detail.root) detail.root; + ui_title := Some (Journal_model.source detail.root) + | Block_updated { block; _ } -> + Hashtbl.replace latest_blocks (Journal_model.id block) block; + ui_title := Some (Journal_model.source block) + | Delete_conflict { block; _ } -> delete_conflict := Some block + | Subtree_deleted _ -> deleted := true + | Rejected _ | Feed_failed _ | Open_failed _ + | Task_statuses_changed { error = Some _; _ } -> + if !expected_failure + then saw_failure := true + else failwith "typed business change application rejected valid data" + | _ -> ()) + responses + in + let rec deliver output = + List.iter send output.Business.requests; + ui output.responses + and send request = + let application = Business.application_for_request business ~request_id:request in + match Business.worker_read business request with + | Some (Business.Read read) -> + let maximum_bytes = + if !fail_budget_once && application <> None + then ( + fail_budget_once := false; + 1) + else 65536 + in + let maximum_items = + match read.query with + | W.Changes _ -> 1 + | _ -> read.maximum_items + in + let ticket = + Host.plan_query host ~owner ~maximum_items ~maximum_bytes read.query + |> Result.get_ok + in + (match read.query with + | W.Changes _ -> incr pending_change_roots + | _ -> ()); + Host.watch_read host ticket ~f:(fun origin reply -> + match reply with + | W.Answered (Ok value) -> + let complete () = + finished + request + application + origin + (Journal_graph_runtime.Query_result (read.query, value)); + match read.query with + | W.Changes _ -> decr pending_change_roots + | _ -> () + in + (match read.query with + | W.Changes _ when !hold_change_root -> + hold_change_root := false; + held_change_root := Some complete + | _ -> complete ()) + | W.Answered (Error (W.Stale_read _)) -> + finished request application origin Business.Changes_retired; + (match read.query with + | W.Changes _ -> decr pending_change_roots + | _ -> ()) + | W.Answered (Error W.Response_budget_exceeded) when !expected_failure -> + (match read.query with + | W.Changes _ -> decr pending_change_roots + | _ -> ()); + finished + request + application + origin + (Business.Read_failed + { code = Logseq_db_worker.Error.Response_too_large + ; message = "Actual encoded read budget failure" + }) + | _ -> failwith "real typed Changes business query failed") + | Some (Business.Acknowledge read) -> + let ticket = + Host.request + host + (W.change_batch_session read.batch) + (W.Acknowledge_changes read.batch) + |> Result.get_ok + in + Host.watch_read host ticket ~f:(fun origin reply -> + match reply with + | W.Answered (Ok value) -> + finished request None origin (Business.Changes_acknowledged value) + | _ -> failwith "real typed Changes business ACK failed") + | Some (Business.Confirm_readback read) -> + (match + Host.reconcile_mutation host ~presentation:read.presentation read.intent + |> Result.get_ok + with + | C.Submission_ticket _ -> + failwith "business readback unexpectedly resubmitted its committed intent" + | C.Confirmation_ticket ticket -> + Host.watch_read host ticket ~f:(fun origin -> function + | W.Answered (Ok confirmation) -> + (match W.confirmation_primary confirmation with + | W.Confirmed_block summary -> + finished + request + application + origin + (Business.Query_result (read.query, summary)) + | _ -> failwith "business confirmation target missing") + | _ -> failwith "business canonical read failed")) + | None -> + let presentation = + Option.get (Option.bind (C.projection (Host.view host)) W.selected) + in + let intent = Business.worker_intent business request session |> Result.get_ok in + require + (Result.is_ok (Business.bind_mutation business request ~presentation ~intent)) + "business intent did not bind"; + let rec watch = function + | C.Submission_ticket ticket -> + Host.watch host ticket ~f:(function + | W.Answered (Ok (W.Submission_committed _)) -> + watch (Host.reconcile_mutation host ~presentation intent |> Result.get_ok) + | W.Answered (Ok (W.Submission_rejected error)) -> + deliver (Business.receive_mutation business request (W.Rejected error)) + | W.Answered (Ok W.Submission_unknown) -> + deliver (Business.receive_mutation business request W.Outcome_unknown) + | _ -> failwith "actual business mutation did not settle") + | C.Confirmation_ticket ticket -> + Host.watch host ticket ~f:(function + | W.Answered (Ok confirmation) -> + deliver + (Business.receive_mutation business request (W.Confirmed confirmation)) + | _ -> failwith "actual business mutation did not confirm") + in + watch (Host.reconcile_mutation host ~presentation intent |> Result.get_ok) + and finished request application origin result = + let application, discard = + match result with + | Business.Query_result (W.Changes _, value) -> + if value.next <> None then more_seen := true; + let view = Host.view host in + if + C.change_application_started view value + && C.change_application_error view value = None + then None, true + else ( + let token = + Host.begin_change_application host value ~root:request |> Result.get_ok + in + require + (Business.set_change_application business ~request_id:request token) + "business application root was not owned"; + incr accepted_change_roots; + batch := Some value; + Some token, false) + | _ -> application, false + in + if discard + then Business.abandon business request + else ( + let child = + match Business.worker_read business request with + | Some (Business.Read { query = W.Changes _; _ }) + | Some (Business.Acknowledge _) -> false + | _ -> true + in + let output = + Business.receive_worker_read ?origin business ~request_id:request result + in + List.iter send output.requests; + let apply () = + ui output.responses; + Option.iter + (fun token -> + Host.post host (C.Change_consumer_applied (token, request, Ok ()))) + application + in + if !active && !hold_once && child && application <> None + then ( + hold_once := false; + held := Some apply) + else apply ()) + in + send (Business.start business); + deliver + (Business.submit + business + (Load_detail + { block_id = W.Uuid.to_string Test_support.authoritative_block_uuid + ; after = None + ; limit = 1 + ; request_generation = 900L + })); + wait (fun () -> !detail_loaded); + active := true; + (process_notice + := fun notice -> + if Result.is_ok (Host.validate_session host ~owner notice.W.notice_session) + then ( + latest_notice := Some notice; + incr notices; + generation := Int64.succ !generation; + deliver + (Business.reconcile_signal + business + ~request_generation:!generation + notice.notice))); + let write n title = + let revision = ref None in + let budget = + W.read_budget session ~maximum_items:1 ~maximum_bytes:65536 |> Result.get_ok + in + let ticket = + Host.request + host + session + (W.Query + ( budget + , W.Outline_block + { uuid = Test_support.authoritative_block_uuid; revision = None } )) + |> Result.get_ok + in + Host.watch host ticket ~f:(function + | W.Answered (Ok (Worker_pure_data.V2_present_block value)) -> + revision := Some value.revision + | _ -> failwith "write revision query failed"); + wait (fun () -> !revision <> None); + let intent = + W.mutation_intent + session + ~mutation_id:(uuid (Printf.sprintf "9d000000-0000-4000-8000-%012d" (7000 + n))) + (W.Save_block + { block = Test_support.authoritative_block_uuid + ; title + ; preconditions = + { blocks = + [ Test_support.authoritative_block_uuid, Option.get !revision ] + ; pages = [] + ; scopes = [] + } + }) + |> Result.get_ok + in + let done_ = ref false in + let ticket = Host.request host session (W.Mutate intent) |> Result.get_ok in + Host.watch host ticket ~f:(function + | W.Answered (Ok (W.Submission_committed _)) -> done_ := true + | _ -> failwith "business window write failed"); + wait (fun () -> !done_) + in + write 1 "Business change A"; + wait (fun () -> !held <> None); + require (!acks = 0) "real ACK preceded the held UI application"; + require + (not (C.change_application_ready (Host.view host) (Option.get !batch))) + "held UI consumer became ready"; + write 2 "Business change B"; + wait (fun () -> !notices >= 2); + require (!acks = 0) "a newer push ACKed an unapplied older consumer"; + write 3 "Business change C"; + wait (fun () -> !notices >= 3); + Queue.add (Option.get !held) jobs; + held := None; + wait (fun () -> !acks >= 3 && !ui_title = Some "Business change C"); + require !more_seen "real byte/item-bounded Changes did not exercise More"; + require + (C.change_application_error (Host.view host) (Option.get !batch) = None) + "real applied consumers retained a failure"; + (* Original Capture/reconcile/status and stale-delete combinations, using + the production business reducer and the actual typed Service throughout. *) + let operation : type a. a W.request -> a = + fun request -> + let ticket = Host.request host session request |> Result.get_ok in + let result = ref None in + Host.watch host ticket ~f:(fun reply -> result := Some reply); + wait (fun () -> !result <> None); + match Option.get !result with + | W.Answered (Ok value) -> value + | _ -> failwith "combination fixture failed" + in + let target = uuid "9d000000-0000-4000-8000-000000008000" in + let target_id = W.Uuid.to_string target in + let append = + W.append_intent + session + ~mutation_id:(uuid "9d000000-0000-4000-8000-000000008001") + ~block_id:target + ~day:(W.journal_day 20260902 |> Result.get_ok) + ~title:"Captured regression block" + ~task_status:None + ~children:Rrbvec.empty + |> Result.get_ok + in + let receipt = + match operation (W.Append append) with + | W.Submission_committed value -> value + | _ -> failwith "capture seed failed" + in + let presentation = + Option.get (Option.bind (C.projection (Host.view host)) W.selected) + in + ignore (operation (W.Confirm (presentation, receipt))); + ignore (operation (W.Acknowledge_intent (W.Append_intent append))); + deliver + (Business.submit + business + (Load_detail + { block_id = target_id; after = None; limit = 32; request_generation = 910L })); + wait (fun () -> Hashtbl.mem latest_blocks target_id); + let save_target n title = + let budget = + W.read_budget session ~maximum_items:1 ~maximum_bytes:65536 |> Result.get_ok + in + let revision = + match + operation (W.Query (budget, W.Outline_block { uuid = target; revision = None })) + with + | Worker_pure_data.V2_present_block value -> value.revision + | _ -> failwith "captured target missing" + in + let intent = + W.mutation_intent + session + ~mutation_id:(uuid (Printf.sprintf "9d000000-0000-4000-8000-%012d" n)) + (W.Save_block + { block = target + ; title + ; preconditions = { blocks = [ target, revision ]; pages = []; scopes = [] } + }) + |> Result.get_ok + in + match operation (W.Mutate intent) with + | W.Submission_committed _ -> () + | _ -> failwith "concurrent target save failed" + in + save_target 8002 "Reconciled capture"; + wait (fun () -> + Journal_model.source (Hashtbl.find latest_blocks target_id) = "Reconciled capture"); + let reconciled = Hashtbl.find latest_blocks target_id in + let actual = + operation + (W.Query + ( W.read_budget session ~maximum_items:1 ~maximum_bytes:65536 |> Result.get_ok + , W.Outline_block { uuid = target; revision = None } )) + in + require + (match actual with + | Worker_pure_data.V2_present_block value -> + value.revision = Journal_model.revision reconciled + | _ -> false) + "changefeed reconciliation emitted an obsolete target revision"; + write 5 "Unrelated target change"; + let status = Test_support.status_uuid 3 in + deliver + (Business.submit + business + (Set_task_state + { mutation_id = "9d000000-0000-4000-8000-000000008003" + ; block_id = target_id + ; expected_revision = Journal_model.revision reconciled + ; task_state = Journal_model.Status (W.Uuid.to_string status) + })); + wait (fun () -> + Journal_model.task_state (Hashtbl.find latest_blocks target_id) + = Journal_model.Status (W.Uuid.to_string status)); + let stale = Hashtbl.find latest_blocks target_id in + save_target 8004 "Concurrent edit before delete deadline"; + let deletion revision id = + Business.submit + business + (Delete_subtree + { mutation_id = id; block_id = target_id; expected_revision = revision }) + in + deliver + (deletion (Journal_model.revision stale) "9d000000-0000-4000-8000-000000008005"); + wait (fun () -> !delete_conflict <> None); + let refreshed = Option.get !delete_conflict in + require + (Journal_model.source refreshed = "Concurrent edit before delete deadline") + "delete conflict retained stale canonical source"; + deliver + (deletion (Journal_model.revision refreshed) "9d000000-0000-4000-8000-000000008006"); + wait (fun () -> !deleted); + deliver + (Business.submit + business + (Load_detail + { block_id = W.Uuid.to_string Test_support.authoritative_block_uuid + ; after = None + ; limit = 1 + ; request_generation = 911L + })); + wait (fun () -> !ui_title = Some "Unrelated target change"); + expected_failure := true; + fail_budget_once := true; + let accepted_before = !accepted_change_roots in + write 4 "Business change D"; + wait (fun () -> + !saw_failure + && Option.exists + (fun value -> C.change_application_error (Host.view host) value <> None) + !batch); + let failed_batch = Option.get !batch in + require + (not (Hashtbl.mem acknowledged_batches failed_batch.through)) + "an actual failed read acknowledged its unapplied window"; + hold_change_root := true; + generation := Int64.succ !generation; + deliver + (Business.reconcile_signal + business + ~request_generation:!generation + (Option.get !latest_notice).notice); + wait (fun () -> Option.is_some !held_change_root); + let complete_root = Option.get !held_change_root in + held_change_root := None; + Queue.add complete_root jobs; + wait (fun () -> !accepted_change_roots > accepted_before && !pending_change_roots = 0); + require + (not (Hashtbl.mem acknowledged_batches failed_batch.through)) + "root settlement acknowledged the failed UI application"; + active := false; + ui_title := None; + detail_loaded := false; + Business.reset business; + wait (fun () -> Hashtbl.mem acknowledged_batches failed_batch.through); + require + (!ui_title = None) + "retired consumer callbacks revived the discarded business scope"; + ignore (Host.shutdown host); + wait (fun () -> C.shutdown_state (Host.view host) = C.Stopped)) +;; + +let with_observation_service + ?epoch_ms + ?prepare + ?change_max_items + ?(observe_push = fun _ -> ()) + f + = + with_client_output_wait + ?epoch_ms + ?prepare + ?change_max_items + (fun client ~wait:wait_output -> + let replies = Hashtbl.create 16 + and baseline = ref None + and notices = ref Rrbvec.empty in + Mailbox.on_event client (fun event -> + (match event with + | Mailbox.Push { payload; _ } -> observe_push payload + | _ -> ()); + match event with + | Response { request_id; outcome; _ } -> + Hashtbl.replace replies request_id outcome + | Push { payload = Service.Coordination_updates updates; _ } -> + List.iter + (function + | Service.Coordination_snapshot value -> remember_baseline baseline value + | _ -> ()) + updates + | Push { payload = Service.Coordination_observations updates; _ } -> + notices := Rrbvec.append !notices updates + | _ -> ()); + let wait predicate = + wait_output + ~pump:(fun () -> Mailbox.Private.deliver client ~max_events:64) + ~timeout:5. + ~label:"real observation Service did not settle" + predicate + in + let send value = + match Mailbox.send client value with + | Accepted id -> id + | _ -> failwith "observation Service request not admitted" + in + let call value = + let id = send value in + wait (fun () -> Hashtbl.mem replies id); + Hashtbl.find replies id + in + ignore + (call (Service.Client_command (Restore_local_account { user_id = "user-1" }))); + let registration = 910000L in + (match call (Service.Register_coordination registration) with + | Completed (Service.Coordination_registered value) -> + remember_baseline baseline value + | _ -> failwith "observation baseline registration failed"); + wait (fun () -> + match !baseline with + | Some value -> + List.exists + (fun (_, graph) -> + graph.Logseq_db_worker.phase = Logseq_db_worker.Graph_open) + value.baseline.state.residents + | None -> false); + let owner = Option.get (Option.get !baseline).baseline.state.sync.selected in + f client ~call ~send ~wait ~replies ~notices ~baseline ~owner ~registration) +;; + +let test_native_registration_network_epochs ?terminal ?(confirmation_first = false) () = + let module Registration = Logseq_db_worker_lui.Journal_sync_registration in + let module Pure = Logseq_db_worker.Pure_reducer in + let exception Finished in + T.with_managed (fun fixture -> + with_service_fixture (fun _ config -> + Eio_main.run (fun environment -> + Eio.Time.with_timeout_exn (Eio.Stdenv.clock environment) 10. (fun () -> + try + Eio.Switch.run (fun sw -> + let epochs = ref [] in + let platform_user = ref None in + let expected_generation = ref 1 in + let bridge = + Registration.create + ~sw + ~clock:(Eio.Stdenv.mono_clock environment) + ~network:(Eio.Stdenv.net environment) + ~authenticator:(fun ?ip:_ ~host:_ _ -> Ok None) + ~config + ~overlay:fixture.overlay + ~token:(fun ~user ~account_generation ~epoch ~refresh:_ -> + epochs := epoch :: !epochs; + require + (account_generation = !expected_generation) + "network reopen changed token account identity"; + if !platform_user <> Some user + then Error (Authentication_failure "Authentication authority changed") + else ( + match terminal with + | Some "protocol" -> + Error (Protocol_failure "Original protocol diagnostic") + | Some "storage" -> + Error (Storage_failure "Original storage diagnostic") + | Some "crypto" -> + Error (Crypto_failure "Original crypto diagnostic") + | Some "integrity" -> + Error + (Authoritative_failure + Logseq_overlay_db.Types.Authoritative_checksum_mismatch) + | Some _ -> + Error (Authentication_failure "Original auth diagnostic") + | None -> Error (Authentication_failure "Synthetic token rejection"))) + () + |> Result.get_ok + in + let module Db = Logseq_db_worker in + let module Runner = Db.Effect_runner in + let limits = + Sync.limits + ~maximum_response_bytes:config.response_budget_bytes + ~maximum_artifact_bytes:1048576 + ~submission_batch_size:16 + |> Result.get_ok + in + let sync = + Sync.config + ~managed_sync_origin:(Uri.of_string "https://api.logseq.io") + ~limits + |> Result.get_ok + in + let runtime = + Runner.runtime + ~fork:(fun ~sw task -> Eio.Fiber.fork ~sw task) + ~sleep:(Eio.Time.sleep (Eio.Stdenv.clock environment)) + |> Result.get_ok + in + let dependencies = + Runner.dependencies + ~runtime + ~config + ~overlay:fixture.overlay + ~services:bridge.services + ~publish:(fun _ -> ()) + |> Result.get_ok + in + let worker = + Db.create_lifecycle + ~sw + ~config:(Pure.config ~worker:config ~sync) + ~runner_dependencies:dependencies + |> Result.get_ok + in + bridge.bind worker; + Fun.protect + ~finally:(fun () -> Db.shutdown worker) + (fun () -> + let settle () = + for _ = 1 to 200 do + Eio.Fiber.yield () + done + in + let intent event = + bridge.lifecycle (Pure.Sync_intent event); + settle () + in + intent (Sync.Restore_local_account { user_id = "user-1" }); + require + ((Db.graph_state worker).phase = Db.Graph_open) + "native registration did not hand the local mirror to Worker"; + require + (not (Db.view worker).sync.snapshot.startup.bootstrapping) + "native registration left an attached mirror bootstrapping"; + let scope : Sync.startup_scope = + { account_generation = 1; presentation_generation = 1 } + in + if confirmation_first + then ( + platform_user := Some "user-1"; + intent (Sync.Account_authenticated { user_id = Some "user-1" }); + require + (!epochs = []) + "identity confirmation invented network intent"; + require + (Db.view worker).sync.snapshot.startup.authenticated + "verified identity was dropped before timeline admission"); + let timeline : Sync.timeline_scope = + { startup = scope + ; graph_id = Test_support.graph_uuid + ; graph_generation = + (Db.view worker).sync.snapshot.startup.graph_generation + } + in + require + (Result.is_ok + (bridge.validate_control (Registration.Recover_online timeline))) + "cold restored timeline scope is not current"; + intent (Sync.Timeline_presented timeline); + require + ((Db.view worker).sync.snapshot.startup.network_phase + = Sync.Online_after_timeline) + "cold restored timeline did not request online admission"; + for _ = 1 to 3 do + bridge.lifecycle (Pure.Foreground_intent false); + settle (); + bridge.lifecycle (Pure.Foreground_intent true); + settle () + done; + if not confirmation_first + then ( + require + (!epochs = []) + "cold restored timeline requested an unverified token"; + require + ((Db.view worker).sync.snapshot.last_error = None) + "cold restored timeline failed before identity confirmation"; + intent (Sync.Sign_in_requested scope); + intent (Sync.Online_recovery_requested timeline); + require + (!epochs = []) + "explicit recovery opened cached service before platform \ + confirmation"; + platform_user := Some "user-1"; + intent (Sync.Account_authenticated { user_id = Some "user-1" })); + require (!epochs <> []) "native registration did not request a token"; + let expected_error = + match terminal with + | Some "protocol" -> "Protocol failure: Original protocol diagnostic" + | Some "storage" -> "Storage failure: Original storage diagnostic" + | Some "crypto" -> "Crypto failure: Original crypto diagnostic" + | Some "integrity" -> + "Authoritative transition failed: checksum mismatch" + | Some _ -> "Original auth diagnostic" + | None -> "Synthetic token rejection" + in + let snapshot = (Db.view worker).sync.snapshot in + require + (snapshot.last_error = Some expected_error) + ("native registration lost the actual failure: " + ^ Option.value snapshot.last_error ~default:"none"); + require + (snapshot.sync_phase = Sync.Failed) + "native registration reports a failure as connecting"; + (match terminal with + | None -> () + | Some _ -> + let original_epochs = !epochs in + let assert_failure phase = + let snapshot = (Db.view worker).sync.snapshot in + require + (snapshot.sync_phase = phase) + "terminal recovery lost its visible phase"; + require + (snapshot.last_error = Some expected_error) + "terminal recovery replaced the original diagnostic" + in + intent (Sync.Online_recovery_requested timeline); + require + (!epochs = original_epochs) + "terminal explicit recovery requested network IO"; + assert_failure Sync.Failed; + bridge.lifecycle (Pure.Foreground_intent false); + settle (); + assert_failure Sync.Offline; + bridge.lifecycle (Pure.Foreground_intent true); + settle (); + assert_failure Sync.Failed; + intent (Sync.Online_recovery_requested timeline); + require + (!epochs = original_epochs) + "terminal foreground or recovery reconnected"; + assert_failure Sync.Failed; + raise Finished); + let previous = ref (List.hd !epochs) in + intent (Sync.Online_recovery_requested timeline); + require + (List.hd !epochs = !previous) + "generic recovery renewed authentication failure authorization"; + previous := List.hd !epochs; + let reopened label = + epochs := []; + intent (Sync.Catalog_refresh_requested scope); + require + (!epochs <> []) + (label ^ ": current gate did not admit catalog IO"); + let epoch = List.hd !epochs in + require + (epoch > !previous) + (label ^ ": network authorization epoch was reused"); + previous := epoch + in + for _ = 1 to 3 do + bridge.lifecycle (Pure.Foreground_intent false); + settle (); + bridge.lifecycle (Pure.Foreground_intent true); + settle (); + reopened "background/foreground" + done; + let stale_controls stale = + let timeline : Sync.timeline_scope = + { startup = stale + ; graph_id = Test_support.graph_uuid + ; graph_generation = 1 + } + in + List.iter + (fun control -> + require + (match bridge.validate_control control with + | Error Stale_context -> true + | _ -> false) + "stale registration control did not return Stale_context") + [ Registration.Sign_in stale + ; Cancel_startup stale + ; Refresh_catalog stale + ; Recover_online timeline + ]; + let before = List.length !epochs in + List.iter + intent + [ Sync.Online_cancelled stale + ; Catalog_refresh_requested stale + ; Online_recovery_requested timeline + ; Sign_in_requested stale + ]; + require + (List.length !epochs = before) + "stale registration control performed IO" + in + stale_controls + { scope with + presentation_generation = scope.presentation_generation + 1 + }; + stale_controls + { scope with account_generation = scope.account_generation + 1 }; + let wrong_graph : Sync.timeline_scope = + { startup = scope + ; graph_id = Test_support.graph_uuid + ; graph_generation = max_int + } + in + require + (match + bridge.validate_control (Registration.Recover_online wrong_graph) + with + | Error Stale_context -> true + | _ -> false) + "retired graph recovery was admitted"; + let before = List.length !epochs in + intent (Sync.Online_recovery_requested wrong_graph); + require + (List.length !epochs = before) + "retired graph recovery performed IO"; + epochs := []; + intent (Sync.Catalog_refresh_requested scope); + require + (!epochs <> [] && List.hd !epochs = !previous) + "stale control changed current gate"; + for _ = 1 to 3 do + intent (Sync.Online_cancelled scope); + epochs := []; + stale_controls + { scope with + presentation_generation = scope.presentation_generation + 1 + }; + stale_controls + { scope with account_generation = scope.account_generation + 1 }; + intent (Sync.Catalog_refresh_requested scope); + intent (Sync.Online_recovery_requested timeline); + require (!epochs = []) "recovery reopened cancelled authorization"; + bridge.lifecycle (Pure.Foreground_intent false); + settle (); + bridge.lifecycle (Pure.Foreground_intent true); + settle (); + intent (Sync.Account_authenticated { user_id = Some "user-1" }); + intent (Sync.Catalog_refresh_requested scope); + require (!epochs = []) "cancelled authorization was resurrected"; + intent (Sync.Sign_in_requested scope); + intent (Sync.Account_authenticated { user_id = Some "user-1" }); + reopened "cancel/sign-in same account" + done; + expected_generation := 2; + intent + (Sync.Restore_local_account { user_id = "other-synthetic-user" }); + let current : Sync.startup_scope = + { account_generation = 2; presentation_generation = 2 } + in + intent (Sync.Sign_in_requested current); + platform_user := Some "other-synthetic-user"; + intent + (Sync.Account_authenticated { user_id = Some "other-synthetic-user" }); + let epoch = List.hd !epochs in + require + (epoch > !previous) + "account replacement reused authorization epoch"; + stale_controls scope; + epochs := []; + intent (Sync.Catalog_refresh_requested current); + require + (!epochs <> [] && List.hd !epochs = epoch) + "old account controls changed new account gate"); + Eio.Switch.fail sw Finished) + with + | Finished -> ())))) +;; + +let test_native_service_cold_restore_waits_for_identity ?(confirmation_first = false) () = + let requests = ref [] + and snapshot = ref None in + let observe_push = function + | Service.Need_id_token request -> requests := request :: !requests + | Service.Client_state_changed state -> snapshot := Some state.snapshot + | _ -> () + in + with_observation_service + ~observe_push + (fun _ ~call ~send:_ ~wait ~replies:_ ~notices:_ ~baseline:_ ~owner ~registration:_ -> + let command value = + require + (match call (Service.Client_command value) with + | Mailbox.Completed Service.Client_command_completed -> true + | _ -> false) + "native cold restore control failed" + in + let confirm () = + command (Service.Reconcile_authenticated_user { user_id = Some "user-1" }) + in + let foreground_cycles () = + for _ = 1 to 3 do + command (Service.Set_foreground false); + command (Service.Set_foreground true) + done + in + let no_premature_io () = + require (!requests = []) "native cold restore performed premature token IO"; + require + (Option.exists (fun value -> value.Service.last_error = None) !snapshot) + "native cold restore published a premature authentication failure" + in + if confirmation_first + then ( + confirm (); + foreground_cycles (); + no_premature_io ()); + command (Service.Acknowledge_timeline_presented owner); + if not confirmation_first + then ( + foreground_cycles (); + wait (fun () -> + Option.exists + (fun value -> + value.Service.startup.network_phase = Sync.Online_after_timeline) + !snapshot); + no_premature_io (); + command (Service.Request_sign_in owner.startup); + no_premature_io (); + confirm ()); + wait (fun () -> !requests <> []); + require (List.length !requests = 1) "native gate opened more than one token flight"; + (match call (Service.Register_coordination 910001L) with + | Mailbox.Completed (Service.Coordination_registered value) -> + require + (value.authority_user = Some (Some "user-1")) + "native token authority did not retain the verified user" + | _ -> failwith "native verified authority baseline failed"); + confirm (); + command (Service.Acknowledge_timeline_presented owner); + require + (List.length !requests = 1) + "native gate did not reuse its pending token flight"; + command (Service.Reject_token (List.hd !requests)); + wait (fun () -> + Option.exists (fun value -> value.Service.last_error <> None) !snapshot); + require + (Option.exists + (fun value -> + value.Service.last_error <> Some "Authentication authority changed") + !snapshot) + "same-user confirmation did not authorize the actual token owner"; + command (Service.Cancel_online owner.startup); + let count = List.length !requests in + foreground_cycles (); + confirm (); + command (Service.Acknowledge_timeline_presented owner); + command (Service.Refresh_catalog owner.startup); + require + (List.length !requests = count) + "native cancelled authorization was reopened by confirmation or timeline") +;; + +let test_native_service_rejects_stale_network_commands () = + with_observation_service + (fun _ ~call ~send:_ ~wait ~replies:_ ~notices:_ ~baseline ~owner ~registration:_ -> + let rejected command = + require + (match call (Service.Client_command command) with + | Mailbox.Failed _ -> true + | _ -> false) + "stale network command reported success" + in + let stale : Sync.startup_scope = + { owner.startup with + presentation_generation = owner.startup.presentation_generation + 1 + } + in + rejected (Service.Cancel_online stale); + rejected (Service.Refresh_catalog stale); + rejected (Service.Request_sign_in stale); + rejected + (Service.Begin_online_recovery + { owner with graph_generation = owner.graph_generation + 1 }); + ignore (call (Service.Client_command Return_to_graph_picker)); + wait (fun () -> (Option.get !baseline).Service.baseline.state.sync.selected = None); + rejected (Service.Cancel_online owner.startup); + rejected (Service.Refresh_catalog owner.startup); + rejected (Service.Begin_online_recovery owner); + ignore + (call + (Service.Client_command + (Restore_local_account { user_id = "other-synthetic-user" }))); + wait (fun () -> + Option.exists + (fun account -> account.Sync.user_id = "other-synthetic-user") + (Option.get !baseline).Service.baseline.state.sync.account); + rejected (Service.Cancel_online owner.startup); + rejected (Service.Refresh_catalog owner.startup); + rejected (Service.Begin_online_recovery owner)) +;; + +let prepare_enriched_read_data ~count ~title support = + let module Storage = Logseq_db_storage.Logseq_sqlite_storage in + let module Session = Logseq_db_storage.Storage_session in + let path = + Filename.concat + support + ("logseq-db-worker/synced-graphs/" + ^ W.Uuid.to_string Test_support.graph_uuid + ^ "/db.sqlite") + in + let connection = Storage.open_database path |> Result.get_ok in + let storage = Storage.datascript_storage connection in + let database = Storage.restore_database connection |> Result.get_ok in + let session = + Session.create + ~tail:(Datascript.Storage.restore_tail_groups storage) + ~callbacks:(Storage.connection_callbacks connection) + in + Fun.protect + ~finally:(fun () -> ignore (Session.close session)) + (fun () -> + let open Datascript in + let tag = Temp_id "budget-tag" in + let page = + Lookup_ref ("block/uuid", Uuid (W.Uuid.to_string Test_support.page_uuid)) + in + let changes = + Rrbvec.init count (fun n -> + let block = Temp_id ("budget-block-" ^ string_of_int n) in + [ Add + ( block + , "block/uuid" + , Uuid (W.Uuid.to_string (Test_support.mutation_uuid (10000 + n))) ) + ; Add (block, "block/title", String "Small fixture body") + ; Add (block, "block/parent", Ref_to page) + ; Add (block, "block/page", Ref_to page) + ; Add (block, "block/order", String (Printf.sprintf "b%08d" n)) + ; Add (block, "block/created-at", Int64 1704067200000L) + ; Add (block, "block/updated-at", Int64 1704067200000L) + ; Add (block, "block/tags", Ref_to tag) + ] + |> Rrbvec.of_list) + |> Rrbvec.concat_map Fun.id + |> Rrbvec.to_list + in + let staged = + Session.stage_transact + session + ~authoritative_before:database + (Add + ( tag + , "block/uuid" + , Uuid (W.Uuid.to_string (Test_support.mutation_uuid 9999)) ) + :: Add (tag, "block/name", String "budget-fixture-tag") + :: Add (tag, "block/title", String title) + :: changes) + |> Result.get_ok + in + Session.commit_staged session staged |> Result.get_ok) +;; + +let service_read + : type a. + (Service.request, Service.response, Service.push) Mailbox.client + -> Service.coordination_baseline option + -> Service.timeline_scope + -> int64 + -> call:(Service.request -> Service.response Mailbox.outcome) + -> request_id:W.Uuid.t + -> maximum_items:int + -> maximum_bytes:int + -> a Read.query + -> (a, Read.error) result + = + fun client + baseline + owner + registration + ~call + ~request_id + ~maximum_items + ~maximum_bytes + query -> + let source = + observation_read + client + baseline + owner + registration + ~maximum_items + ~maximum_bytes + query + in + match + call (Service.Coordination_query { registration; owner; request_id; read = source }) + with + | Mailbox.Completed (Service.Coordination_queried response) + when response.request_id = request_id + && response.registration = registration + && response.owner = owner -> + let (Read.Query_answer (actual_budget, actual_query, result)) = response.answer in + (match Read.query_result_equality query actual_query with + | Some Read.Refl + when Read.equal_packed_query + source + (Read.For_worker.pack_query actual_budget actual_query) -> result + | _ -> failwith "Actual Service returned another query or budget") + | _ -> failwith "Actual typed Service read did not complete" +;; + +let test_enriched_response_budget () = + let title = String.make 4096 't' in + with_observation_service + ~prepare:(prepare_enriched_read_data ~count:195 ~title) + (fun + client + ~call + ~send:_ + ~wait:_ + ~replies:_ + ~notices:_ + ~baseline + ~owner + ~registration + -> + let module Ids = Set.Make (W.Uuid) in + let expected = + List.init 195 (fun n -> Test_support.mutation_uuid (10000 + n)) + |> List.fold_left (fun set id -> Ids.add id set) Ids.empty + in + let serial = ref 41000 in + let read : type a. a Read.query -> a = + fun query -> + incr serial; + service_read + client + !baseline + owner + registration + ~call + ~request_id:(Test_support.mutation_uuid !serial) + ~maximum_items:200 + ~maximum_bytes:Worker_pure_data.maximum_response_bytes + query + |> Result.get_ok + in + let check_record seen (value : Worker_pure_data.block_record) = + if not (Ids.mem value.block.uuid expected) + then seen + else ( + require + (value.tag_titles = [ title ]) + "adaptive page truncated enriched tag DTO"; + require + (not (Ids.mem value.block.uuid seen)) + "adaptive structural pages repeated a member"; + Ids.add value.block.uuid seen) + in + let rec children after pages seen = + require (pages < 16) "enriched children pagination did not terminate"; + let query = + Read.Child_members { parent = Test_support.page_uuid; after; revision = None } + in + let value = read query in + require + (Read.encoded_value_bytes query value + <= Worker_pure_data.maximum_response_bytes) + "enriched children exceeded final DTO budget"; + let seen = + List.fold_left + (fun seen (item : Worker_pure_data.child_member) -> + check_record seen item.value) + seen + value.items + in + match value.next with + | None -> + require + (pages > 0 && Ids.equal seen expected) + "adaptive children lost tagged members" + | Some _ -> + require (value.next <> after) "child cursor did not advance"; + children value.next (pages + 1) seen + in + let rec tree after pages seen = + require (pages < 16) "enriched Tree pagination did not terminate"; + let query = + Read.Tree + { page = Test_support.page_uuid; maximum_depth = 1; after; revision = None } + in + let value = read query in + require + (Read.encoded_value_bytes query value + <= Worker_pure_data.maximum_response_bytes) + "enriched Tree exceeded final DTO budget"; + let seen = + List.fold_left + (fun seen (item : Worker_pure_data.tree_member) -> + check_record seen item.value) + seen + value.items + in + match value.next with + | None -> + require + (pages > 0 && Ids.equal seen expected) + "adaptive Tree lost tagged members" + | Some _ -> + require (value.next <> after) "Tree cursor did not advance"; + tree value.next (pages + 1) seen + in + children None 0 Ids.empty; + tree None 0 Ids.empty) +;; + +let test_encoded_utf8_budget_boundary () = + let title = String.concat "" (List.init 300 (fun _ -> "日\"\n")) in + with_observation_service + ~prepare:(prepare_enriched_read_data ~count:1 ~title) + (fun + client + ~call + ~send:_ + ~wait:_ + ~replies:_ + ~notices:_ + ~baseline + ~owner + ~registration + -> + let query = + Read.Outline_block { uuid = Test_support.mutation_uuid 10000; revision = None } + in + let read n maximum_bytes = + service_read + client + !baseline + owner + registration + ~call + ~request_id:(Test_support.mutation_uuid (42000 + n)) + ~maximum_items:1 + ~maximum_bytes + query + in + let value = read 0 Worker_pure_data.maximum_response_bytes |> Result.get_ok in + (match value with + | Worker_pure_data.V2_present_block { value; _ } -> + require (value.tag_titles = [ title ]) "actual UTF-8 tag title was not resolved" + | _ -> failwith "UTF-8 block missing"); + let size = Read.encoded_value_bytes query value in + let exact = read 1 size |> Result.get_ok in + require + (Read.encoded_value_bytes query exact = size) + "exact final byte budget was not preserved"; + require + (match read 2 (size - 1) with + | Error Read.Response_budget_exceeded -> true + | _ -> false) + "UTF-8 plus escaped JSON N-1 did not fail explicitly") +;; + +let test_canonical_target_summary_query () = + let target = Test_support.mutation_uuid 16000 in + let prepare support = + Eio_main.run (fun _ -> + Eio.Switch.run (fun sw -> + prepare_enriched_read_data ~count:1000 ~title:"Unrelated sibling tag" support; + let module D = Logseq_overlay_db.Database in + let module O = Logseq_overlay_db.Types in + let inspection = + D.inspect_mirror + ~application_support_directory:support + ~graph_id:Test_support.graph_uuid + |> Result.get_ok + in + let db = + D.open_ + ~sw + (Test_support.dependencies ~behavior:"canonical target summary") + inspection + ~graph_name:"Target summary" + |> Result.get_ok + in + Fun.protect + ~finally:(fun () -> D.close db |> Result.get_ok) + (fun () -> + let children = + List.init 201 (fun n -> + O. + { uuid = Test_support.mutation_uuid (17000 + n) + ; title = Printf.sprintf "Canonical child %d" n + ; children = [] + }) + in + let expected = + Test_support.insert_precondition + db + ~parent:Test_support.page_uuid + ~behavior:"canonical target summary" + in + D.commit_local + db + ~expected + (O.Insert_blocks + { mutation_id = Test_support.mutation_uuid 15999 + ; parent = Test_support.page_uuid + ; tree = { uuid = target; title = "Canonical target"; children } + ; asset = None + }) + |> Result.get_ok + |> ignore))) + in + with_observation_service + ~prepare + (fun + client + ~call + ~send:_ + ~wait:_ + ~replies:_ + ~notices:_ + ~baseline + ~owner + ~registration + -> + let serial = ref 43000 in + let inspect + ?(maximum_items = Worker_pure_data.default_page_size) + ?(maximum_bytes = Worker_pure_data.maximum_response_bytes) + after + = + incr serial; + let query = Read.Block_summary { block = target; after } in + let result = + service_read + client + !baseline + owner + registration + ~call + ~request_id:(Test_support.mutation_uuid !serial) + ~maximum_items + ~maximum_bytes + query + in + query, result + in + let query, result = inspect ~maximum_items:200 ~maximum_bytes:16384 None in + let adapted = result |> Result.get_ok in + require + (Read.encoded_value_bytes query adapted <= 16384 + && adapted.items <> [] + && List.length adapted.items < 200 + && Option.is_some adapted.next) + "metadata-heavy summary did not shrink complete DTOs with a real continuation"; + List.iteri + (fun index (item : Worker_pure_data.child_member) -> + require + (item.value.block.uuid = Test_support.mutation_uuid (17000 + index) + && item.value.block.title = Printf.sprintf "Canonical child %d" index) + "adaptive target page truncated or skipped canonical DTO") + adapted.items; + let _, first = inspect None in + let first = Result.get_ok first in + require + (List.length first.items = Worker_pure_data.default_page_size) + "bounded direct summary children"; + let revision = + match first.lookup with + | Worker_pure_data.V2_present_block { value; revision } -> + require (value.block.title = "Canonical target") "canonical target title"; + revision + | _ -> failwith "canonical target missing" + in + (match first.page with + | Some (Worker_pure_data.V2_present_page { page; _ }) -> + require (page.uuid = Test_support.page_uuid) "owning page is canonical" + | _ -> failwith "owning page missing"); + let old_cursor = first.next in + require (Option.is_some old_cursor) "target child continuation missing"; + let rec collect count items (value : Read.block_summary) = + require + (count < 8 && List.length value.items <= Worker_pure_data.default_page_size) + "summary pages lost their bound"; + require + (value.generation = first.generation + && value.projection_revision = first.projection_revision) + "summary pages mixed snapshot versions"; + let items = Rrbvec.append items (Rrbvec.of_list value.items) in + match value.next with + | None -> count, items + | Some _ -> + let _, next = inspect value.next in + collect (count + 1) items (Result.get_ok next) + in + let pages, items = collect 1 Rrbvec.empty first in + require + (pages = 5 && Rrbvec.length items = 201) + "canonical summary did not return all 201 children"; + Rrbvec.iteri + (fun index (item : Worker_pure_data.child_member) -> + require + (item.value.block.uuid = Test_support.mutation_uuid (17000 + index)) + "canonical child order changed") + items; + let request_id = Test_support.mutation_uuid 43999 in + require + (match + call + (mutation_request + client + !baseline + owner + registration + ~request_id + ~mutation_id:request_id + (W.Save_block + { block = target + ; title = "Changed canonical title" + ; preconditions = + { blocks = [ target, revision ]; pages = []; scopes = [] } + })) + with + | Mailbox.Completed + (Service.Coordination_mutation_result + { outcome = Logseq_db_worker_contract.Worker_pure_mutation.Committed _ + ; _ + }) -> true + | _ -> false) + "actual canonical target update failed"; + let _, stale = inspect old_cursor in + require + (match stale with + | Error (Read.Stale_read _) -> true + | _ -> false) + "old canonical continuation admitted a mixed projection") +;; + +(* Pure registry retirement already passes; the defect is the absent production + Db -> runner -> registry call after the actual unregister joins. Exercise that + public Service boundary without injecting cleanup ACKs or registry state. *) +let test_service_binding_retirement_reclaims_observation_history () = + with_observation_service + (fun + client ~call ~send:_ ~wait:_ ~replies ~notices:_ ~baseline ~owner ~registration -> + for round = 0 to 8199 do + let registration = Int64.add registration (Int64.of_int round) in + if round > 0 + then + require + (match call (Service.Register_coordination registration) with + | Completed (Service.Coordination_registered _) -> true + | _ -> false) + "actual observation binding could not be registered again"; + let id = + uuid (Printf.sprintf "9d000000-0000-4000-8000-%012d" (100000 + round)) + in + require + (match + call + (Service.Coordination_observe + { registration + ; owner + ; observation_id = id + ; request_id = id + ; read = + observation_read + client + !baseline + owner + registration + Read.Graph_info + }) + with + | Completed (Service.Coordination_observed _) -> true + | _ -> false) + (Printf.sprintf + "actual binding retirement exhausted observation history at round %d" + (round + 1)); + require + (call (Service.Release_coordination registration) + = Completed (Service.Coordination_released registration)) + "actual observation unregister did not join its resource owner"; + Hashtbl.clear replies + done) +;; + +let test_service_writer_fence_waits_for_canceled_protected_commit () = + let armed = Atomic.make false + and entered = Atomic.make false + and released = Atomic.make false in + let epoch_ms () = + if Atomic.get armed + then ( + Atomic.set entered true; + while not (Atomic.get released) do + Eio.Fiber.yield () + done); + 1_700_000_000_000L + in + Fun.protect + ~finally:(fun () -> Atomic.set released true) + (fun () -> + with_observation_service + ~epoch_ms + (fun + client + ~call + ~send + ~wait + ~replies + ~notices:_ + ~baseline + ~owner + ~registration + -> + let module Wire = Logseq_db_worker_contract.Worker_pure_mutation in + let id n = uuid (Printf.sprintf "9f000000-0000-4000-8000-%012d" (9000 + n)) in + let read owner n = + call + (query_request + client + !baseline + owner + registration + (id n) + (Read.Outline_block + { uuid = Test_support.authoritative_block_uuid; revision = None })) + in + let revision = + match read owner 1 with + | Mailbox.Completed + (Service.Coordination_queried + { answer = + Read.Query_answer + ( _ + , Read.Outline_block _ + , Ok (Worker_pure_data.V2_present_block { revision; _ }) ) + ; _ + }) -> revision + | _ -> failwith "writer fence source read failed" + in + let intent n title = + Wire.create_mutation + ~origin:(Uri.of_string "https://api.logseq.io") + ~user:"user-1" + ~graph:owner.graph_id + ~mutation_id:(id n) + (Wire.Save_block + { block = Test_support.authoritative_block_uuid + ; title + ; preconditions = + { blocks = [ Test_support.authoritative_block_uuid, revision ] + ; pages = [] + ; scopes = [] + } + }) + |> Result.get_ok + |> fun intent -> Wire.Mutation_intent intent + in + let original = intent 2 "Protected writer actually committed" in + Atomic.set armed true; + let writer = + send + (Service.Coordination_mutation + { registration; owner; request_id = id 3; intent = original }) + in + wait (fun () -> Atomic.get entered); + Mailbox.cancel client ~request_id:writer; + let scope = + fst + (List.find + (fun ((scope : Service.asset_scope), _) -> + scope.graph_id = owner.graph_id) + (Option.get !baseline).baseline.state.residents) + in + let fence = + send + (Service.Coordination_quiesce_writers + { registration; owner = scope; request_id = id 4 }) + in + let rejected = + call + (Service.Coordination_mutation + { registration + ; owner + ; request_id = id 5 + ; intent = intent 6 "Must not enter held DB" + }) + in + require + (match rejected with + | Mailbox.Completed + (Service.Coordination_mutation_result { outcome = Wire.Unknown; _ }) -> + true + | _ -> false) + "writer fence admitted another durable operation into the held database"; + require + (not (Hashtbl.mem replies fence)) + "writer fence ACK preceded actual canceled protected commit"; + Atomic.set armed false; + Atomic.set released true; + wait (fun () -> Hashtbl.mem replies fence); + require + (match Hashtbl.find replies fence with + | Mailbox.Completed (Service.Coordination_writers_quiesced value) -> + value.registration = registration + && value.owner = scope + && value.request_id = id 4 + | _ -> false) + "writer fence failed to join actual protected I/O and callback barrier"; + require + (match read owner 7 with + | Mailbox.Failed _ -> true + | Completed + (Service.Coordination_queried + { answer = Read.Query_answer (_, _, Error _); _ }) -> true + | _ -> false) + "old graph incarnation resumed admission after writer fence"; + ignore (call (Service.Client_command Return_to_graph_picker)); + ignore (call (Service.Client_command (Select_graph owner.graph_id))); + wait (fun () -> + List.exists + (fun ((next : Service.asset_scope), graph) -> + next.graph_id = owner.graph_id + && next.graph_generation <> owner.graph_generation + && graph.Logseq_db_worker.phase = Graph_open) + (Option.get !baseline).baseline.state.residents); + let fresh = Option.get (Option.get !baseline).baseline.state.sync.selected in + require + (match read fresh 8 with + | Mailbox.Completed + (Service.Coordination_queried + { answer = + Read.Query_answer + ( _ + , Read.Outline_block _ + , Ok (Worker_pure_data.V2_present_block { value; _ }) ) + ; _ + }) -> value.block.title = "Protected writer actually committed" + | _ -> false) + "new incarnation lost actual canceled commit or inherited the old \ + admission fence")) +;; + +let test_service_import_rejects_retired_registration () = + with_client (fun client -> + let replies = Hashtbl.create 16 + and baseline = ref None in + Mailbox.on_event client (function + | Mailbox.Response { request_id; outcome; _ } -> + Hashtbl.replace replies request_id outcome + | Push { payload = Service.Coordination_updates updates; _ } -> + List.iter + (function + | Service.Coordination_snapshot value -> remember_baseline baseline value + | _ -> ()) + updates + | _ -> ()); + let wait predicate = + let deadline = Unix.gettimeofday () +. 4. in + while (not (predicate ())) && Unix.gettimeofday () < deadline do + Mailbox.Private.deliver client ~max_events:64; + Thread.delay 0.001 + done; + require (predicate ()) "typed Import registration fixture did not settle" + in + let call request = + let id = + match Mailbox.send client request with + | Accepted id -> id + | _ -> failwith "typed Import fixture admission" + in + wait (fun () -> Hashtbl.mem replies id); + Hashtbl.find replies id + in + ignore (call (Service.Client_command (Restore_local_account { user_id = "user-1" }))); + let issuer = + Worker_effect_authority.issuer + ~managed_origin:"https://api.logseq.io" + ~user_id:"user-1" + ~runtime_epoch:(Mailbox.runtime_epoch client) + ~worker_generation:(Mailbox.worker_generation client) + |> Result.get_ok + in + let module I = (val issuer : Worker_effect_authority.ISSUER) in + (match call (I.bootstrap_request () |> Result.get_ok) with + | Completed (Service.Coordination_registered value) -> + remember_baseline baseline value + | _ -> failwith "typed Import baseline request failed"); + wait (fun () -> + Option.exists + (fun value -> + List.exists + (fun (_, graph) -> graph.Logseq_db_worker.phase = Graph_open) + value.Service.baseline.state.residents) + !baseline); + let raw = Option.get !baseline in + let projection = + match I.bootstrap (Service.Coordination_registered raw) with + | Ok (W.Ready value) -> value + | _ -> failwith "typed Import baseline invalid" + in + let session = (List.hd (W.residents projection)).session in + let path = Filename.temp_file "journal-retired-import-synthetic-" ".png" in + Fun.protect + ~finally:(fun () -> if Sys.file_exists path then Sys.remove path) + (fun () -> + Out_channel.with_open_bin path (fun output -> output_string output "file"); + let source : Logseq_db_types.Asset_import.t = + { operation = uuid "9c000000-0000-4000-8000-000000001885" + ; asset = uuid "9c000000-0000-4000-8000-000000001886" + ; target = Test_support.authoritative_block_uuid + ; replace_reference = None + ; local_mutation = uuid "9c000000-0000-4000-8000-000000001887" + ; metadata_mutation = uuid "9c000000-0000-4000-8000-000000001888" + ; source_file = path + ; title = "Retired registration import" + ; file_type = "png" + } + in + let intent = W.asset_import_intent session source |> Result.get_ok in + let queued = + I.request session ~request_id:(W.request_id source.operation) (W.Import intent) + |> Result.get_ok + in + ignore (call (Service.Release_coordination raw.registration)); + require + (match call queued with + | Mailbox.Failed _ -> true + | _ -> false) + "retired coordination registration still prepared a queued typed Import")) +;; + +(* This crosses actual Service/Db/runner lifetimes and reopens the source store. + A pure reducer cannot establish that source bytes or the Prepared checkpoint + survived closing their physical owners. No unprepared recovery is claimed. *) +(* The facade's unregister implementation owns actual cache/DB/IO joins; Core + has no physical lease result from which to reproduce an optimistic ACK. *) +let test_revoked_hidden_lease_blocks_unregister_ack () = + with_client (fun client -> + let replies = Hashtbl.create 16 + and baseline = ref None + and manager = ref None in + Mailbox.on_event client (function + | Mailbox.Response { request_id; outcome; _ } -> + Hashtbl.replace replies request_id outcome + | Push { payload = Service.Coordination_updates updates; _ } -> + List.iter + (function + | Service.Coordination_snapshot value -> remember_baseline baseline value + | _ -> ()) + updates + | Push { payload = Service.Client_state_changed value; _ } -> + manager := Some value.snapshot + | _ -> ()); + let wait label predicate = + let deadline = Unix.gettimeofday () +. 4. in + while (not (predicate ())) && Unix.gettimeofday () < deadline do + Mailbox.Private.deliver client ~max_events:64; + Thread.delay 0.001 + done; + require (predicate ()) label + in + let call request = + let id = + match Mailbox.send client request with + | Accepted id -> id + | _ -> failwith "retirement fixture admission" + in + wait "actual retirement command did not settle" (fun () -> Hashtbl.mem replies id); + Hashtbl.find replies id + in + ignore (call (Service.Client_command (Restore_local_account { user_id = "user-1" }))); + (match call (Service.Register_coordination 7991L) with + | Completed (Service.Coordination_registered value) -> + remember_baseline baseline value + | _ -> failwith "retirement baseline"); + ignore + (call + (Service.Client_command + (Configure_resident_graphs [ Test_support.graph_uuid; second_graph ]))); + wait "two retirement residents did not open" (fun () -> + Option.exists + (fun value -> + List.length value.Service.baseline.state.residents = 2 + && List.for_all + (fun (_, state) -> state.Logseq_db_worker.phase = Graph_open) + value.baseline.state.residents) + !baseline); + let scope = + List.find_map + (fun ((scope : Service.asset_scope), _) -> + if scope.graph_id = second_graph then Some scope else None) + (Option.get !baseline).baseline.state.residents + |> Option.get + in + let owner : Service.timeline_scope = + { startup = + { account_generation = scope.account.account_generation + ; presentation_generation = scope.account.presentation_generation + } + ; graph_id = scope.graph_id + ; graph_generation = scope.graph_generation + } + in + let source_file = Filename.temp_file "journal-unregister-synthetic-" ".png" in + Fun.protect + ~finally:(fun () -> if Sys.file_exists source_file then Sys.remove source_file) + (fun () -> + Out_channel.with_open_bin source_file (fun output -> output_string output "file"); + let source : Logseq_db_types.Asset_import.t = + { operation = uuid "9c000000-0000-4000-8000-000000003885" + ; asset = uuid "9c000000-0000-4000-8000-000000003886" + ; target = Test_support.authoritative_block_uuid + ; replace_reference = None + ; local_mutation = uuid "9c000000-0000-4000-8000-000000003887" + ; metadata_mutation = uuid "9c000000-0000-4000-8000-000000003888" + ; source_file + ; title = "Retired hidden preview" + ; file_type = "png" + } + in + let imported = + match call (Service.Import_asset_at (owner, source)) with + | Completed (Service.Asset_imported (Ok value)) -> value + | _ -> failwith "actual hidden import failed" + in + let handle, path = imported.preview |> Option.get in + require (Sys.file_exists path) "physical preview missing before revocation"; + ignore + (call + (Service.Client_command (Reconcile_authenticated_user { user_id = None }))); + wait "actual account authority was not revoked" (fun () -> + Option.exists + (fun (value : Service.snapshot) -> + (not value.startup.authenticated) + && value.startup.account_generation <> scope.account.account_generation) + !manager); + require + (match call (Service.Release_coordination 7991L) with + | Mailbox.Failed _ -> true + | _ -> false) + "unregister ACK escaped while an actual old hidden native lease remained live"; + require + (Sys.file_exists path) + "failed unregister discarded retained renderer bytes"; + require + (call (Service.Release_asset_file { scope; handle }) + = Completed Service.Client_command_completed) + "revoked exact hidden lease lost physical cleanup authority"; + require + (call (Service.Release_coordination 7991L) + = Completed (Service.Coordination_released 7991L)) + "unregister did not retry its original physical cleanup after actual release"; + require + (call (Service.Acquire_asset_file { scope; handle }) + = Completed (Service.Asset_file None)) + "successful unregister retained old preview acquisition authority"; + require + (Sys.file_exists path) + "unregister deleted the durable pending import source")) +;; + +let test_prepared_import_survives_worker_lifetime () = + with_client_sessions (fun start -> + let source_file = Filename.temp_file "journal-restarted-import-synthetic-" ".png" in + Fun.protect + ~finally:(fun () -> if Sys.file_exists source_file then Sys.remove source_file) + (fun () -> + Out_channel.with_open_bin source_file (fun output -> output_string output "file"); + let source : Logseq_db_types.Asset_import.t = + { operation = uuid "9c000000-0000-4000-8000-000000002885" + ; asset = uuid "9c000000-0000-4000-8000-000000002886" + ; target = Test_support.authoritative_block_uuid + ; replace_reference = None + ; local_mutation = uuid "9c000000-0000-4000-8000-000000002887" + ; metadata_mutation = uuid "9c000000-0000-4000-8000-000000002888" + ; source_file + ; title = "Prepared across actual worker restart" + ; file_type = "png" + } + in + let lifetime saved = + let client = start () in + let replies = Hashtbl.create 16 + and baseline = ref None in + Mailbox.on_event client (function + | Mailbox.Response { request_id; outcome; _ } -> + Hashtbl.replace replies request_id outcome + | Push { payload = Service.Coordination_updates updates; _ } -> + List.iter + (function + | Service.Coordination_snapshot value -> + remember_baseline baseline value + | _ -> ()) + updates + | _ -> ()); + let wait predicate = + let deadline = Unix.gettimeofday () +. 5. in + while (not (predicate ())) && Unix.gettimeofday () < deadline do + Mailbox.Private.deliver client ~max_events:64; + Thread.delay 0.001 + done; + require (predicate ()) "restarted Import actual Service did not settle" + in + let call request = + let id = + match Mailbox.send client request with + | Accepted id -> id + | _ -> failwith "restarted Import admission" + in + wait (fun () -> Hashtbl.mem replies id); + Hashtbl.find replies id + in + ignore + (call + (Service.Client_command (Restore_local_account { user_id = "user-1" }))); + let issuer = + Worker_effect_authority.issuer + ~managed_origin:"https://api.logseq.io" + ~user_id:"user-1" + ~runtime_epoch:(Mailbox.runtime_epoch client) + ~worker_generation:(Mailbox.worker_generation client) + |> Result.get_ok + in + let module I = (val issuer : Worker_effect_authority.ISSUER) in + (match call (I.bootstrap_request () |> Result.get_ok) with + | Completed (Service.Coordination_registered value) -> + remember_baseline baseline value + | _ -> failwith "restarted Import bootstrap failed"); + wait (fun () -> + Option.exists + (fun value -> + List.exists + (fun (_, graph) -> graph.Logseq_db_worker.phase = Graph_open) + value.Service.baseline.state.residents) + !baseline); + let projection = + match + I.bootstrap (Service.Coordination_registered (Option.get !baseline)) + with + | Ok (W.Ready value) -> value + | _ -> failwith "restarted Import baseline invalid" + in + let session = + (List.find + (fun resident -> W.graph_id resident.W.session = Test_support.graph_uuid) + (W.residents projection)) + .session + in + let sequence = ref 0 in + let request : type a. a W.request -> a W.reply = + fun operation -> + incr sequence; + let request_id = + W.request_id + (uuid + (Printf.sprintf "9c000000-0000-4000-8000-%012d" (3000 + !sequence))) + in + let raw = I.request session ~request_id operation |> Result.get_ok in + match call raw with + | Completed response -> + I.reply session ~request_id operation response |> Result.get_ok + | Failed message -> failwith ("restarted Import request failed: " ^ message) + | _ -> failwith "restarted Import request interrupted" + in + let children () = + let budget = + W.read_budget session ~maximum_items:100 ~maximum_bytes:65536 + |> Result.get_ok + in + match + request + (W.Query (budget, W.Children { parent = source.target; after = None })) + with + | W.Answered (Ok value) -> + require (value.next = None) "synthetic Import child count was partial"; + value.items + | _ -> failwith "restarted Import children unavailable" + in + let before = children () in + let intent = + match saved with + | None -> W.asset_import_intent session source |> Result.get_ok + | Some (encoded, _, _) -> + W.restore_asset_import_intent session encoded |> Result.get_ok + in + let imported = + match request (W.Import intent) with + | W.Answered (Ok (W.Imported imported)) -> imported + | W.Answered (Error (W.Service_failed message)) -> + failwith ("restarted Import failed: " ^ message) + | _ -> failwith "restarted Import did not reconcile Prepared" + in + let preview = W.import_preview imported |> Option.get in + require + (Sys.file_exists (W.file_path preview)) + "restarted Import lost physically staged source"; + require + ((W.import_data imported).operation = source.operation + && (W.import_data imported).target = source.target) + "restarted Import replaced original operation/target"; + let after = children () in + require + (List.exists (fun block -> W.block_uuid block = source.asset) after) + "actual Import did not materialize its asset child"; + (match saved with + | None -> + require + (List.length after = List.length before + 1) + "first Import duplicated asset children" + | Some (_, asset, count) -> + require + ((W.import_data imported).asset = asset) + "restarted Import changed original descriptor"; + require + (List.length before = count && List.length after = count) + "restarted Import replayed local insertion"); + require + (request (W.Release_file preview) = W.Answered (Ok ())) + "restarted Import physical release was not acknowledged"; + ( W.encode_asset_import_intent intent + , (W.import_data imported).asset + , List.length after ) + in + let saved = lifetime None in + Sys.remove source_file; + ignore (lifetime (Some saved)))) +;; + +let test_service_observation_lifecycle () = + with_observation_service + (fun client ~call ~send:_ ~wait ~replies:_ ~notices ~baseline ~owner ~registration -> + let logical = uuid "9d000000-0000-4000-8000-000000000001" in + let rid n = uuid (Printf.sprintf "9d000000-0000-4000-8000-%012d" n) in + let source = + observation_read + client + !baseline + owner + registration + (Read.Outline_block + { uuid = Test_support.authoritative_block_uuid; revision = None }) + in + let observe = + Service.Coordination_observe + { registration + ; owner + ; observation_id = logical + ; request_id = rid 2 + ; read = source + } + in + let revision = + match call observe with + | Completed + (Service.Coordination_observed + { epoch = 0L + ; answer = + Read.Query_answer + ( _ + , Read.Outline_block _ + , Ok (Worker_pure_data.V2_present_block { revision; _ }) ) + ; _ + }) -> revision + | _ -> failwith "real initial observation did not sample its predecessor" + in + require + (match call observe with + | Completed (Service.Coordination_observed { epoch = 0L; _ }) -> true + | _ -> false) + "duplicate Observe did not retain original registration"; + let retry_request = rid 10 in + require + (match + call + (Service.Coordination_observe + { registration + ; owner + ; observation_id = logical + ; request_id = retry_request + ; read = source + }) + with + | Completed (Service.Coordination_observed { epoch = 0L; request_id; _ }) -> + request_id = retry_request + | _ -> false) + "retried Observe lost current RPC correlation"; + require + (call + (Service.Client_command + (Configure_resident_graphs [ Test_support.graph_uuid; second_graph ])) + = Mailbox.Completed Service.Client_command_completed) + "synthetic resident configuration failed"; + wait (fun () -> + match !baseline with + | Some value -> + List.length value.baseline.state.residents = 2 + && List.for_all + (fun (_, graph) -> + graph.Logseq_db_worker.phase = Logseq_db_worker.Graph_open) + value.baseline.state.residents + | None -> false); + require + (call (Service.Client_command (Focus_resident_graph second_graph)) + = Mailbox.Completed Service.Client_command_completed) + "synthetic focus change failed"; + wait (fun () -> + match !baseline with + | Some value -> + Option.exists + (fun (selected : Service.timeline_scope) -> selected.graph_id = second_graph) + value.baseline.state.sync.selected + | None -> false); + require + (match + call + (Service.Coordination_refresh_observation + { registration; owner; observation_id = logical; request_id = rid 9 }) + with + | Completed + (Service.Coordination_observed + { answer = + Read.Query_answer + ( _ + , Read.Outline_block _ + , Ok (Worker_pure_data.V2_present_block { revision = actual; _ }) + ) + ; _ + }) -> actual = revision + | _ -> false) + "focus retired a surviving resident observation"; + let write n revision title = + call + (mutation_request + client + !baseline + owner + registration + ~request_id:(rid n) + ~mutation_id:(rid (n + 100)) + (W.Save_block + { block = Test_support.authoritative_block_uuid + ; title + ; preconditions = + { blocks = [ Test_support.authoritative_block_uuid, revision ] + ; pages = [] + ; scopes = [] + } + })) + in + require + (match write 3 revision "Observed write" with + | Completed + (Service.Coordination_mutation_result + { outcome = Logseq_db_worker_contract.Worker_pure_mutation.Committed _ + ; _ + }) -> true + | _ -> false) + "observation synthetic mutation failed"; + wait (fun () -> + Rrbvec.exists + (function + | Service.Observation_invalidated value -> + value.registration_id = registration + && value.observation_id = logical + && value.epoch > 0L + | _ -> false) + !notices); + let refreshed = + call + (Service.Coordination_refresh_observation + { registration; owner; observation_id = logical; request_id = rid 4 }) + in + let current_revision = + match refreshed with + | Completed + (Service.Coordination_observed + { epoch + ; answer = + Read.Query_answer + ( _ + , Read.Outline_block _ + , Ok (Worker_pure_data.V2_present_block { value; revision }) ) + ; _ + }) -> + require + (epoch > 0L && value.block.title = "Observed write") + "Refresh lost epoch/data"; + revision + | _ -> failwith "real observation refresh failed" + in + let finalize = + Service.Coordination_finalize_observation + { registration; owner; observation_id = logical } + in + require + (match call finalize with + | Failed _ -> true + | _ -> false) + "actual Service finalized a live observation before physical release"; + let release = + Service.Coordination_release_observation + { registration; owner; observation_id = logical } + in + require + (match call release with + | Completed (Service.Coordination_observation_released _) -> true + | _ -> false) + "release did not acknowledge listener/read retirement"; + require + (match call release with + | Completed (Service.Coordination_observation_released _) -> true + | _ -> false) + "release ACK was not idempotent"; + require + (match call observe with + | Failed _ -> true + | _ -> false) + "retired logical observation was allocated again"; + require + (match call finalize with + | Completed (Service.Coordination_observation_finalized value) -> + value.registration = registration + && value.owner = owner + && value.observation_id = logical + | _ -> false) + "actual creator/release ACK did not permit observation finalization"; + require + (match call finalize with + | Completed (Service.Coordination_observation_finalized _) -> true + | _ -> false) + "uncertain finalization retry was not idempotent"; + notices := Rrbvec.empty; + require + (match write 5 current_revision "After observation release" with + | Completed + (Service.Coordination_mutation_result + { outcome = Logseq_db_worker_contract.Worker_pure_mutation.Committed _ + ; _ + }) -> true + | _ -> false) + "post-release synthetic write did not commit"; + for _ = 1 to 12 do + Mailbox.Private.deliver client ~max_events:64; + Thread.delay 0.001 + done; + require + (Rrbvec.is_empty !notices) + "released observation published a later invalidation"; + let cancelled = uuid "9d000000-0000-4000-8000-000000000006" in + require + (match + call + (Service.Coordination_release_observation + { registration; owner; observation_id = cancelled }) + with + | Completed (Service.Coordination_observation_released _) -> true + | _ -> false) + "release before registration did not create a fence"; + require + (match + call + (Service.Coordination_observe + { registration + ; owner + ; observation_id = cancelled + ; request_id = rid 2 + ; read = source + }) + with + | Failed _ -> true + | _ -> false) + "late observation escaped its pre-registration release"; + ignore (call (Service.Release_coordination registration))) +;; + +let test_service_empty_observation_is_retained () = + with_observation_service + (fun _client ~call ~send:_ ~wait ~replies:_ ~notices ~baseline ~owner ~registration -> + let target = uuid "9d000000-0000-4000-8000-000000005000" in + let logical = uuid "9d000000-0000-4000-8000-000000005001" in + let rid n = uuid (Printf.sprintf "9d000000-0000-4000-8000-%012d" n) in + let point = + observation_read + _client + !baseline + owner + registration + (Read.Outline_block { uuid = target; revision = None }) in - let changes = - Rrbvec.init count (fun n -> - let block = Temp_id ("budget-block-" ^ string_of_int n) in - [ Add - ( block - , "block/uuid" - , Uuid (W.Uuid.to_string (Test_support.mutation_uuid (10000 + n))) ) - ; Add (block, "block/title", String "Small fixture body") - ; Add (block, "block/parent", Ref_to page) - ; Add (block, "block/page", Ref_to page) - ; Add (block, "block/order", String (Printf.sprintf "b%08d" n)) - ; Add (block, "block/created-at", Int64 1704067200000L) - ; Add (block, "block/updated-at", Int64 1704067200000L) - ; Add (block, "block/tags", Ref_to tag) - ] - |> Rrbvec.of_list) - |> Rrbvec.concat_map Fun.id - |> Rrbvec.to_list + require + (match + call + (Service.Coordination_observe + { registration + ; owner + ; observation_id = logical + ; request_id = rid 5002 + ; read = point + }) + with + | Completed + (Service.Coordination_observed + { answer = + Read.Query_answer + ( _ + , Read.Outline_block _ + , Ok (Worker_pure_data.V2_missing_block { uuid; _ }) ) + ; _ + }) -> uuid = target + | _ -> false) + "empty initial sample was not observable"; + let read n query = + call (query_request _client !baseline owner registration (rid n) query) in - let staged = - Session.stage_transact - session - ~authoritative_before:database - (Add - (tag, "block/uuid", Uuid (W.Uuid.to_string (Test_support.mutation_uuid 9999))) - :: Add (tag, "block/name", String "budget-fixture-tag") - :: Add (tag, "block/title", String title) - :: changes) - |> Result.get_ok + let page_revision = + match + read 5003 (Read.Page { uuid = Test_support.page_uuid; revision = None }) + with + | Completed + (Service.Coordination_queried + { answer = + Read.Query_answer + ( _ + , Read.Page _ + , Ok (Worker_pure_data.V2_present_page { revision; _ }) ) + ; _ + }) -> revision + | _ -> failwith "observation insert page revision unavailable" in - Session.commit_staged session staged |> Result.get_ok) -;; - -let service_read : type a. - (Service.request,Service.response,Service.push)Mailbox.client -> Service.coordination_baseline option -> Service.timeline_scope -> int64 - -> call:(Service.request -> Service.response Mailbox.outcome) - -> request_id:W.Uuid.t -> maximum_items:int -> maximum_bytes:int -> a Read.query - -> (a,Read.error)result = - fun client baseline owner registration ~call ~request_id ~maximum_items ~maximum_bytes query-> - let source=observation_read client baseline owner registration - ~maximum_items ~maximum_bytes query in - match call(Service.Coordination_query{registration;owner;request_id;read=source})with - |Mailbox.Completed(Service.Coordination_queried response) - when response.request_id=request_id && response.registration=registration && response.owner=owner-> - let Read.Query_answer(actual_budget,actual_query,result)=response.answer in - (match Read.query_result_equality query actual_query with - |Some Read.Refl when Read.equal_packed_query source(Read.For_worker.pack_query actual_budget actual_query)->result - |_->failwith "Actual Service returned another query or budget") - |_->failwith "Actual typed Service read did not complete" -;; - -let test_enriched_response_budget ()= - let title=String.make 4096 't'in - with_observation_service ~prepare:(prepare_enriched_read_data ~count:195 ~title) - (fun client ~call ~send:_ ~wait:_ ~replies:_ ~notices:_ ~baseline ~owner ~registration-> - let module Ids=Set.Make(W.Uuid)in - let expected=List.init 195(fun n->Test_support.mutation_uuid(10000+n)) - |>List.fold_left(fun set id->Ids.add id set)Ids.empty in - let serial=ref 41000 in - let read :type a. a Read.query->a=fun query->incr serial; - service_read client !baseline owner registration ~call - ~request_id:(Test_support.mutation_uuid !serial) ~maximum_items:200 - ~maximum_bytes:Worker_pure_data.maximum_response_bytes query |> Result.get_ok in - let check_record seen (value:Worker_pure_data.block_record)= - if not(Ids.mem value.block.uuid expected)then seen else( - require(value.tag_titles=[title]) "adaptive page truncated enriched tag DTO"; - require(not(Ids.mem value.block.uuid seen)) "adaptive structural pages repeated a member"; - Ids.add value.block.uuid seen)in - let rec children after pages seen= - require(pages<16) "enriched children pagination did not terminate"; - let query=Read.Child_members{parent=Test_support.page_uuid;after;revision=None}in - let value=read query in - require(Read.encoded_value_bytes query value<=Worker_pure_data.maximum_response_bytes) - "enriched children exceeded final DTO budget"; - let seen=List.fold_left(fun seen(item:Worker_pure_data.child_member)->check_record seen item.value)seen value.items in - match value.next with - |None->require(pages>0 && Ids.equal seen expected) "adaptive children lost tagged members" - |Some _->require(value.next<>after) "child cursor did not advance";children value.next(pages+1)seen in - let rec tree after pages seen= - require(pages<16) "enriched Tree pagination did not terminate"; - let query=Read.Tree{page=Test_support.page_uuid;maximum_depth=1;after;revision=None}in - let value=read query in - require(Read.encoded_value_bytes query value<=Worker_pure_data.maximum_response_bytes) - "enriched Tree exceeded final DTO budget"; - let seen=List.fold_left(fun seen(item:Worker_pure_data.tree_member)->check_record seen item.value)seen value.items in - match value.next with - |None->require(pages>0 && Ids.equal seen expected) "adaptive Tree lost tagged members" - |Some _->require(value.next<>after) "Tree cursor did not advance";tree value.next(pages+1)seen in - children None 0 Ids.empty;tree None 0 Ids.empty) -;; - -let test_encoded_utf8_budget_boundary ()= - let title=String.concat ""(List.init 300(fun _->"日\"\n"))in - with_observation_service ~prepare:(prepare_enriched_read_data ~count:1 ~title) - (fun client ~call ~send:_ ~wait:_ ~replies:_ ~notices:_ ~baseline ~owner ~registration-> - let query=Read.Outline_block{uuid=Test_support.mutation_uuid 10000;revision=None}in - let read n maximum_bytes=service_read client !baseline owner registration ~call - ~request_id:(Test_support.mutation_uuid(42000+n)) ~maximum_items:1 ~maximum_bytes query in - let value=read 0 Worker_pure_data.maximum_response_bytes |> Result.get_ok in - (match value with Worker_pure_data.V2_present_block{value;_}-> - require(value.tag_titles=[title]) "actual UTF-8 tag title was not resolved"|_->failwith "UTF-8 block missing"); - let size=Read.encoded_value_bytes query value in - let exact=read 1 size |> Result.get_ok in - require(Read.encoded_value_bytes query exact=size) "exact final byte budget was not preserved"; - require(match read 2(size-1)with Error Read.Response_budget_exceeded->true|_->false) - "UTF-8 plus escaped JSON N-1 did not fail explicitly") + let scope_revision = + match + read + 5004 + (Read.Child_members + { parent = Test_support.page_uuid; revision = None; after = None }) + with + | Completed + (Service.Coordination_queried + { answer = Read.Query_answer (_, Read.Child_members _, Ok children); _ }) + -> children.scope_revision + | _ -> failwith "observation insert children revision unavailable" + in + require + (match + call + (mutation_request + _client + !baseline + owner + registration + ~request_id:(rid 5005) + ~mutation_id:(rid 5006) + (W.Insert_blocks + { parent = Test_support.page_uuid + ; roots = + [ { uuid = target; title = "First observed block"; children = [] } + ] + ; preconditions = + { blocks = [] + ; pages = [ Test_support.page_uuid, page_revision ] + ; scopes = + [ ( Worker_pure_data.V2_children_scope Test_support.page_uuid + , scope_revision ) + ] + } + })) + with + | Completed + (Service.Coordination_mutation_result + { outcome = Logseq_db_worker_contract.Worker_pure_mutation.Committed _ + ; _ + }) -> true + | _ -> false) + "synthetic first insertion did not commit"; + wait (fun () -> + Rrbvec.exists + (function + | Service.Observation_invalidated value -> value.observation_id = logical + | _ -> false) + !notices); + require + (match + call + (Service.Coordination_refresh_observation + { registration; owner; observation_id = logical; request_id = rid 5007 }) + with + | Completed + (Service.Coordination_observed + { answer = + Read.Query_answer + ( _ + , Read.Outline_block _ + , Ok (Worker_pure_data.V2_present_block { value; _ }) ) + ; _ + }) -> + value.block.uuid = target && value.block.title = "First observed block" + | _ -> false) + "missing sample discarded the interest before first value"; + ignore (call (Service.Release_coordination registration))) ;; -let test_canonical_target_summary_query ()= - let target = Test_support.mutation_uuid 16000 in - let prepare support=Eio_main.run(fun _->Eio.Switch.run(fun sw-> - prepare_enriched_read_data ~count:1000 ~title:"Unrelated sibling tag" support; - let module D = Logseq_overlay_db.Database in - let module O = Logseq_overlay_db.Types in - let inspection = - D.inspect_mirror ~application_support_directory:support ~graph_id:Test_support.graph_uuid - |> Result.get_ok - in - let db = - D.open_ - ~sw - (Test_support.dependencies ~behavior:"canonical target summary") - inspection - ~graph_name:"Target summary" - |> Result.get_ok - in - Fun.protect - ~finally:(fun () -> D.close db |> Result.get_ok) - (fun () -> - let children = - List.init 201 (fun n -> - O. - { uuid = Test_support.mutation_uuid (17000 + n) - ; title = Printf.sprintf "Canonical child %d" n - ; children = [] - }) - in - let expected = - Test_support.insert_precondition - db - ~parent:Test_support.page_uuid - ~behavior:"canonical target summary" - in - D.commit_local - db - ~expected - (O.Insert_blocks - { mutation_id = Test_support.mutation_uuid 15999 - ; parent = Test_support.page_uuid - ; tree = { uuid = target; title = "Canonical target"; children } - ; asset = None +let test_service_observation_read_failure_retains_interest () = + with_observation_service + (fun _client ~call ~send:_ ~wait ~replies:_ ~notices ~baseline ~owner ~registration -> + let logical = uuid "9e000000-0000-4000-8000-000000000001" in + let rid n = uuid (Printf.sprintf "9e000000-0000-4000-8000-%012d" n) in + let query = + Read.Outline_block + { uuid = Test_support.authoritative_block_uuid; revision = None } + in + let read n = + match + call + (query_request + _client + !baseline + owner + registration + (rid n) + ~maximum_bytes:Worker_pure_data.maximum_response_bytes + query) + with + | Completed + (Service.Coordination_queried + { answer = + Read.Query_answer + ( _ + , Read.Outline_block _ + , Ok (Worker_pure_data.V2_present_block { revision; _ } as value) ) + ; _ + }) -> revision, Read.encoded_value_bytes query value + | _ -> failwith "observation recovery source read failed" + in + let _, encoded_bytes = read 2 in + let budget = encoded_bytes + 128 in + require (budget < 60000) "recovery fixture envelope unexpectedly large"; + let write n title = + let revision, _ = read (n + 100) in + require + (match + call + (mutation_request + _client + !baseline + owner + registration + ~request_id:(rid n) + ~mutation_id:(rid (n + 200)) + (W.Save_block + { block = Test_support.authoritative_block_uuid + ; title + ; preconditions = + { blocks = [ Test_support.authoritative_block_uuid, revision ] + ; pages = [] + ; scopes = [] + } + })) + with + | Completed + (Service.Coordination_mutation_result + { outcome = Logseq_db_worker_contract.Worker_pure_mutation.Committed _ + ; _ + }) -> true + | _ -> false) + "synthetic observation recovery write failed" + in + let too_large = String.make (budget + 1024) 'L' in + write 3 too_large; + let initial = + call + (Service.Coordination_observe + { registration + ; owner + ; observation_id = logical + ; request_id = rid 4 + ; read = + observation_read + _client + !baseline + owner + registration + ~maximum_bytes:budget + (Read.Outline_block + { uuid = Test_support.authoritative_block_uuid; revision = None }) }) - |> Result.get_ok - |> ignore) -)) in - with_observation_service ~prepare - (fun client ~call ~send:_ ~wait:_ ~replies:_ ~notices:_ ~baseline ~owner ~registration-> - let serial=ref 43000 in - let inspect ?(maximum_items=Worker_pure_data.default_page_size) ?(maximum_bytes=Worker_pure_data.maximum_response_bytes) after= - incr serial;let query=Read.Block_summary{block=target;after}in - let result=service_read client !baseline owner registration ~call - ~request_id:(Test_support.mutation_uuid !serial) ~maximum_items ~maximum_bytes query in - query,result in - let query,result=inspect ~maximum_items:200 ~maximum_bytes:16384 None in - let adapted=result |> Result.get_ok in - require(Read.encoded_value_bytes query adapted<=16384 && adapted.items<>[] - && List.length adapted.items<200 && Option.is_some adapted.next) - "metadata-heavy summary did not shrink complete DTOs with a real continuation"; - List.iteri(fun index(item:Worker_pure_data.child_member)-> - require(item.value.block.uuid=Test_support.mutation_uuid(17000+index) - && item.value.block.title=Printf.sprintf "Canonical child %d" index) - "adaptive target page truncated or skipped canonical DTO")adapted.items; - let _,first=inspect None in let first=Result.get_ok first in - require(List.length first.items=Worker_pure_data.default_page_size) "bounded direct summary children"; - let revision=match first.lookup with - |Worker_pure_data.V2_present_block{value;revision}-> - require(value.block.title="Canonical target") "canonical target title";revision - |_->failwith "canonical target missing"in - (match first.page with Some(Worker_pure_data.V2_present_page{page;_})-> - require(page.uuid=Test_support.page_uuid) "owning page is canonical"|_->failwith "owning page missing"); - let old_cursor=first.next in require(Option.is_some old_cursor) "target child continuation missing"; - let rec collect count items (value:Read.block_summary)= - require(count<8 && List.length value.items<=Worker_pure_data.default_page_size) "summary pages lost their bound"; - require(value.generation=first.generation && value.projection_revision=first.projection_revision) - "summary pages mixed snapshot versions"; - let items=Rrbvec.append items(Rrbvec.of_list value.items)in - match value.next with - |None->count,items - |Some _->let _,next=inspect value.next in collect(count+1)items(Result.get_ok next)in - let pages,items=collect 1 Rrbvec.empty first in - require(pages=5 && Rrbvec.length items=201) "canonical summary did not return all 201 children"; - Rrbvec.iteri(fun index(item:Worker_pure_data.child_member)-> - require(item.value.block.uuid=Test_support.mutation_uuid(17000+index)) "canonical child order changed")items; - let request_id=Test_support.mutation_uuid 43999 in - require(match call(mutation_request client !baseline owner registration ~request_id ~mutation_id:request_id - (W.Save_block{block=target;title="Changed canonical title"; - preconditions={blocks=[target,revision];pages=[];scopes=[]}}))with - |Mailbox.Completed(Service.Coordination_mutation_result{outcome=Logseq_db_worker_contract.Worker_pure_mutation.Committed _;_})->true - |_->false) "actual canonical target update failed"; - let _,stale=inspect old_cursor in - require(match stale with Error(Read.Stale_read _)->true|_->false) - "old canonical continuation admitted a mixed projection") -;; - -(* Pure registry retirement already passes; the defect is the absent production - Db -> runner -> registry call after the actual unregister joins. Exercise that - public Service boundary without injecting cleanup ACKs or registry state. *) -let test_service_binding_retirement_reclaims_observation_history () = - with_observation_service(fun client ~call ~send:_ ~wait:_ ~replies ~notices:_ ~baseline ~owner ~registration -> - for round=0 to 8199 do - let registration=Int64.add registration(Int64.of_int round) in - if round>0 then require(match call(Service.Register_coordination registration) with - |Completed(Service.Coordination_registered _)->true|_->false) - "actual observation binding could not be registered again"; - let id=uuid(Printf.sprintf "9d000000-0000-4000-8000-%012d" (100000+round)) in - require(match call(Service.Coordination_observe{registration;owner;observation_id=id;request_id=id; - read=observation_read client !baseline owner registration Read.Graph_info}) with - |Completed(Service.Coordination_observed _)->true|_->false) - (Printf.sprintf "actual binding retirement exhausted observation history at round %d" (round+1)); - require(call(Service.Release_coordination registration)=Completed(Service.Coordination_released registration)) - "actual observation unregister did not join its resource owner"; - Hashtbl.clear replies - done) -;; - -let test_service_writer_fence_waits_for_canceled_protected_commit () = - let armed=Atomic.make false and entered=Atomic.make false and released=Atomic.make false in - let epoch_ms()= - if Atomic.get armed then (Atomic.set entered true; - while not(Atomic.get released)do Eio.Fiber.yield() done); - 1_700_000_000_000L in - Fun.protect ~finally:(fun()->Atomic.set released true)(fun()-> - with_observation_service ~epoch_ms(fun client ~call ~send ~wait ~replies ~notices:_ ~baseline ~owner ~registration-> - let module Wire=Logseq_db_worker_contract.Worker_pure_mutation in - let id n=uuid(Printf.sprintf "9f000000-0000-4000-8000-%012d" (9000+n)) in - let read owner n=call(query_request client !baseline owner registration(id n) - (Read.Outline_block{uuid=Test_support.authoritative_block_uuid;revision=None}))in - let revision=match read owner 1 with - |Mailbox.Completed(Service.Coordination_queried{answer=Read.Query_answer(_,Read.Outline_block _,Ok(Worker_pure_data.V2_present_block{revision;_}));_})->revision - |_->failwith "writer fence source read failed"in - let intent n title=Wire.create_mutation ~origin:(Uri.of_string "https://api.logseq.io") - ~user:"user-1" ~graph:owner.graph_id ~mutation_id:(id n) - (Wire.Save_block{block=Test_support.authoritative_block_uuid;title; - preconditions={blocks=[Test_support.authoritative_block_uuid,revision];pages=[];scopes=[]}}) - |>Result.get_ok |>fun intent->Wire.Mutation_intent intent in - let original=intent 2 "Protected writer actually committed" in - Atomic.set armed true; - let writer=send(Service.Coordination_mutation{registration;owner;request_id=id 3;intent=original}) in - wait(fun()->Atomic.get entered); - Mailbox.cancel client ~request_id:writer; - let scope=fst(List.find(fun((scope:Service.asset_scope),_)->scope.graph_id=owner.graph_id) - (Option.get !baseline).baseline.state.residents) in - let fence=send(Service.Coordination_quiesce_writers{registration;owner=scope;request_id=id 4}) in - let rejected=call(Service.Coordination_mutation{registration;owner;request_id=id 5;intent=intent 6 "Must not enter held DB"}) in - require(match rejected with Mailbox.Completed(Service.Coordination_mutation_result{outcome=Wire.Unknown;_})->true|_->false) - "writer fence admitted another durable operation into the held database"; - require(not(Hashtbl.mem replies fence)) "writer fence ACK preceded actual canceled protected commit"; - Atomic.set armed false;Atomic.set released true; - wait(fun()->Hashtbl.mem replies fence); - require(match Hashtbl.find replies fence with - |Mailbox.Completed(Service.Coordination_writers_quiesced value)-> - value.registration=registration && value.owner=scope && value.request_id=id 4 - |_->false) "writer fence failed to join actual protected I/O and callback barrier"; - require(match read owner 7 with Mailbox.Failed _->true - |Completed(Service.Coordination_queried{answer=Read.Query_answer(_,_,Error _);_})->true|_->false) - "old graph incarnation resumed admission after writer fence"; - ignore(call(Service.Client_command Return_to_graph_picker)); - ignore(call(Service.Client_command(Select_graph owner.graph_id))); - wait(fun()->List.exists(fun((next:Service.asset_scope),graph)->next.graph_id=owner.graph_id - && next.graph_generation<>owner.graph_generation && graph.Logseq_db_worker.phase=Graph_open) - (Option.get !baseline).baseline.state.residents); - let fresh=Option.get(Option.get !baseline).baseline.state.sync.selected in - require(match read fresh 8 with - |Mailbox.Completed(Service.Coordination_queried{answer=Read.Query_answer(_,Read.Outline_block _,Ok(Worker_pure_data.V2_present_block{value;_}));_})-> - value.block.title="Protected writer actually committed" - |_->false) "new incarnation lost actual canceled commit or inherited the old admission fence")) + in + require + (match initial with + | Completed + (Service.Coordination_observed + { answer = + Read.Query_answer + (_, Read.Outline_block _, Error Read.Response_budget_exceeded) + ; _ + }) -> true + | _ -> false) + "oversized initial sample did not report a read failure"; + let refresh n = + call + (Service.Coordination_refresh_observation + { registration; owner; observation_id = logical; request_id = rid n }) + in + let recover n text = + notices := Rrbvec.empty; + write n text; + wait (fun () -> + Rrbvec.exists + (function + | Service.Observation_invalidated value -> value.observation_id = logical + | _ -> false) + !notices); + require + (match refresh (n + 10) with + | Completed + (Service.Coordination_observed + { answer = + Read.Query_answer + ( _ + , Read.Outline_block _ + , Ok (Worker_pure_data.V2_present_block { value; _ }) ) + ; _ + }) -> value.block.title = text + | _ -> false) + "read failure silently discarded its observation interest" + in + recover 5 "Recovered initial observation"; + write 6 too_large; + require + (match refresh 17 with + | Completed + (Service.Coordination_observed + { answer = + Read.Query_answer + (_, Read.Outline_block _, Error Read.Response_budget_exceeded) + ; _ + }) -> true + | _ -> false) + "Refresh did not preserve its read error"; + recover 8 "Recovered refreshed observation"; + ignore (call (Service.Release_coordination registration))) ;; -let test_service_import_rejects_retired_registration () = - with_client(fun client-> - let replies=Hashtbl.create 16 and baseline=ref None in - Mailbox.on_event client(function - |Mailbox.Response{request_id;outcome;_}->Hashtbl.replace replies request_id outcome - |Push{payload=Service.Coordination_updates updates;_}->List.iter(function Service.Coordination_snapshot value->remember_baseline baseline value|_->())updates - |_->()); - let wait predicate= - let deadline=Unix.gettimeofday()+.4. in - while not(predicate()) && Unix.gettimeofday()id|_->failwith "typed Import fixture admission" in - wait(fun()->Hashtbl.mem replies id);Hashtbl.find replies id in - ignore(call(Service.Client_command(Restore_local_account{user_id="user-1"}))); - let issuer=Worker_effect_authority.issuer ~managed_origin:"https://api.logseq.io" ~user_id:"user-1" - ~runtime_epoch:(Mailbox.runtime_epoch client) ~worker_generation:(Mailbox.worker_generation client) |> Result.get_ok in - let module I=(val issuer:Worker_effect_authority.ISSUER) in - (match call(I.bootstrap_request() |> Result.get_ok)with - |Completed(Service.Coordination_registered value)->remember_baseline baseline value|_->failwith "typed Import baseline request failed"); - wait(fun()->Option.exists(fun value->List.exists(fun(_,graph)->graph.Logseq_db_worker.phase=Graph_open)value.Service.baseline.state.residents)!baseline); - let raw=Option.get !baseline in - let projection=match I.bootstrap(Service.Coordination_registered raw)with Ok(W.Ready value)->value|_->failwith "typed Import baseline invalid" in - let session=(List.hd(W.residents projection)).session in - let path=Filename.temp_file "journal-retired-import-synthetic-" ".png" in - Fun.protect ~finally:(fun()->if Sys.file_exists path then Sys.remove path)(fun()-> - Out_channel.with_open_bin path(fun output->output_string output "file"); - let source:Logseq_db_types.Asset_import.t={operation=uuid "9c000000-0000-4000-8000-000000001885"; - asset=uuid "9c000000-0000-4000-8000-000000001886";target=Test_support.authoritative_block_uuid;replace_reference=None; - local_mutation=uuid "9c000000-0000-4000-8000-000000001887";metadata_mutation=uuid "9c000000-0000-4000-8000-000000001888"; - source_file=path;title="Retired registration import";file_type="png"} in - let intent=W.asset_import_intent session source |> Result.get_ok in - let queued=I.request session ~request_id:(W.request_id source.operation)(W.Import intent) |> Result.get_ok in - ignore(call(Service.Release_coordination raw.registration)); - require((match call queued with Mailbox.Failed _->true|_->false)) - "retired coordination registration still prepared a queued typed Import")) +let test_service_unallocated_observation_cleanup_after_close () = + with_observation_service + (fun + client ~call ~send:_ ~wait ~replies:_ ~notices:_ ~baseline ~owner ~registration -> + let logical = uuid "9d000000-0000-4000-8000-000000000990" in + ignore (call (Service.Client_command Return_to_graph_picker)); + wait (fun () -> + match !baseline with + | Some value -> + value.baseline.state.sync.residents = [] && value.baseline.state.residents = [] + | _ -> false); + require + (match + call + (Service.Coordination_release_observation + { registration; owner; observation_id = logical }) + with + | Completed (Service.Coordination_observation_released value) -> + value.observation_id = logical + | _ -> false) + "unallocated observation could not settle after graph close"; + ignore (call (Service.Client_command (Select_graph Test_support.graph_uuid))); + wait (fun () -> + match !baseline with + | Some value -> + List.exists + (fun (_, state) -> state.Logseq_db_worker.phase = Graph_open) + value.baseline.state.residents + && Option.is_some value.baseline.state.sync.selected + | _ -> false); + let reopened = Option.get (Option.get !baseline).baseline.state.sync.selected in + let observe observation_id request_id = + Service.Coordination_observe + { registration + ; owner = reopened + ; observation_id + ; request_id + ; read = + observation_read client !baseline reopened registration Read.Graph_info + } + in + require + (match call (observe logical (uuid "9d000000-0000-4000-8000-000000000991")) with + | Failed _ -> true + | _ -> false) + "canceled logical ID allocated in a reopened generation"; + require + (match + call + (observe + (uuid "9d000000-0000-4000-8000-000000000992") + (uuid "9d000000-0000-4000-8000-000000000993")) + with + | Completed (Service.Coordination_observed _) -> true + | _ -> false) + "cleanup of one unknown ID incorrectly fenced the reopened graph"; + ignore (call (Service.Release_coordination registration))) ;; -(* This crosses actual Service/Db/runner lifetimes and reopens the source store. - A pure reducer cannot establish that source bytes or the Prepared checkpoint - survived closing their physical owners. No unprepared recovery is claimed. *) -(* The facade's unregister implementation owns actual cache/DB/IO joins; Core - has no physical lease result from which to reproduce an optimistic ACK. *) -let test_revoked_hidden_lease_blocks_unregister_ack () = - with_client(fun client-> - let replies=Hashtbl.create 16 and baseline=ref None and manager=ref None in - Mailbox.on_event client(function - |Mailbox.Response{request_id;outcome;_}->Hashtbl.replace replies request_id outcome - |Push{payload=Service.Coordination_updates updates;_}->List.iter(function - |Service.Coordination_snapshot value->remember_baseline baseline value|_->())updates - |Push{payload=Service.Client_state_changed value;_}->manager:=Some value.snapshot - |_->()); - let wait label predicate=let deadline=Unix.gettimeofday()+.4. in - while not(predicate()) && Unix.gettimeofday()id|_->failwith "retirement fixture admission" in - wait "actual retirement command did not settle"(fun()->Hashtbl.mem replies id);Hashtbl.find replies id in - ignore(call(Service.Client_command(Restore_local_account{user_id="user-1"}))); - (match call(Service.Register_coordination 7991L)with - |Completed(Service.Coordination_registered value)->remember_baseline baseline value|_->failwith "retirement baseline"); - ignore(call(Service.Client_command(Configure_resident_graphs[Test_support.graph_uuid;second_graph]))); - wait "two retirement residents did not open"(fun()->Option.exists(fun value-> - List.length value.Service.baseline.state.residents=2 && List.for_all(fun(_,state)->state.Logseq_db_worker.phase=Graph_open) - value.baseline.state.residents)!baseline); - let scope=List.find_map(fun((scope:Service.asset_scope),_)->if scope.graph_id=second_graph then Some scope else None) - (Option.get !baseline).baseline.state.residents |> Option.get in - let owner:Service.timeline_scope={startup={account_generation=scope.account.account_generation; - presentation_generation=scope.account.presentation_generation};graph_id=scope.graph_id;graph_generation=scope.graph_generation} in - let source_file=Filename.temp_file "journal-unregister-synthetic-" ".png" in - Fun.protect ~finally:(fun()->if Sys.file_exists source_file then Sys.remove source_file)(fun()-> - Out_channel.with_open_bin source_file(fun output->output_string output "file"); - let source:Logseq_db_types.Asset_import.t={operation=uuid "9c000000-0000-4000-8000-000000003885"; - asset=uuid "9c000000-0000-4000-8000-000000003886";target=Test_support.authoritative_block_uuid;replace_reference=None; - local_mutation=uuid "9c000000-0000-4000-8000-000000003887";metadata_mutation=uuid "9c000000-0000-4000-8000-000000003888"; - source_file;title="Retired hidden preview";file_type="png"} in - let imported=match call(Service.Import_asset_at(owner,source))with - |Completed(Service.Asset_imported(Ok value))->value|_->failwith "actual hidden import failed" in - let handle,path=imported.preview |> Option.get in - require(Sys.file_exists path)"physical preview missing before revocation"; - ignore(call(Service.Client_command(Reconcile_authenticated_user{user_id=None}))); - wait "actual account authority was not revoked"(fun()->Option.exists(fun(value:Service.snapshot)-> - not value.startup.authenticated && value.startup.account_generation<>scope.account.account_generation)!manager); - require(match call(Service.Release_coordination 7991L)with Mailbox.Failed _->true|_->false) - "unregister ACK escaped while an actual old hidden native lease remained live"; - require(Sys.file_exists path)"failed unregister discarded retained renderer bytes"; - require(call(Service.Release_asset_file{scope;handle})=Completed Service.Client_command_completed) - "revoked exact hidden lease lost physical cleanup authority"; - require(call(Service.Release_coordination 7991L)=Completed(Service.Coordination_released 7991L)) - "unregister did not retry its original physical cleanup after actual release"; - require(not(Sys.file_exists path))"successful unregister left old scoped resource bytes live")) +let test_service_unknown_recovery_is_read_only ?(large = false) () = + let module D = Logseq_overlay_db.Database in + let module Types = Logseq_overlay_db.Types in + let module Wire = Logseq_db_worker_contract.Worker_pure_mutation in + let mutation_id = uuid "9f000000-0000-4000-8000-000000000900" in + let encoded = ref "" + and pending = ref None in + let prepare support = + Eio_main.run (fun _ -> + Eio.Switch.run (fun sw -> + let inspection = + D.inspect_mirror + ~application_support_directory:support + ~graph_id:Test_support.graph_uuid + |> Result.get_ok + in + let database = + D.open_ + ~sw + (Test_support.dependencies ~behavior:"pending recovery fixture") + inspection + ~graph_name:"pending recovery" + |> Result.get_ok + in + Fun.protect + ~finally:(fun () -> ignore (D.close database)) + (fun () -> + let snapshot = D.current_snapshot database |> Result.get_ok in + let revision = + Fun.protect + ~finally:(fun () -> D.release_snapshot snapshot) + (fun () -> + match + D.get_outline_blocks + snapshot + [ Test_support.authoritative_block_uuid ] + |> Result.get_ok + with + | [ Types.Present_block { revision; _ } ] -> revision + | _ -> failwith "pending fixture block missing") + in + let preconditions = + Worker_pure_data. + { blocks = + [ ( Test_support.authoritative_block_uuid + , Types.Block_state_revision.to_string revision ) + ] + ; pages = [] + ; scopes = [] + } + in + let command, mutation = + if large + then ( + let children = + List.init 5 (fun n -> + Worker_pure_data. + { uuid = + uuid (Printf.sprintf "9f000000-0000-4000-8000-%012d" (910 + n)) + ; title = String.make 60000 '"' + ; children = [] + }) + in + let root = + Worker_pure_data. + { uuid = uuid "9f000000-0000-4000-8000-000000000909" + ; title = "Never automatically replay" + ; children + } + in + let tree = + Types. + { uuid = root.uuid + ; title = root.title + ; children = + List.map + (fun (child : Worker_pure_data.block_tree) -> + { uuid = child.uuid; title = child.title; children = [] }) + children + } + in + ( Wire.Insert_blocks + { parent = Test_support.authoritative_block_uuid + ; roots = [ root ] + ; preconditions + } + , Types.Insert_blocks + { mutation_id + ; parent = Test_support.authoritative_block_uuid + ; tree + ; asset = None + } )) + else + ( Wire.Save_block + { block = Test_support.authoritative_block_uuid + ; title = "Never automatically replay" + ; preconditions + } + , Types.Save_block + { mutation_id + ; block = Test_support.authoritative_block_uuid + ; title = "Never automatically replay" + } ) + in + let wire = + Wire.create_mutation + ~mutation_id + ~origin:(Uri.of_string "https://api.logseq.io") + ~user:"user-1" + ~graph:Test_support.graph_uuid + command + |> Result.get_ok + |> fun intent -> Wire.Mutation_intent intent + in + pending := Some wire; + encoded := Wire.encode wire; + if large + then + require + (String.length !encoded > Worker_pure_data.maximum_response_bytes + && String.length !encoded <= Worker_pure_data.maximum_request_bytes) + "large recovery fixture must be a valid intent exceeding the ordinary \ + read cap"; + let expected = + D.write_precondition + ~blocks:[ Test_support.authoritative_block_uuid, revision ] + ~pages:[] + ~scopes:[] + |> Result.get_ok + in + ignore + (D.persist_local_intent + database + ~namespace:(Wire.namespace wire) + ~payload:!encoded + ~expected + mutation + |> Result.get_ok)))) + in + with_observation_service + ~prepare + (fun + client + ~call + ~send:_ + ~wait:_ + ~replies:_ + ~notices:_ + ~baseline + ~owner + ~registration + -> + let request_id = uuid "9f000000-0000-4000-8000-000000000901" in + let recover () = + match + call + (Service.Coordination_recover_mutations + { registration + ; owner + ; request_id + ; after = None + ; maximum_items = 8 + ; maximum_bytes = (if large then Wire.maximum_recovery_bytes else 16384) + }) + with + | Mailbox.Completed (Service.Coordination_mutations_recovered { page; _ }) -> + page + | _ -> failwith "pending recovery failed" + in + let page = recover () in + require + (Rrbvec.length page.items = 1 + && Wire.encode (Rrbvec.nth page.items 0).intent = !encoded + && (Rrbvec.nth page.items 0).outcome = Wire.Unknown) + "recovery inferred a pending write outcome"; + require + (match + call + (Service.Coordination_acknowledge_mutation + { registration + ; owner + ; request_id + ; presentation = (Option.get !baseline).baseline.presentation + ; intent = Option.get !pending + ; original = None + }) + with + | Mailbox.Failed _ -> true + | _ -> false) + "Unknown intent could be acknowledged away"; + require + (Rrbvec.length (recover ()).items = 1) + "Unknown ACK or recovery dropped pending identity"; + require + (match + call + (query_request + client + !baseline + owner + registration + request_id + (Read.Outline_block + { uuid = Test_support.authoritative_block_uuid; revision = None })) + with + | Mailbox.Completed + (Service.Coordination_queried + { answer = + Read.Query_answer + ( _ + , Read.Outline_block _ + , Ok (Worker_pure_data.V2_present_block { value; _ }) ) + ; _ + }) -> value.block.title <> "Never automatically replay" + | _ -> false) + "recovery automatically executed Unknown intent") ;; -let test_prepared_import_survives_worker_lifetime () = - with_client_sessions (fun start -> - let source_file=Filename.temp_file "journal-restarted-import-synthetic-" ".png" in - Fun.protect ~finally:(fun()->if Sys.file_exists source_file then Sys.remove source_file)(fun()-> - Out_channel.with_open_bin source_file(fun output->output_string output "file"); - let source:Logseq_db_types.Asset_import.t= - {operation=uuid "9c000000-0000-4000-8000-000000002885"; - asset=uuid "9c000000-0000-4000-8000-000000002886"; - target=Test_support.authoritative_block_uuid;replace_reference=None; - local_mutation=uuid "9c000000-0000-4000-8000-000000002887"; - metadata_mutation=uuid "9c000000-0000-4000-8000-000000002888"; - source_file;title="Prepared across actual worker restart";file_type="png"} in - let lifetime saved = - let client=start() in - let replies=Hashtbl.create 16 and baseline=ref None in - Mailbox.on_event client(function - |Mailbox.Response{request_id;outcome;_}->Hashtbl.replace replies request_id outcome - |Push{payload=Service.Coordination_updates updates;_}-> - List.iter(function Service.Coordination_snapshot value->remember_baseline baseline value|_->())updates - |_->()); - let wait predicate= - let deadline=Unix.gettimeofday()+.5. in - while not(predicate()) && Unix.gettimeofday()id|_->failwith "restarted Import admission" in - wait(fun()->Hashtbl.mem replies id);Hashtbl.find replies id in - ignore(call(Service.Client_command(Restore_local_account{user_id="user-1"}))); - let issuer=Worker_effect_authority.issuer ~managed_origin:"https://api.logseq.io" ~user_id:"user-1" - ~runtime_epoch:(Mailbox.runtime_epoch client) ~worker_generation:(Mailbox.worker_generation client) |> Result.get_ok in - let module I=(val issuer:Worker_effect_authority.ISSUER) in - (match call(I.bootstrap_request() |> Result.get_ok)with - |Completed(Service.Coordination_registered value)->remember_baseline baseline value - |_->failwith "restarted Import bootstrap failed"); - wait(fun()->Option.exists(fun value->List.exists(fun(_,graph)-> - graph.Logseq_db_worker.phase=Graph_open)value.Service.baseline.state.residents)!baseline); - let projection=match I.bootstrap(Service.Coordination_registered(Option.get !baseline))with - |Ok(W.Ready value)->value|_->failwith "restarted Import baseline invalid" in - let session=(List.find(fun resident->W.graph_id resident.W.session=Test_support.graph_uuid)(W.residents projection)).session in - let sequence=ref 0 in - let request : type a. a W.request -> a W.reply = fun operation -> - incr sequence; - let request_id=W.request_id(uuid(Printf.sprintf "9c000000-0000-4000-8000-%012d" (3000 + !sequence))) in - let raw=I.request session ~request_id operation |> Result.get_ok in - match call raw with - |Completed response->I.reply session ~request_id operation response |> Result.get_ok - |Failed message->failwith("restarted Import request failed: "^message) - |_->failwith "restarted Import request interrupted" in - let children ()= - let budget=W.read_budget session ~maximum_items:100 ~maximum_bytes:65536 |> Result.get_ok in - match request(W.Query(budget,W.Children{parent=source.target;after=None}))with - |W.Answered(Ok value)->require(value.next=None) "synthetic Import child count was partial";value.items - |_->failwith "restarted Import children unavailable" in - let before=children() in - let intent=match saved with None->W.asset_import_intent session source |> Result.get_ok - |Some(encoded,_,_)->W.restore_asset_import_intent session encoded |> Result.get_ok in - let imported=match request(W.Import intent)with - |W.Answered(Ok(W.Imported imported))->imported - |W.Answered(Error(W.Service_failed message))->failwith("restarted Import failed: "^message) - |_->failwith "restarted Import did not reconcile Prepared" in - let preview=W.import_preview imported |> Option.get in - require(Sys.file_exists(W.file_path preview)) "restarted Import lost physically staged source"; - require((W.import_data imported).operation=source.operation && (W.import_data imported).target=source.target) - "restarted Import replaced original operation/target"; - let after=children() in - require(List.exists(fun block->W.block_uuid block=source.asset)after) - "actual Import did not materialize its asset child"; - (match saved with - |None->require(List.length after=List.length before+1) "first Import duplicated asset children" - |Some(_,asset,count)-> - require((W.import_data imported).asset=asset) "restarted Import changed original descriptor"; - require(List.length before=count && List.length after=count) "restarted Import replayed local insertion"); - require(request(W.Release_file preview)=W.Answered(Ok())) "restarted Import physical release was not acknowledged"; - W.encode_asset_import_intent intent,(W.import_data imported).asset,List.length after +let prepare_reusable_asset support = + let module D = Logseq_overlay_db.Database in + let module O = Logseq_overlay_db.Types in + Eio_main.run (fun _ -> + Eio.Switch.run (fun sw -> + let inspection = + D.inspect_mirror + ~application_support_directory:support + ~graph_id:Test_support.graph_uuid + |> Result.get_ok + in + let database = + D.open_ + ~sw + (Test_support.dependencies ~behavior:"reusable asset") + inspection + ~graph_name:"reusable asset" + |> Result.get_ok in - let saved=lifetime None in - Sys.remove source_file; - ignore(lifetime(Some saved)))) + Fun.protect + ~finally:(fun () -> ignore (D.close database)) + (fun () -> + let version = + Logseq_db_types.Asset_descriptor.version + ~checksum:(String.make 64 'a') + ~file_type:"png" + |> Result.get_ok + in + let parent = Test_support.page_uuid in + let expected = + Test_support.insert_precondition database ~parent ~behavior:"reusable asset" + in + ignore + (D.commit_local + database + ~expected + (O.Insert_blocks + { mutation_id = Test_support.mutation_uuid 981 + ; parent + ; tree = + { uuid = Test_support.mutation_uuid 980 + ; title = "Reusable attachment" + ; children = [] + } + ; asset = Some { replace_reference = None; version; size = 4L } + }) + |> Result.get_ok)))) ;; -let test_service_observation_lifecycle () = +let test_service_durable_recovery_and_ack () = + let module Wire = Logseq_db_worker_contract.Worker_pure_mutation in + let changes = ref 0 in + let observe_push = function + | Service.Graph_push (Worker_pure_data.Changes_available _) -> incr changes + | Graph_pushes values -> + List.iter + (function + | _, Worker_pure_data.Changes_available _ -> incr changes + | _ -> ()) + values + | _ -> () + in with_observation_service - (fun client ~call ~send:_ ~wait ~replies:_ ~notices ~baseline ~owner ~registration -> - let logical = uuid "9d000000-0000-4000-8000-000000000001" in - let rid n=uuid(Printf.sprintf "9d000000-0000-4000-8000-%012d" n) in - let source=observation_read client !baseline owner registration - (Read.Outline_block{uuid=Test_support.authoritative_block_uuid;revision=None}) in - let observe = - Service.Coordination_observe - { registration - ; owner - ; observation_id = logical - ; request_id=rid 2 - ; read=source - } - in - let revision = - match call observe with - | Completed - (Service.Coordination_observed - { epoch = 0L - ; answer=Read.Query_answer(_,Read.Outline_block _,Ok(Worker_pure_data.V2_present_block { revision; _ })) + ~prepare:prepare_reusable_asset + ~observe_push + (fun + client ~call ~send:_ ~wait ~replies:_ ~notices:_ ~baseline ~owner ~registration -> + let id n = uuid (Printf.sprintf "9f000000-0000-4000-8000-%012d" n) in + let owner = ref owner in + let presentation () = (Option.get !baseline).Service.baseline.presentation in + let read n = + match + call + (query_request + client + !baseline + !owner + registration + (id n) + (Read.Outline_block + { uuid = Test_support.authoritative_block_uuid; revision = None })) + with + | Mailbox.Completed + (Service.Coordination_queried + { answer = + Read.Query_answer + ( _ + , Read.Outline_block _ + , Ok (Worker_pure_data.V2_present_block { revision; value }) ) ; _ - }) -> revision - | _ -> failwith "real initial observation did not sample its predecessor" + }) -> revision, value.block.title + | _ -> failwith "durable lifecycle source read failed" + in + let recover ?(maximum_bytes = 16384) ?(maximum_items = 1) n after = + call + (Service.Coordination_recover_mutations + { registration + ; owner = !owner + ; request_id = id n + ; after + ; maximum_items + ; maximum_bytes + }) + in + let page n after = + match recover n after with + | Mailbox.Completed + (Service.Coordination_mutations_recovered { request_id; page }) + when request_id = id n -> + require + (Wire.recovery_page_bytes page <= 16384) + "recovery exceeded complete byte budget"; + page + | _ -> failwith "real durable recovery did not return its bounded page" in require - (match call observe with - | Completed (Service.Coordination_observed { epoch = 0L; _ }) -> true + (Rrbvec.is_empty (page 1 None).items) + "new fixture has unexpected pending intents"; + let rev, _ = read 2 in + let intent n revision title = + Wire.create_mutation + ~origin:(Uri.of_string "https://api.logseq.io") + ~user:"user-1" + ~graph:!owner.graph_id + ~mutation_id:(id n) + (Wire.Save_block + { block = Test_support.authoritative_block_uuid + ; title + ; preconditions = + { blocks = [ Test_support.authoritative_block_uuid, revision ] + ; pages = [] + ; scopes = [] + } + }) + |> Result.get_ok + |> fun intent -> Wire.Mutation_intent intent + in + let first = intent 10 rev "Durable first" in + let mutate n intent = + match + call + (Service.Coordination_mutation + { registration; owner = !owner; request_id = id n; intent }) + with + | Mailbox.Completed (Service.Coordination_mutation_result { outcome; _ }) -> + outcome + | _ -> failwith "durable lifecycle mutation did not settle" + in + let original = + match mutate 11 first with + | Wire.Committed response -> Wire.committed_point first response |> Result.get_ok + | _ -> failwith "first durable fixture write failed" + in + let rejected = intent 20 rev "Stale rejected" in + require + (match mutate 21 rejected with + | Wire.Rejected _ -> true | _ -> false) - "duplicate Observe did not retain original registration"; - let retry_request=rid 10 in + "stale durable write was not rejected"; + let current, _ = read 3 in + let second = intent 30 current "Durable second" in require - (match call(Service.Coordination_observe {registration;owner;observation_id=logical; - request_id=retry_request;read=source}) with - | Completed(Service.Coordination_observed {epoch=0L;request_id;_})->request_id=retry_request - | _->false) - "retried Observe lost current RPC correlation"; + (match mutate 31 second with + | Wire.Committed _ -> true + | _ -> false) + "second durable write failed"; + let collect n = + let rec loop n after count items = + require (count < 8) "recovery continuation did not terminate"; + let current = page n after in + require (Rrbvec.length current.items <= 1) "recovery exceeded item budget"; + let items = Rrbvec.append items current.items in + match current.next with + | None -> items + | Some next -> loop (n + 1) (Some next) (count + 1) items + in + loop n None 0 Rrbvec.empty + in + let before = read 4 in + let recovered = collect 40 in require - (call - (Service.Client_command - (Configure_resident_graphs [ Test_support.graph_uuid; second_graph ])) - = Mailbox.Completed Service.Client_command_completed) - "synthetic resident configuration failed"; - wait (fun () -> - match !baseline with - | Some value -> - List.length value.baseline.state.residents = 2 - && List.for_all - (fun (_, graph) -> - graph.Logseq_db_worker.phase = Logseq_db_worker.Graph_open) - value.baseline.state.residents - | None -> false); + (Rrbvec.length recovered = 3) + "bounded recovery skipped committed or rejected intent"; + let first_page = page 44 None in + let byte_budget = Wire.recovery_page_bytes first_page in + (match recover ~maximum_bytes:byte_budget ~maximum_items:3 45 None with + | Mailbox.Completed (Service.Coordination_mutations_recovered { page; _ }) -> + require + (Rrbvec.length page.items = 1 && page.next = first_page.next) + "byte-bounded prefix skipped the next complete row or forged completeness" + | _ -> failwith "byte-bounded recovery prefix failed"); + require (read 5 = before) "read-only recovery replayed a mutation"; require - (call (Service.Client_command (Focus_resident_graph second_graph)) - = Mailbox.Completed Service.Client_command_completed) - "synthetic focus change failed"; + (match recover ~maximum_bytes:1 6 None with + | Mailbox.Failed _ -> true + | _ -> false) + "oversized first recovery item was silently skipped"; + let old_presentation = presentation () in + let old_generation = !owner.graph_generation in + ignore (call (Service.Client_command Return_to_graph_picker)); + ignore (call (Service.Client_command (Select_graph !owner.graph_id))); wait (fun () -> match !baseline with - | Some value -> - Option.exists - (fun (selected : Service.timeline_scope) -> selected.graph_id = second_graph) - value.baseline.state.sync.selected + | Some b -> + List.exists + (fun ((scope : Service.asset_scope), graph) -> + scope.graph_id = !owner.graph_id + && scope.graph_generation <> old_generation + && graph.Logseq_db_worker.phase = Logseq_db_worker.Graph_open) + b.baseline.state.residents | None -> false); + owner := Option.get (Option.get !baseline).baseline.state.sync.selected; + let restored = collect 50 in + require (Rrbvec.length restored = 3) "reopening lost unsettled durable intents"; + let item = + Rrbvec.fold_left + (fun found (item : Wire.recovery_item) -> + if item.intent = first then Some item else found) + None + restored + |> Option.get + in + let restored_point = + match item.outcome with + | Wire.Committed proof -> Wire.committed_point first proof |> Result.get_ok + | _ -> failwith "recovered commit lost original proof" + in + require + (restored_point = original) + "recovery rewrote original commit generation or revisions"; + let ack n presentation intent original = + call + (Service.Coordination_acknowledge_mutation + { registration + ; owner = !owner + ; presentation + ; request_id = id n + ; intent + ; original + }) + in + let proof = original.generation, original.before, original.after in + require + (match ack 60 old_presentation first (Some proof) with + | Mailbox.Failed _ -> true + | _ -> false) + "old presentation acknowledged a committed intent"; + require + (match ack 61 (presentation ()) first None with + | Mailbox.Failed _ -> true + | _ -> false) + "committed intent was acknowledged as a rejection"; + require + (match ack 62 (presentation ()) first (Some ("wrong", "wrong", "wrong")) with + | Mailbox.Failed _ -> true + | _ -> false) + "wrong immutable proof acknowledged intent"; + require (Rrbvec.length (collect 70) = 3) "invalid ACK removed recovery ownership"; require (match call - (Service.Coordination_refresh_observation + (Service.Coordination_confirm { registration - ; owner - ; observation_id = logical - ; request_id = (rid 9) + ; owner = !owner + ; presentation = presentation () + ; request_id = id 80 + ; intent = first + ; original = proof + ; read = + observation_read + client + !baseline + !owner + registration + ~maximum_items:Worker_pure_data.default_page_size + ~maximum_bytes:(Option.get !baseline).limits.response_budget_bytes + (Read.Block_summary + { block = Test_support.authoritative_block_uuid; after = None }) }) with - | Completed - (Service.Coordination_observed - { answer=Read.Query_answer(_,Read.Outline_block _,Ok(Worker_pure_data.V2_present_block { revision = actual; _ })) - ; _ - }) -> actual = revision + | Mailbox.Completed + (Service.Coordination_confirmed + { answer = Read.Query_answer (_, Read.Block_summary _, Ok _); _ }) -> + true | _ -> false) - "focus retired a surviving resident observation"; - let write n revision title = - call(mutation_request client !baseline owner registration ~request_id:(rid n) ~mutation_id:(rid(n+100)) - (W.Save_block{block=Test_support.authoritative_block_uuid;title; - preconditions={blocks=[Test_support.authoritative_block_uuid,revision];pages=[];scopes=[]}})) in + "reopened intent could not confirm its original proof"; + let acknowledged n result = + require + (match result with + | Mailbox.Completed + (Service.Coordination_mutation_acknowledged { request_id }) -> + request_id = id n + | _ -> false) + "durable ACK did not return its exact completion" + in + acknowledged 81 (ack 81 (presentation ()) first (Some proof)); + acknowledged 82 (ack 82 (presentation ()) rejected None); + acknowledged 84 (ack 84 (presentation ()) rejected None); + acknowledged 83 (ack 83 (presentation ()) first (Some proof)); require - (match write 3 revision "Observed write" with + (Rrbvec.length (collect 90) = 1) + "ACK removed the wrong intent or retained settled rows"; + let revision_before = read 7 in + require + (match mutate 100 first with + | Wire.Committed response -> Wire.committed_point first response = Ok original + | _ -> false) + "ACK deleted immutable original commit proof"; + require + (read 8 = revision_before) + "same-intent reconciliation after ACK replayed the write"; + let block = Test_support.reference_source_uuid in + let revision = + match + call + (query_request + client + !baseline + !owner + registration + (id 110) + (Read.Outline_block { uuid = block; revision = None })) + with + | Completed + (Service.Coordination_queried + { answer = + Read.Query_answer + ( _ + , Read.Outline_block _ + , Ok (Worker_pure_data.V2_present_block { revision; _ }) ) + ; _ + }) -> revision + | _ -> failwith "asset reference source missing" + in + let asset = Test_support.mutation_uuid 980 in + let linked = + Wire.create_mutation + ~origin:(Uri.of_string "https://api.logseq.io") + ~user:"user-1" + ~graph:!owner.graph_id + ~mutation_id:(id 111) + (Wire.Set_asset_reference + { block + ; previous = None + ; asset + ; preconditions = { blocks = [ block, revision ]; pages = []; scopes = [] } + }) + |> Result.get_ok + |> fun value -> Wire.Mutation_intent value + in + let before = !changes in + require + (match mutate 112 linked with + | Wire.Committed _ -> true + | _ -> false) + "real Service did not execute asset reference mutation"; + wait (fun () -> !changes > before); + require + (match + call + (query_request + client + !baseline + !owner + registration + (id 113) + (Read.Assets { roots = [ block ]; recursive = false; after = None })) + with | Completed - (Service.Coordination_mutation_result{outcome=Logseq_db_worker_contract.Worker_pure_mutation.Committed _;_}) -> true + (Service.Coordination_queried + { answer = Read.Query_answer (_, Read.Assets _, Ok page); _ }) -> + List.exists + (fun (value : Logseq_db_types.Asset_descriptor.t) -> value.uuid = asset) + page.items | _ -> false) - "observation synthetic mutation failed"; - wait (fun () -> - Rrbvec.exists - (function - | Service.Observation_invalidated value -> - value.registration_id = registration - && value.observation_id = logical - && value.epoch > 0L - | _ -> false) - !notices); - let refreshed = + "real typed Assets read missed committed reference") +;; + +let test_service_changefeed_has_one_owner () = + let pulses = ref 0 + and resyncs = ref 0 in + let observe_push = function + | Service.Graph_push (Worker_pure_data.Changes_available _) -> incr pulses + | Graph_push (Worker_pure_data.Resync_required _) -> + incr pulses; + incr resyncs + | Graph_pushes values -> + List.iter + (function + | _, Worker_pure_data.Changes_available _ -> incr pulses + | _, Worker_pure_data.Resync_required _ -> + incr pulses; + incr resyncs) + values + | _ -> () + in + with_observation_service + ~change_max_items:16 + ~observe_push + (fun + client ~call ~send:_ ~wait ~replies:_ ~notices:_ ~baseline ~owner ~registration -> + let id n = uuid (Printf.sprintf "9d000000-0000-4000-8000-%012d" (6000 + n)) in + let active = ref registration in + let query registration n query = call - (Service.Coordination_refresh_observation - { registration - ; owner - ; observation_id = logical - ; request_id = (rid 4) - }) + (query_request + client + !baseline + owner + registration + (id n) + ~maximum_items:32 + query) in - let current_revision = - match refreshed with + let generation = + match query registration 0 Read.Graph_info with | Completed - (Service.Coordination_observed - { epoch - ; answer=Read.Query_answer(_,Read.Outline_block _,Ok(Worker_pure_data.V2_present_block { value; revision })) + (Service.Coordination_queried + { answer = Read.Query_answer (_, Read.Graph_info, Ok info); _ }) -> + info.generation + | _ -> failwith "changefeed fixture graph info" + in + let write n title = + let revision = + match + query + !active + (n + 100) + (Read.Outline_block + { uuid = Test_support.authoritative_block_uuid; revision = None }) + with + | Completed + (Service.Coordination_queried + { answer = + Read.Query_answer + ( _ + , Read.Outline_block _ + , Ok (Worker_pure_data.V2_present_block { revision; _ }) ) + ; _ + }) -> revision + | _ -> failwith "changefeed write revision" + in + let before = !pulses in + require + (match + call + (mutation_request + client + !baseline + owner + !active + ~request_id:(id n) + ~mutation_id:(id n) + (W.Save_block + { block = Test_support.authoritative_block_uuid + ; title + ; preconditions = + { blocks = [ Test_support.authoritative_block_uuid, revision ] + ; pages = [] + ; scopes = [] + } + })) + with + | Completed + (Service.Coordination_mutation_result + { outcome = Logseq_db_worker_contract.Worker_pure_mutation.Committed _ + ; _ + }) -> true + | _ -> false) + "real changefeed write failed"; + wait (fun () -> !pulses > before) + in + let pull registration n after : Read.change_batch = + match query registration n (Read.Changes { generation; after }) with + | Completed + (Service.Coordination_queried + { request_id + ; answer = Read.Query_answer (_, Read.Changes _, Ok batch) ; _ }) -> - require - (epoch > 0L && value.block.title = "Observed write") - "Refresh lost epoch/data"; - revision - | _ -> failwith "real observation refresh failed" + require (request_id = id n) "cached Changes kept an old RPC correlation"; + batch + | _ -> failwith "scoped coordinated changefeed pull failed" in - let finalize=Service.Coordination_finalize_observation{registration;owner;observation_id=logical} in - require(match call finalize with Failed _->true|_->false) - "actual Service finalized a live observation before physical release"; - let release = - Service.Coordination_release_observation - { registration; owner; observation_id = logical } + let ack registration n batch = + call + (Service.Coordination_acknowledge_changes + { registration; owner; request_id = id n; batch }) in + let acknowledged registration n batch = + require + (match ack registration n batch with + | Completed + (Service.Coordination_changes_acknowledged + { request_id; result = Ok value; _ }) -> + request_id = id n + && Read.change_ack_generation value = batch.W.generation + && Read.change_ack_through value = batch.through + | _ -> false) + "owned applied window did not receive its actual ACK" + in + let data (batch : W.change_batch) = batch.through, batch.windows, batch.next in + let empty = pull registration 12 None in require - (match call release with - | Completed (Service.Coordination_observation_released _) -> true - | _ -> false) - "release did not acknowledge listener/read retirement"; - require - (match call release with - | Completed (Service.Coordination_observation_released _) -> true - | _ -> false) - "release ACK was not idempotent"; + (empty.windows = [] && empty.next = None) + "fresh physical changefeed was not empty"; + acknowledged registration 13 empty; + write 20 "Owned change window A"; + let first = pull registration 1 None in + require (first.windows <> []) "actual write produced no retained change window"; + write 21 "Owned change window B"; require - (match call observe with - | Failed _ -> true - | _ -> false) - "retired logical observation was allocated again"; - require(match call finalize with Completed(Service.Coordination_observation_finalized value)-> - value.registration=registration && value.owner=owner && value.observation_id=logical|_->false) - "actual creator/release ACK did not permit observation finalization"; - require(match call finalize with Completed(Service.Coordination_observation_finalized _)->true|_->false) - "uncertain finalization retry was not idempotent"; - notices := Rrbvec.empty; + (data (pull registration 4 None) = data first) + "unapplied batch was replaced"; + let foreign = Int64.succ registration in require - (match write 5 current_revision "After observation release" with - | Completed - (Service.Coordination_mutation_result{outcome=Logseq_db_worker_contract.Worker_pure_mutation.Committed _;_}) -> true + (match query foreign 5 (Read.Changes { generation; after = None }) with + | Completed (Service.Coordination_query_revoked value) -> + value.registration = foreign && value.owner = owner && value.request_id = id 5 | _ -> false) - "post-release synthetic write did not commit"; - for _ = 1 to 12 do - Mailbox.Private.deliver client ~max_events:64; - Thread.delay 0.001 - done; + "another binding remained an active second changefeed owner"; + acknowledged registration 6 first; + acknowledged registration 7 first; + let second = pull registration 8 (Some first.through) in require - (Rrbvec.is_empty !notices) - "released observation published a later invalidation"; - let cancelled = uuid "9d000000-0000-4000-8000-000000000006" in + (second.through <> first.through && second.windows <> []) + "ACK lost the second actual write window"; + ignore (call (Service.Release_coordination registration)); + let replacement = Int64.succ registration in require - (match - call - (Service.Coordination_release_observation - { registration; owner; observation_id = cancelled }) - with - | Completed (Service.Coordination_observation_released _) -> true + (match call (Service.Register_coordination replacement) with + | Completed (Service.Coordination_registered _) -> true | _ -> false) - "release before registration did not create a fence"; + "replacement binding did not register"; + active := replacement; + let adopted = pull replacement 9 None in require - (match - call - (Service.Coordination_observe - { registration - ; owner - ; observation_id = cancelled - ; request_id=rid 2 - ; read=source - }) - with + (data adopted = data second) + "binding retirement discarded an unapplied physical window"; + require + (match ack registration 10 second with | Failed _ -> true | _ -> false) - "late observation escaped its pre-registration release"; - ignore (call (Service.Release_coordination registration))) -;; - -let test_service_empty_observation_is_retained () = - with_observation_service - (fun - _client - ~call - ~send:_ - ~wait - ~replies:_ - ~notices - ~baseline - ~owner - ~registration - -> - let target = uuid "9d000000-0000-4000-8000-000000005000" in - let logical = uuid "9d000000-0000-4000-8000-000000005001" in - let rid n=uuid(Printf.sprintf "9d000000-0000-4000-8000-%012d" n) in - let point=observation_read _client !baseline owner registration - (Read.Outline_block{uuid=target;revision=None}) in + "old binding consumed the replacement window"; + acknowledged replacement 11 adopted; + write 3000 "held retention window"; + let held = pull replacement 3001 None in + for n = 3002 to 3258 do + write n (Printf.sprintf "retention change %d" n) + done; require - (match - call - (Service.Coordination_observe - { registration - ; owner - ; observation_id = logical - ; request_id=rid 5002 - ; read=point - }) - with + (!resyncs > 0) + "unacknowledged physical change windows grew without explicit resync"; + require + (match ack replacement 3260 held with | Completed - (Service.Coordination_observed - { answer=Read.Query_answer(_,Read.Outline_block _,Ok(Worker_pure_data.V2_missing_block { uuid; _ })) - ; _ - }) -> uuid = target + (Service.Coordination_changes_acknowledged + { result = Error (Read.Resync_required _); _ }) -> true | _ -> false) - "empty initial sample was not observable"; - let read n query=call(query_request _client !baseline owner registration (rid n) query) in - let page_revision=match read 5003(Read.Page{uuid=Test_support.page_uuid;revision=None})with - |Completed(Service.Coordination_queried{answer=Read.Query_answer(_,Read.Page _,Ok(Worker_pure_data.V2_present_page{revision;_}));_})->revision - |_->failwith "observation insert page revision unavailable" in - let scope_revision=match read 5004(Read.Child_members{parent=Test_support.page_uuid;revision=None;after=None})with - |Completed(Service.Coordination_queried{answer=Read.Query_answer(_,Read.Child_members _,Ok children);_})->children.scope_revision - |_->failwith "observation insert children revision unavailable" in - require(match call(mutation_request _client !baseline owner registration ~request_id:(rid 5005) ~mutation_id:(rid 5006) - (W.Insert_blocks{parent=Test_support.page_uuid; - roots=[{uuid=target;title="First observed block";children=[]}]; - preconditions={blocks=[];pages=[Test_support.page_uuid,page_revision]; - scopes=[Worker_pure_data.V2_children_scope Test_support.page_uuid,scope_revision]}})) with - |Completed(Service.Coordination_mutation_result{outcome=Logseq_db_worker_contract.Worker_pure_mutation.Committed _;_})->true - |_->false) "synthetic first insertion did not commit"; - wait (fun () -> - Rrbvec.exists - (function - | Service.Observation_invalidated value -> value.observation_id = logical - | _ -> false) - !notices); + "retention overflow fabricated ACK of discarded physical history"; + let recovered = pull replacement 3261 None in + require + (List.for_all + (fun (window : Worker_pure_data.change_window) -> + not + (List.exists + (fun (old : Worker_pure_data.change_window) -> old.id = window.id) + held.windows)) + recovered.windows) + "overflow retained the discarded owned window"; + acknowledged replacement 3262 recovered; require (match - call - (Service.Coordination_refresh_observation - { registration - ; owner - ; observation_id = logical - ; request_id = (rid 5007) - }) + query + replacement + 3263 + (Read.Changes { generation; after = Some held.through }) with | Completed - (Service.Coordination_observed - { answer=Read.Query_answer(_,Read.Outline_block _,Ok(Worker_pure_data.V2_present_block { value; _ })) + (Service.Coordination_queried + { answer = + Read.Query_answer (_, Read.Changes _, Error (Read.Resync_required _)) ; _ - }) -> - value.block.uuid = target && value.block.title = "First observed block" + }) -> true | _ -> false) - "missing sample discarded the interest before first value"; - ignore (call (Service.Release_coordination registration))) -;; - -let test_service_observation_read_failure_retains_interest () = - with_observation_service - (fun - _client - ~call - ~send:_ - ~wait - ~replies:_ - ~notices - ~baseline - ~owner - ~registration - -> - let logical = uuid "9e000000-0000-4000-8000-000000000001" in - let rid n=uuid(Printf.sprintf "9e000000-0000-4000-8000-%012d" n) in - let query=Read.Outline_block{uuid=Test_support.authoritative_block_uuid;revision=None} in - let read n=match call(query_request _client !baseline owner registration (rid n) - ~maximum_bytes:Worker_pure_data.maximum_response_bytes query)with - |Completed(Service.Coordination_queried{answer=Read.Query_answer(_,Read.Outline_block _,Ok(Worker_pure_data.V2_present_block{revision;_} as value));_})-> - revision,Read.encoded_value_bytes query value - |_->failwith "observation recovery source read failed" in - let _,encoded_bytes=read 2 in - let budget=encoded_bytes+128 in - require(budget<60000) "recovery fixture envelope unexpectedly large"; - let write n title= - let revision,_=read(n+100)in - require(match call(mutation_request _client !baseline owner registration ~request_id:(rid n) ~mutation_id:(rid(n+200)) - (W.Save_block{block=Test_support.authoritative_block_uuid;title; - preconditions={blocks=[Test_support.authoritative_block_uuid,revision];pages=[];scopes=[]}}))with - |Completed(Service.Coordination_mutation_result{outcome=Logseq_db_worker_contract.Worker_pure_mutation.Committed _;_})->true - |_->false) "synthetic observation recovery write failed" in - let too_large = String.make (budget + 1024) 'L' in - write 3 too_large; - let initial = - call - (Service.Coordination_observe - { registration - ; owner - ; observation_id = logical - ; request_id=rid 4 - ; read=observation_read _client !baseline owner registration ~maximum_bytes:budget - (Read.Outline_block{uuid=Test_support.authoritative_block_uuid;revision=None}) - }) + "overflow admitted an obsolete physical cursor"; + write 3264 "After retention reset"; + let after_reset = pull replacement 3265 None in + require + (List.length after_reset.windows = 1) + "retention reset swallowed or mixed the later publication"; + acknowledged replacement 3266 after_reset; + (* The original oversized-publication case, through the actual Service. *) + for n = 3300 to 3363 do + write n (Printf.sprintf "Before explicit reset %d" n) + done; + let old = pull replacement 3364 None in + let parent = Test_support.page_uuid in + let page_revision = + match query replacement 3365 (Read.Page { uuid = parent; revision = None }) with + | Completed + (Service.Coordination_queried + { answer = + Read.Query_answer + ( _ + , Read.Page _ + , Ok (Worker_pure_data.V2_present_page { revision; _ }) ) + ; _ + }) -> revision + | _ -> failwith "resync page missing" + in + let scope_revision = + match + query + replacement + 3366 + (Read.Child_members { parent; after = None; revision = None }) + with + | Completed + (Service.Coordination_queried + { answer = Read.Query_answer (_, Read.Child_members _, Ok page); _ }) -> + page.scope_revision + | _ -> failwith "resync children scope missing" + in + let before = !resyncs in + let roots = + [ Worker_pure_data. + { uuid = id 3400 + ; title = "Resync tree" + ; children = + List.init 32 (fun n -> + { uuid = id (3401 + n); title = "Child"; children = [] }) + } + ] in require - (match initial with + (match + call + (mutation_request + client + !baseline + owner + replacement + ~request_id:(id 3440) + ~mutation_id:(id 3440) + (W.Insert_blocks + { parent + ; roots + ; preconditions = + { blocks = [] + ; pages = [ parent, page_revision ] + ; scopes = + [ Worker_pure_data.V2_children_scope parent, scope_revision ] + } + })) + with | Completed - (Service.Coordination_observed - { answer=Read.Query_answer(_,Read.Outline_block _,Error Read.Response_budget_exceeded) + (Service.Coordination_mutation_result + { outcome = Logseq_db_worker_contract.Worker_pure_mutation.Committed _ ; _ }) -> true | _ -> false) - "oversized initial sample did not report a read failure"; - let refresh n = - call - (Service.Coordination_refresh_observation - { registration - ; owner - ; observation_id = logical - ; request_id = rid n - }) - in - let recover n text = - notices := Rrbvec.empty; - write n text; - wait (fun () -> - Rrbvec.exists - (function - | Service.Observation_invalidated value -> value.observation_id = logical - | _ -> false) - !notices); - require - (match refresh (n + 10) with - | Completed - (Service.Coordination_observed - { answer=Read.Query_answer(_,Read.Outline_block _,Ok(Worker_pure_data.V2_present_block { value; _ })) - ; _ - }) -> value.block.title = text - | _ -> false) - "read failure silently discarded its observation interest" - in - recover 5 "Recovered initial observation"; - write 6 too_large; + "oversized publication insertion failed"; + wait (fun () -> !resyncs > before); + let cleared = pull replacement 3441 None in require - (match refresh 17 with + (cleared.windows = [] && cleared.next = None) + "explicit resync retained obsolete windows"; + acknowledged replacement 3443 cleared; + require + (match + query replacement 3442 (Read.Changes { generation; after = Some old.through }) + with | Completed - (Service.Coordination_observed - { answer=Read.Query_answer(_,Read.Outline_block _,Error Read.Response_budget_exceeded) + (Service.Coordination_queried + { answer = + Read.Query_answer (_, Read.Changes _, Error (Read.Resync_required _)) ; _ }) -> true | _ -> false) - "Refresh did not preserve its read error"; - recover 8 "Recovered refreshed observation"; - ignore (call (Service.Release_coordination registration))) -;; -let test_service_unallocated_observation_cleanup_after_close () = - with_observation_service(fun client ~call ~send:_ ~wait ~replies:_ ~notices:_ ~baseline ~owner ~registration -> - let logical=uuid "9d000000-0000-4000-8000-000000000990" in - ignore(call(Service.Client_command Return_to_graph_picker)); - wait(fun ()->match !baseline with - | Some value->value.baseline.state.sync.residents=[] && value.baseline.state.residents=[] - | _ -> false); - require(match call(Service.Coordination_release_observation {registration;owner;observation_id=logical}) with - | Completed(Service.Coordination_observation_released value)->value.observation_id=logical - | _ -> false) "unallocated observation could not settle after graph close"; - ignore(call(Service.Client_command(Select_graph Test_support.graph_uuid))); - wait(fun ()->match !baseline with - | Some value -> List.exists(fun (_,state)->state.Logseq_db_worker.phase=Graph_open) value.baseline.state.residents - && Option.is_some value.baseline.state.sync.selected - | _ -> false); - let reopened=Option.get(Option.get !baseline).baseline.state.sync.selected in - let observe observation_id request_id=Service.Coordination_observe - {registration;owner=reopened;observation_id;request_id; - read=observation_read client !baseline reopened registration Read.Graph_info} in - require(match call(observe logical (uuid "9d000000-0000-4000-8000-000000000991")) with - | Failed _->true|_->false) "canceled logical ID allocated in a reopened generation"; - require(match call(observe (uuid "9d000000-0000-4000-8000-000000000992") - (uuid "9d000000-0000-4000-8000-000000000993")) with - | Completed(Service.Coordination_observed _)->true|_->false) - "cleanup of one unknown ID incorrectly fenced the reopened graph"; - ignore(call(Service.Release_coordination registration))) -;; - -let test_service_unknown_recovery_is_read_only ?(large=false) () = - let module D=Logseq_overlay_db.Database in - let module Types=Logseq_overlay_db.Types in - let module Wire=Logseq_db_worker_contract.Worker_pure_mutation in - let mutation_id=uuid "9f000000-0000-4000-8000-000000000900" in - let encoded=ref "" and pending=ref None in - let prepare support=Eio_main.run(fun _->Eio.Switch.run(fun sw-> - let inspection=D.inspect_mirror ~application_support_directory:support ~graph_id:Test_support.graph_uuid |> Result.get_ok in - let database=D.open_ ~sw (Test_support.dependencies ~behavior:"pending recovery fixture") inspection ~graph_name:"pending recovery" |> Result.get_ok in - Fun.protect ~finally:(fun()->ignore(D.close database)) (fun()-> - let snapshot=D.current_snapshot database |> Result.get_ok in - let revision=Fun.protect ~finally:(fun()->D.release_snapshot snapshot)(fun()-> - match D.get_outline_blocks snapshot [Test_support.authoritative_block_uuid] |> Result.get_ok with - | [Types.Present_block{revision;_}]->revision|_->failwith "pending fixture block missing") in - let preconditions=Worker_pure_data.{blocks=[Test_support.authoritative_block_uuid,Types.Block_state_revision.to_string revision];pages=[];scopes=[]} in - let command,mutation=if large then ( - let children=List.init 5(fun n->Worker_pure_data.{uuid=uuid(Printf.sprintf "9f000000-0000-4000-8000-%012d" (910+n)); - title=String.make 60000 '"';children=[]}) in - let root=Worker_pure_data.{uuid=uuid "9f000000-0000-4000-8000-000000000909";title="Never automatically replay";children} in - let tree=Types.{uuid=root.uuid;title=root.title;children=List.map(fun (child:Worker_pure_data.block_tree)-> - {uuid=child.uuid;title=child.title;children=[]})children} in - Wire.Insert_blocks{parent=Test_support.authoritative_block_uuid;roots=[root];preconditions}, - Types.Insert_blocks{mutation_id;parent=Test_support.authoritative_block_uuid;tree;asset=None}) - else Wire.Save_block{block=Test_support.authoritative_block_uuid;title="Never automatically replay";preconditions}, - Types.Save_block{mutation_id;block=Test_support.authoritative_block_uuid;title="Never automatically replay"} in - let wire=Wire.create_mutation ~mutation_id ~origin:(Uri.of_string "https://api.logseq.io") ~user:"user-1" ~graph:Test_support.graph_uuid command - |> Result.get_ok |>fun intent->Wire.Mutation_intent intent in - pending:=Some wire;encoded:=Wire.encode wire; - if large then require(String.length !encoded>Worker_pure_data.maximum_response_bytes && String.length !encoded<=Worker_pure_data.maximum_request_bytes) - "large recovery fixture must be a valid intent exceeding the ordinary read cap"; - let expected=D.write_precondition ~blocks:[Test_support.authoritative_block_uuid,revision] ~pages:[] ~scopes:[] |> Result.get_ok in - ignore(D.persist_local_intent database ~namespace:(Wire.namespace wire) ~payload: !encoded ~expected mutation |> Result.get_ok)))) in - with_observation_service ~prepare (fun client ~call ~send:_ ~wait:_ ~replies:_ ~notices:_ ~baseline ~owner ~registration-> - let request_id=uuid "9f000000-0000-4000-8000-000000000901" in - let recover()=match call(Service.Coordination_recover_mutations{registration;owner;request_id; - after=None;maximum_items=8;maximum_bytes=(if large then Wire.maximum_recovery_bytes else 16384)}) with - | Mailbox.Completed(Service.Coordination_mutations_recovered{page;_})->page - | _->failwith "pending recovery failed" in - let page=recover() in - require(Rrbvec.length page.items=1 && Wire.encode(Rrbvec.nth page.items 0).intent= !encoded - && (Rrbvec.nth page.items 0).outcome=Wire.Unknown) "recovery inferred a pending write outcome"; - require(match call(Service.Coordination_acknowledge_mutation{registration;owner;request_id; - presentation=(Option.get !baseline).baseline.presentation;intent=Option.get !pending;original=None}) with - | Mailbox.Failed _->true|_->false) "Unknown intent could be acknowledged away"; - require(Rrbvec.length(recover()).items=1) "Unknown ACK or recovery dropped pending identity"; - require(match call(query_request client !baseline owner registration request_id - (Read.Outline_block{uuid=Test_support.authoritative_block_uuid;revision=None}))with - |Mailbox.Completed(Service.Coordination_queried{answer=Read.Query_answer(_,Read.Outline_block _,Ok(Worker_pure_data.V2_present_block{value;_}));_})-> - value.block.title<>"Never automatically replay" - |_->false) "recovery automatically executed Unknown intent") -;; - -let prepare_reusable_asset support = - let module D=Logseq_overlay_db.Database in - let module O=Logseq_overlay_db.Types in - Eio_main.run(fun _->Eio.Switch.run(fun sw-> - let inspection=D.inspect_mirror ~application_support_directory:support ~graph_id:Test_support.graph_uuid |> Result.get_ok in - let database=D.open_ ~sw (Test_support.dependencies ~behavior:"reusable asset") inspection ~graph_name:"reusable asset" |> Result.get_ok in - Fun.protect ~finally:(fun()->ignore(D.close database))(fun()-> - let version=Logseq_db_types.Asset_descriptor.version ~checksum:(String.make 64 'a') ~file_type:"png" |> Result.get_ok in - let parent=Test_support.page_uuid in - let expected=Test_support.insert_precondition database ~parent ~behavior:"reusable asset" in - ignore(D.commit_local database ~expected(O.Insert_blocks{ - mutation_id=Test_support.mutation_uuid 981;parent; - tree={uuid=Test_support.mutation_uuid 980;title="Reusable attachment";children=[]}; - asset=Some{replace_reference=None;version;size=4L}}) |> Result.get_ok)))) -;; - -let test_service_durable_recovery_and_ack () = - let module Wire=Logseq_db_worker_contract.Worker_pure_mutation in - let changes=ref 0 in - let observe_push=function Service.Graph_push(Worker_pure_data.Changes_available _)->incr changes - |Graph_pushes values->List.iter(function _,Worker_pure_data.Changes_available _->incr changes|_->())values - |_->() in - with_observation_service ~prepare:prepare_reusable_asset ~observe_push(fun client ~call ~send:_ ~wait ~replies:_ ~notices:_ ~baseline ~owner ~registration-> - let id n=uuid(Printf.sprintf "9f000000-0000-4000-8000-%012d" n) in - let owner=ref owner in - let presentation ()=(Option.get !baseline).Service.baseline.presentation in - let read n=match call(query_request client !baseline !owner registration(id n) - (Read.Outline_block{uuid=Test_support.authoritative_block_uuid;revision=None}))with - |Mailbox.Completed(Service.Coordination_queried{answer=Read.Query_answer(_,Read.Outline_block _,Ok(Worker_pure_data.V2_present_block{revision;value}));_})->revision,value.block.title - |_->failwith "durable lifecycle source read failed"in - let recover ?(maximum_bytes=16384) ?(maximum_items=1) n after=call(Service.Coordination_recover_mutations - {registration;owner= !owner;request_id=id n;after;maximum_items;maximum_bytes}) in - let page n after=match recover n after with - | Mailbox.Completed(Service.Coordination_mutations_recovered{request_id;page}) when request_id=id n-> - require(Wire.recovery_page_bytes page<=16384) "recovery exceeded complete byte budget";page - | _->failwith "real durable recovery did not return its bounded page" in - require(Rrbvec.is_empty(page 1 None).items) "new fixture has unexpected pending intents"; - let rev,_=read 2 in - let intent n revision title=Wire.create_mutation ~origin:(Uri.of_string "https://api.logseq.io") - ~user:"user-1" ~graph:(!owner).graph_id ~mutation_id:(id n) - (Wire.Save_block{block=Test_support.authoritative_block_uuid;title; - preconditions={blocks=[Test_support.authoritative_block_uuid,revision];pages=[];scopes=[]}}) - |>Result.get_ok |>fun intent->Wire.Mutation_intent intent in - let first=intent 10 rev "Durable first" in - let mutate n intent=match call(Service.Coordination_mutation - {registration;owner= !owner;request_id=id n;intent}) with - | Mailbox.Completed(Service.Coordination_mutation_result{outcome;_})->outcome - | _->failwith "durable lifecycle mutation did not settle" in - let original=match mutate 11 first with - | Wire.Committed response->Wire.committed_point first response |> Result.get_ok - | _->failwith "first durable fixture write failed" in - let rejected=intent 20 rev "Stale rejected" in - require(match mutate 21 rejected with Wire.Rejected _->true|_->false) "stale durable write was not rejected"; - let current,_=read 3 in - let second=intent 30 current "Durable second" in - require(match mutate 31 second with Wire.Committed _->true|_->false) "second durable write failed"; - let collect n = - let rec loop n after count items= - require(count<8) "recovery continuation did not terminate"; - let current=page n after in - require(Rrbvec.length current.items<=1) "recovery exceeded item budget"; - let items=Rrbvec.append items current.items in - match current.next with None->items|Some next->loop(n+1)(Some next)(count+1)items in - loop n None 0 Rrbvec.empty in - let before=read 4 in - let recovered=collect 40 in - require(Rrbvec.length recovered=3) "bounded recovery skipped committed or rejected intent"; - let first_page=page 44 None in - let byte_budget=Wire.recovery_page_bytes first_page in - (match recover ~maximum_bytes:byte_budget ~maximum_items:3 45 None with - | Mailbox.Completed(Service.Coordination_mutations_recovered{page;_})-> - require(Rrbvec.length page.items=1 && page.next=first_page.next) - "byte-bounded prefix skipped the next complete row or forged completeness" - | _->failwith "byte-bounded recovery prefix failed"); - require(read 5=before) "read-only recovery replayed a mutation"; - require(match recover ~maximum_bytes:1 6 None with Mailbox.Failed _->true|_->false) - "oversized first recovery item was silently skipped"; - let old_presentation=presentation() in - let old_generation=(!owner).graph_generation in - ignore(call(Service.Client_command Return_to_graph_picker)); - ignore(call(Service.Client_command(Select_graph(!owner).graph_id))); - wait(fun()->match !baseline with Some b->List.exists(fun((scope:Service.asset_scope),graph)-> - scope.graph_id=(!owner).graph_id && scope.graph_generation<>old_generation - && graph.Logseq_db_worker.phase=Logseq_db_worker.Graph_open)b.baseline.state.residents|None->false); - owner:=Option.get(Option.get !baseline).baseline.state.sync.selected; - let restored=collect 50 in - require(Rrbvec.length restored=3) "reopening lost unsettled durable intents"; - let item=Rrbvec.fold_left(fun found (item:Wire.recovery_item)-> - if item.intent=first then Some item else found)None restored |> Option.get in - let restored_point=match item.outcome with - | Wire.Committed proof->Wire.committed_point first proof |> Result.get_ok - | _->failwith "recovered commit lost original proof" in - require(restored_point=original) "recovery rewrote original commit generation or revisions"; - let ack n presentation intent original=call(Service.Coordination_acknowledge_mutation - {registration;owner= !owner;presentation;request_id=id n;intent;original}) in - let proof=original.generation,original.before,original.after in - require(match ack 60 old_presentation first(Some proof) with Mailbox.Failed _->true|_->false) - "old presentation acknowledged a committed intent"; - require(match ack 61 (presentation()) first None with Mailbox.Failed _->true|_->false) - "committed intent was acknowledged as a rejection"; - require(match ack 62 (presentation()) first(Some("wrong","wrong","wrong")) with Mailbox.Failed _->true|_->false) - "wrong immutable proof acknowledged intent"; - require(Rrbvec.length(collect 70)=3) "invalid ACK removed recovery ownership"; - require(match call(Service.Coordination_confirm{registration;owner= !owner;presentation=presentation(); - request_id=id 80;intent=first;original=proof; - read=observation_read client !baseline !owner registration ~maximum_items:Worker_pure_data.default_page_size - ~maximum_bytes:(Option.get !baseline).limits.response_budget_bytes - (Read.Block_summary{block=Test_support.authoritative_block_uuid;after=None})}) with - | Mailbox.Completed(Service.Coordination_confirmed{answer=Read.Query_answer(_,Read.Block_summary _,Ok _);_})->true|_->false) - "reopened intent could not confirm its original proof"; - let acknowledged n result=require(match result with - | Mailbox.Completed(Service.Coordination_mutation_acknowledged{request_id})->request_id=id n|_->false) - "durable ACK did not return its exact completion" in - acknowledged 81 (ack 81 (presentation()) first(Some proof)); - acknowledged 82 (ack 82 (presentation()) rejected None); - acknowledged 84 (ack 84 (presentation()) rejected None); - acknowledged 83 (ack 83 (presentation()) first(Some proof)); - require(Rrbvec.length(collect 90)=1) "ACK removed the wrong intent or retained settled rows"; - let revision_before=read 7 in - require(match mutate 100 first with Wire.Committed response-> - Wire.committed_point first response=Ok original|_->false) - "ACK deleted immutable original commit proof"; - require(read 8=revision_before) "same-intent reconciliation after ACK replayed the write"; - let block=Test_support.reference_source_uuid in - let revision=match call(query_request client !baseline !owner registration(id 110) - (Read.Outline_block{uuid=block;revision=None})) with - |Completed(Service.Coordination_queried{answer=Read.Query_answer(_,Read.Outline_block _,Ok(Worker_pure_data.V2_present_block{revision;_}));_})->revision - |_->failwith "asset reference source missing" in - let asset=Test_support.mutation_uuid 980 in - let linked=Wire.create_mutation ~origin:(Uri.of_string "https://api.logseq.io") ~user:"user-1" - ~graph:(!owner).graph_id ~mutation_id:(id 111) - (Wire.Set_asset_reference{block;previous=None;asset; - preconditions={blocks=[block,revision];pages=[];scopes=[]}}) |> Result.get_ok - |> fun value->Wire.Mutation_intent value in - let before= !changes in - require(match mutate 112 linked with Wire.Committed _->true|_->false) - "real Service did not execute asset reference mutation"; - wait(fun()-> !changes>before); - require(match call(query_request client !baseline !owner registration(id 113) - (Read.Assets{roots=[block];recursive=false;after=None})) with - |Completed(Service.Coordination_queried{answer=Read.Query_answer(_,Read.Assets _,Ok page);_})-> - List.exists(fun(value:Logseq_db_types.Asset_descriptor.t)->value.uuid=asset)page.items - |_->false) "real typed Assets read missed committed reference") -;; - -let test_service_changefeed_has_one_owner () = - let pulses=ref 0 and resyncs=ref 0 in - let observe_push=function - |Service.Graph_push(Worker_pure_data.Changes_available _)->incr pulses - |Graph_push(Worker_pure_data.Resync_required _)->incr pulses;incr resyncs - |Graph_pushes values->List.iter(function - |_,Worker_pure_data.Changes_available _->incr pulses - |_,Worker_pure_data.Resync_required _->incr pulses;incr resyncs)values - |_->() in - with_observation_service ~change_max_items:16 ~observe_push - (fun client ~call ~send:_ ~wait ~replies:_ ~notices:_ ~baseline ~owner ~registration -> - let id n=uuid(Printf.sprintf "9d000000-0000-4000-8000-%012d" (6000+n)) in - let active=ref registration in - let query registration n query=call(query_request client !baseline owner registration(id n) ~maximum_items:32 query)in - let generation=match query registration 0 Read.Graph_info with - |Completed(Service.Coordination_queried{answer=Read.Query_answer(_,Read.Graph_info,Ok info);_})->info.generation - |_->failwith "changefeed fixture graph info"in - let write n title= - let revision=match query !active (n+100)(Read.Outline_block{uuid=Test_support.authoritative_block_uuid;revision=None})with - |Completed(Service.Coordination_queried{answer=Read.Query_answer(_,Read.Outline_block _,Ok(Worker_pure_data.V2_present_block{revision;_}));_})->revision - |_->failwith "changefeed write revision"in - let before= !pulses in - require(match call(mutation_request client !baseline owner !active ~request_id:(id n) ~mutation_id:(id n) - (W.Save_block{block=Test_support.authoritative_block_uuid;title; - preconditions={blocks=[Test_support.authoritative_block_uuid,revision];pages=[];scopes=[]}}))with - |Completed(Service.Coordination_mutation_result{outcome=Logseq_db_worker_contract.Worker_pure_mutation.Committed _;_})->true - |_->false) "real changefeed write failed"; - wait(fun()-> !pulses>before)in - let pull registration n after : Read.change_batch = match query registration n(Read.Changes{generation;after})with - |Completed(Service.Coordination_queried{request_id;answer=Read.Query_answer(_,Read.Changes _,Ok batch);_})-> - require(request_id=id n) "cached Changes kept an old RPC correlation";batch - |_->failwith "scoped coordinated changefeed pull failed"in - let ack registration n batch=call(Service.Coordination_acknowledge_changes{registration;owner;request_id=id n;batch})in - let acknowledged registration n batch=require(match ack registration n batch with - |Completed(Service.Coordination_changes_acknowledged{request_id;result=Ok value;_})-> - request_id=id n && Read.change_ack_generation value=batch.W.generation && Read.change_ack_through value=batch.through - |_->false) "owned applied window did not receive its actual ACK"in - let data(batch:W.change_batch)=batch.through,batch.windows,batch.next in - let empty=pull registration 12 None in - require(empty.windows=[] && empty.next=None) "fresh physical changefeed was not empty"; - acknowledged registration 13 empty; - write 20 "Owned change window A"; - let first=pull registration 1 None in - require(first.windows<>[]) "actual write produced no retained change window"; - write 21 "Owned change window B"; - require(data(pull registration 4 None)=data first) "unapplied batch was replaced"; - let foreign=Int64.succ registration in - require(match query foreign 5(Read.Changes{generation;after=None})with - |Completed(Service.Coordination_query_revoked value)-> - value.registration=foreign && value.owner=owner && value.request_id=id 5 - |_->false) - "another binding remained an active second changefeed owner"; - acknowledged registration 6 first; - acknowledged registration 7 first; - let second=pull registration 8(Some first.through)in - require(second.through<>first.through && second.windows<>[]) "ACK lost the second actual write window"; - ignore(call(Service.Release_coordination registration)); - let replacement=Int64.succ registration in - require(match call(Service.Register_coordination replacement)with - |Completed(Service.Coordination_registered _)->true|_->false) "replacement binding did not register"; - active:=replacement; - let adopted=pull replacement 9 None in - require(data adopted=data second) "binding retirement discarded an unapplied physical window"; - require(match ack registration 10 second with Failed _->true|_->false) "old binding consumed the replacement window"; - acknowledged replacement 11 adopted; - write 3000 "held retention window"; - let held=pull replacement 3001 None in - for n=3002 to 3258 do write n(Printf.sprintf "retention change %d" n)done; - require(!resyncs>0) "unacknowledged physical change windows grew without explicit resync"; - require(match ack replacement 3260 held with - |Completed(Service.Coordination_changes_acknowledged{result=Error(Read.Resync_required _);_})->true - |_->false) "retention overflow fabricated ACK of discarded physical history"; - let recovered=pull replacement 3261 None in - require(List.for_all(fun(window:Worker_pure_data.change_window)-> - not(List.exists(fun(old:Worker_pure_data.change_window)->old.id=window.id)held.windows))recovered.windows) - "overflow retained the discarded owned window"; - acknowledged replacement 3262 recovered; - require(match query replacement 3263(Read.Changes{generation;after=Some held.through})with - |Completed(Service.Coordination_queried{answer=Read.Query_answer(_,Read.Changes _,Error(Read.Resync_required _));_})->true - |_->false) "overflow admitted an obsolete physical cursor"; - write 3264 "After retention reset"; - let after_reset=pull replacement 3265 None in - require(List.length after_reset.windows=1) "retention reset swallowed or mixed the later publication"; - acknowledged replacement 3266 after_reset; - (* The original oversized-publication case, through the actual Service. *) - for n=3300 to 3363 do write n(Printf.sprintf "Before explicit reset %d" n)done; - let old=pull replacement 3364 None in - let parent=Test_support.page_uuid in - let page_revision=match query replacement 3365(Read.Page{uuid=parent;revision=None})with - |Completed(Service.Coordination_queried{answer=Read.Query_answer(_,Read.Page _,Ok(Worker_pure_data.V2_present_page{revision;_}));_})->revision - |_->failwith "resync page missing" in - let scope_revision=match query replacement 3366(Read.Child_members{parent;after=None;revision=None})with - |Completed(Service.Coordination_queried{answer=Read.Query_answer(_,Read.Child_members _,Ok page);_})->page.scope_revision - |_->failwith "resync children scope missing" in - let before= !resyncs in - let roots=[Worker_pure_data.{uuid=id 3400;title="Resync tree"; - children=List.init 32(fun n->{uuid=id(3401+n);title="Child";children=[]})}] in - require(match call(mutation_request client !baseline owner replacement ~request_id:(id 3440) ~mutation_id:(id 3440) - (W.Insert_blocks{parent;roots;preconditions={blocks=[];pages=[parent,page_revision]; - scopes=[Worker_pure_data.V2_children_scope parent,scope_revision]}}))with - |Completed(Service.Coordination_mutation_result{outcome=Logseq_db_worker_contract.Worker_pure_mutation.Committed _;_})->true - |_->false) "oversized publication insertion failed"; - wait(fun()-> !resyncs>before); - let cleared=pull replacement 3441 None in - require(cleared.windows=[] && cleared.next=None) "explicit resync retained obsolete windows"; - acknowledged replacement 3443 cleared; - require(match query replacement 3442(Read.Changes{generation;after=Some old.through})with - |Completed(Service.Coordination_queried{answer=Read.Query_answer(_,Read.Changes _,Error(Read.Resync_required _));_})->true - |_->false) "explicit resync admitted a discarded cursor"; - write 3444 "After explicit reset"; - let later=pull replacement 3445 None in - require(List.length later.windows=1) "explicit resync swallowed the next publication"; - acknowledged replacement 3446 later; - ignore(call(Service.Release_coordination replacement))) + "explicit resync admitted a discarded cursor"; + write 3444 "After explicit reset"; + let later = pull replacement 3445 None in + require + (List.length later.windows = 1) + "explicit resync swallowed the next publication"; + acknowledged replacement 3446 later; + ignore (call (Service.Release_coordination replacement))) ;; (* Only the real Mailbox allocates the interleaved physical IDs. Core's public reducer cannot reproduce a runner lifetime-history admission limit. *) let test_mixed_mailbox_requests_do_not_exhaust_history () = - with_client_output_wait(fun client ~wait:wait_output-> - let events=Queue.create() and state=ref(C.initial ~client_id) in - let runner=R.create ~client ~managed_origin:"https://api.logseq.io" ~user_id:"user-1" - ~post:(fun event->Queue.add event events) ~publish:(fun _->()) |> Result.get_ok in - let legacy_terminal=ref None and legacy_count=ref 0 and typed_count=ref 0 in - Mailbox.on_event client(fun event-> - let delivered=R.on_event runner event in - match event,delivered.disposition with - |Mailbox.Response{request_id;_},R.Unowned_response->legacy_terminal:=Some request_id;incr legacy_count - |Mailbox.Response _,R.Consumed->incr typed_count - |_->()); - let apply transition=state:=transition.C.next;List.iter(R.submit runner)transition.instructions in - let pump()= + with_client_output_wait (fun client ~wait:wait_output -> + let events = Queue.create () + and state = ref (C.initial ~client_id) in + let runner = + R.create + ~client + ~managed_origin:"https://api.logseq.io" + ~user_id:"user-1" + ~post:(fun event -> Queue.add event events) + ~publish:(fun _ -> ()) + |> Result.get_ok + in + let legacy_terminal = ref None + and legacy_count = ref 0 + and typed_count = ref 0 in + Mailbox.on_event client (fun event -> + let delivered = R.on_event runner event in + match event, delivered.disposition with + | Mailbox.Response { request_id; _ }, R.Unowned_response -> + legacy_terminal := Some request_id; + incr legacy_count + | Mailbox.Response _, R.Consumed -> incr typed_count + | _ -> ()); + let apply transition = + state := transition.C.next; + List.iter (R.submit runner) transition.instructions + in + let pump () = Mailbox.Private.deliver client ~max_events:128; - let count=Queue.length events in - for _=1 to count do apply(C.step !state(Queue.take events))done in - let wait label predicate= - wait_output ~pump ~timeout:8. ~label predicate in - ignore(Mailbox.send client(Service.Client_command(Restore_local_account{user_id="user-1"}))); - let current_session()=Option.bind(C.projection(C.view !state))(fun projection-> - List.find_map(fun resident->if resident.W.phase=W.Open then Some resident.session else None)(W.residents projection)) in - wait "mixed-request fixture did not bootstrap its actual Service" (fun()->current_session()<>None); - let session=Option.get(current_session()) in - let budget=W.read_budget session ~maximum_items:1 ~maximum_bytes:65536 |> Result.get_ok in - for round=1 to 4100 do - let legacy=match Mailbox.send client Service.Get_graph_state with - |Mailbox.Accepted id->id|_->failwith "isolated legacy fixture admission failed" in - wait "legacy terminal lost its exclusive owner" (fun()-> !legacy_terminal=Some legacy); - let ticket,planned=C.request !state session(W.Query(budget,W.Graph_info)) |> Result.get_ok in + let count = Queue.length events in + for _ = 1 to count do + apply (C.step !state (Queue.take events)) + done + in + let wait label predicate = wait_output ~pump ~timeout:8. ~label predicate in + ignore + (Mailbox.send + client + (Service.Client_command (Restore_local_account { user_id = "user-1" }))); + let current_session () = + Option.bind + (C.projection (C.view !state)) + (fun projection -> + List.find_map + (fun resident -> + if resident.W.phase = W.Open then Some resident.session else None) + (W.residents projection)) + in + wait "mixed-request fixture did not bootstrap its actual Service" (fun () -> + current_session () <> None); + let session = Option.get (current_session ()) in + let budget = + W.read_budget session ~maximum_items:1 ~maximum_bytes:65536 |> Result.get_ok + in + for round = 1 to 4100 do + let legacy = + match Mailbox.send client Service.Get_graph_state with + | Mailbox.Accepted id -> id + | _ -> failwith "isolated legacy fixture admission failed" + in + wait "legacy terminal lost its exclusive owner" (fun () -> + !legacy_terminal = Some legacy); + let ticket, planned = + C.request !state session (W.Query (budget, W.Graph_info)) |> Result.get_ok + in apply planned; - wait "typed terminal did not settle" (fun()->C.reply(C.view !state)ticket<>None); - require(match C.reply(C.view !state)ticket with Some(W.Answered(Ok _))->true|_->false) - (Printf.sprintf "normal mixed requests exhausted typed admission at round %d" round); - apply(C.step !state(C.Forget ticket)) + wait "typed terminal did not settle" (fun () -> + C.reply (C.view !state) ticket <> None); + require + (match C.reply (C.view !state) ticket with + | Some (W.Answered (Ok _)) -> true + | _ -> false) + (Printf.sprintf + "normal mixed requests exhausted typed admission at round %d" + round); + apply (C.step !state (C.Forget ticket)) done; - require(!legacy_count>=4100 && !typed_count>=4100) "mixed requests did not reach both real owners"; - apply(C.step !state C.Shutdown_requested); - wait "mixed-request owner did not unregister after draining" (fun()->C.shutdown_state(C.view !state)=C.Stopped)) + require + (!legacy_count >= 4100 && !typed_count >= 4100) + "mixed requests did not reach both real owners"; + apply (C.step !state C.Shutdown_requested); + wait "mixed-request owner did not unregister after draining" (fun () -> + C.shutdown_state (C.view !state) = C.Stopped)) ;; let test_host_recovers_only_pending_import_sources () = - let module Host=Journal_worker_coordination_host in - with_client ~statuses:true ~prepare:prepare_atomic_append_fixture(fun client -> - let jobs=Queue.create() and active=ref None and confirmations=ref 0 in - Mailbox.on_event client(fun event-> - (match event with Mailbox.Response{outcome=Completed(Service.Coordination_import_input_confirmed _);_}->incr confirmations|_->()); - Option.iter(fun host->ignore(Host.on_event host event)) !active); - let pump()=Mailbox.Private.deliver client ~max_events:64; - let count=Queue.length jobs in for _=1 to count do (Queue.take jobs)()done in - let wait label condition=let deadline=Unix.gettimeofday()+.5. in - while not(condition()) && Unix.gettimeofday() + let jobs = Queue.create () + and active = ref None + and confirmations = ref 0 in + Mailbox.on_event client (fun event -> + (match event with + | Mailbox.Response + { outcome = Completed (Service.Coordination_import_input_confirmed _); _ } -> + incr confirmations + | _ -> ()); + Option.iter (fun host -> ignore (Host.on_event host event)) !active); + let pump () = + Mailbox.Private.deliver client ~max_events:64; + let count = Queue.length jobs in + for _ = 1 to count do + (Queue.take jobs) () + done + in + let wait label condition = + let deadline = Unix.gettimeofday () +. 5. in + while (not (condition ())) && Unix.gettimeofday () < deadline do + pump (); + Unix.sleepf 0.001 + done; + require (condition ()) label + in let create suffix = - let host=Host.create ~client ~client_id:(uuid suffix) ~managed_origin:"https://api.logseq.io" ~user_id:"user-1" - ~enqueue:(fun f->Queue.add f jobs) ~changed:(fun _->()) |> Result.get_ok in - active:=Some host; - if suffix="9f000000-0000-4000-8000-000000004001" then - ignore(Mailbox.send client(Service.Client_command(Restore_local_account{user_id="user-1"}))); - wait ("source host did not bootstrap: "^suffix) (fun()->Option.exists(fun p->W.selected p<>None && List.exists(fun resident->resident.W.phase=W.Open)(W.residents p))(C.projection(Host.view host))); - host in - let old=create "9f000000-0000-4000-8000-000000004001" in - let session=W.presentation_session(Option.get(Option.bind(C.projection(Host.view old))W.selected)) in - let call : type a. a W.request -> a = fun operation-> - let ticket=Host.request old session operation |> Result.get_ok in - let result=ref None in Host.watch old ticket ~f:(fun reply->result:=Some reply); - wait "original source operation did not finish" (fun()-> !result<>None); - match Option.get !result with W.Answered(Ok value)->value|_->failwith "real source operation failed" in - let source_file=Filename.temp_file "journal-background-source-synthetic-" ".png" in - Fun.protect ~finally:(fun()->if Sys.file_exists source_file then Sys.remove source_file)(fun()-> - Out_channel.with_open_bin source_file(fun output->output_string output "file"); - let source:Logseq_db_types.Asset_import.t={operation=uuid "9f000000-0000-4000-8000-000000004010"; - asset=uuid "9f000000-0000-4000-8000-000000004011";target=Test_support.authoritative_block_uuid; - local_mutation=uuid "9f000000-0000-4000-8000-000000004012"; - metadata_mutation=uuid "9f000000-0000-4000-8000-000000004013";replace_reference=None; - source_file;title="Prepared synthetic source";file_type="png"} in - let intent=W.asset_import_intent session source |> Result.get_ok in - let imported=match call(W.Import intent)with W.Imported value->value|_->failwith "source did not prepare" in - Option.iter(fun lease->ignore(call(W.Release_file lease)))(W.import_preview imported); - ignore(Host.shutdown old);wait "original source registration did not release" (fun()->C.shutdown_state(Host.view old)=C.Stopped); - let current=create "9f000000-0000-4000-8000-000000004002" in - require(C.import_source_proof(Host.view current)intent=None) "new host inherited fabricated evidence"; - let pending=Rrbvec.singleton intent in - for _=1 to 3 do Host.recover_import_sources current pending ~failed:(fun _ _->failwith "source proof lookup failed")done; - wait "background source lookup did not adopt real Prepared proof" (fun()->C.import_source_proof(Host.view current)intent<>None); - for _=1 to 8 do Host.recover_import_sources current pending ~failed:(fun _ _->failwith "source proof lookup failed");pump()done; - require(!confirmations=1) "repeated publication duplicated source proof reads"; - let fresh=W.presentation_session(Option.get(Option.bind(C.projection(Host.view current))W.selected))in - let unknown={source with operation=uuid "9f000000-0000-4000-8000-000000004020"; - asset=uuid "9f000000-0000-4000-8000-000000004021"; - local_mutation=uuid "9f000000-0000-4000-8000-000000004022"; - metadata_mutation=uuid "9f000000-0000-4000-8000-000000004023"}in - let unknown=W.asset_import_intent fresh unknown |> Result.get_ok in - let pending=Rrbvec.of_list[intent;unknown]in - Host.recover_import_sources current pending ~failed:(fun _ _->failwith "unknown source lookup failed"); - wait "missing input was not inspected" (fun()-> !confirmations=2); - for _=1 to 8 do Host.recover_import_sources current pending ~failed:(fun _ _->failwith "unknown source lookup failed");pump()done; - require(!confirmations=2 && C.import_source_proof(Host.view current)unknown=None) - "None source proof replayed an Import or entered a query loop"; - let ticket=Host.request current fresh(W.Recover_import_inputs{after=None;maximum_items=8;maximum_bytes=65536}) |> Result.get_ok in - let result=ref None in Host.watch current ticket ~f:(fun reply->result:=Some reply); - wait "original input ledger could not be inspected" (fun()-> !result<>None); - require(match Option.get !result with W.Answered(Ok page)->Rrbvec.length page.inputs=1 - && not(Rrbvec.nth page.inputs 0).acknowledged|_->false) - "background source confirmation replayed Import or ACKed reviewed input"; - require(Sys.file_exists source_file) "read-only source lookup removed picker bytes"; - ignore(Host.shutdown current);wait "source provider registration did not release" (fun()->C.shutdown_state(Host.view current)=C.Stopped))) + let host = + Host.create + ~client + ~client_id:(uuid suffix) + ~managed_origin:"https://api.logseq.io" + ~user_id:"user-1" + ~enqueue:(fun f -> Queue.add f jobs) + ~changed:(fun _ -> ()) + |> Result.get_ok + in + active := Some host; + if suffix = "9f000000-0000-4000-8000-000000004001" + then + ignore + (Mailbox.send + client + (Service.Client_command (Restore_local_account { user_id = "user-1" }))); + wait ("source host did not bootstrap: " ^ suffix) (fun () -> + Option.exists + (fun p -> + W.selected p <> None + && List.exists (fun resident -> resident.W.phase = W.Open) (W.residents p)) + (C.projection (Host.view host))); + host + in + let old = create "9f000000-0000-4000-8000-000000004001" in + let session = + W.presentation_session + (Option.get (Option.bind (C.projection (Host.view old)) W.selected)) + in + let call : type a. a W.request -> a = + fun operation -> + let ticket = Host.request old session operation |> Result.get_ok in + let result = ref None in + Host.watch old ticket ~f:(fun reply -> result := Some reply); + wait "original source operation did not finish" (fun () -> !result <> None); + match Option.get !result with + | W.Answered (Ok value) -> value + | _ -> failwith "real source operation failed" + in + let source_file = Filename.temp_file "journal-background-source-synthetic-" ".png" in + Fun.protect + ~finally:(fun () -> if Sys.file_exists source_file then Sys.remove source_file) + (fun () -> + Out_channel.with_open_bin source_file (fun output -> output_string output "file"); + let source : Logseq_db_types.Asset_import.t = + { operation = uuid "9f000000-0000-4000-8000-000000004010" + ; asset = uuid "9f000000-0000-4000-8000-000000004011" + ; target = Test_support.authoritative_block_uuid + ; local_mutation = uuid "9f000000-0000-4000-8000-000000004012" + ; metadata_mutation = uuid "9f000000-0000-4000-8000-000000004013" + ; replace_reference = None + ; source_file + ; title = "Prepared synthetic source" + ; file_type = "png" + } + in + let intent = W.asset_import_intent session source |> Result.get_ok in + let imported = + match call (W.Import intent) with + | W.Imported value -> value + | _ -> failwith "source did not prepare" + in + Option.iter + (fun lease -> ignore (call (W.Release_file lease))) + (W.import_preview imported); + ignore (Host.shutdown old); + wait "original source registration did not release" (fun () -> + C.shutdown_state (Host.view old) = C.Stopped); + let current = create "9f000000-0000-4000-8000-000000004002" in + require + (C.import_source_proof (Host.view current) intent = None) + "new host inherited fabricated evidence"; + let pending = Rrbvec.singleton intent in + for _ = 1 to 3 do + Host.recover_import_sources current pending ~failed:(fun _ _ -> + failwith "source proof lookup failed") + done; + wait "background source lookup did not adopt real Prepared proof" (fun () -> + C.import_source_proof (Host.view current) intent <> None); + for _ = 1 to 8 do + Host.recover_import_sources current pending ~failed:(fun _ _ -> + failwith "source proof lookup failed"); + pump () + done; + require (!confirmations = 1) "repeated publication duplicated source proof reads"; + let fresh = + W.presentation_session + (Option.get (Option.bind (C.projection (Host.view current)) W.selected)) + in + let unknown = + { source with + operation = uuid "9f000000-0000-4000-8000-000000004020" + ; asset = uuid "9f000000-0000-4000-8000-000000004021" + ; local_mutation = uuid "9f000000-0000-4000-8000-000000004022" + ; metadata_mutation = uuid "9f000000-0000-4000-8000-000000004023" + } + in + let unknown = W.asset_import_intent fresh unknown |> Result.get_ok in + let pending = Rrbvec.of_list [ intent; unknown ] in + Host.recover_import_sources current pending ~failed:(fun _ _ -> + failwith "unknown source lookup failed"); + wait "missing input was not inspected" (fun () -> !confirmations = 2); + for _ = 1 to 8 do + Host.recover_import_sources current pending ~failed:(fun _ _ -> + failwith "unknown source lookup failed"); + pump () + done; + require + (!confirmations = 2 && C.import_source_proof (Host.view current) unknown = None) + "None source proof replayed an Import or entered a query loop"; + let ticket = + Host.request + current + fresh + (W.Recover_import_inputs + { after = None; maximum_items = 8; maximum_bytes = 65536 }) + |> Result.get_ok + in + let result = ref None in + Host.watch current ticket ~f:(fun reply -> result := Some reply); + wait "original input ledger could not be inspected" (fun () -> !result <> None); + require + (match Option.get !result with + | W.Answered (Ok page) -> + Rrbvec.length page.inputs = 1 && not (Rrbvec.nth page.inputs 0).acknowledged + | _ -> false) + "background source confirmation replayed Import or ACKed reviewed input"; + require + (Sys.file_exists source_file) + "read-only source lookup removed picker bytes"; + ignore (Host.shutdown current); + wait "source provider registration did not release" (fun () -> + C.shutdown_state (Host.view current) = C.Stopped))) ;; (* The real Service owns the authority proof. Core-only events cannot establish that response-first Mailbox delivery carries a checked revocation. *) let test_retired_query_has_correlated_response () = - with_client(fun client-> + with_client (fun client -> let send request = - let id=match Mailbox.send client request with Accepted id->id|_->failwith "revocation request admission" in - let result=ref None in - let deadline=Unix.gettimeofday()+.6. in - while !result=None && Unix.gettimeofday() List.iter(function - |Mailbox.Response{request_id;outcome;_}when Id.Worker.Request_id.equal id request_id->result:=Some outcome - |_->());Thread.delay 0.001 + let id = + match Mailbox.send client request with + | Accepted id -> id + | _ -> failwith "revocation request admission" + in + let result = ref None in + let deadline = Unix.gettimeofday () +. 6. in + while !result = None && Unix.gettimeofday () < deadline do + Mailbox.For_testing.drain_events client ~max_events:64 + |> List.iter (function + | Mailbox.Response { request_id; outcome; _ } + when Id.Worker.Request_id.equal id request_id -> result := Some outcome + | _ -> ()); + Thread.delay 0.001 done; - Option.get !result in - ignore(send(Service.Client_command(Restore_local_account{user_id="user-1"}))); - let issuer=Worker_effect_authority.issuer ~managed_origin:"https://api.logseq.io" ~user_id:"user-1" - ~runtime_epoch:(Mailbox.runtime_epoch client) ~worker_generation:(Mailbox.worker_generation client) |> Result.get_ok in - let module I=(val issuer:Worker_effect_authority.ISSUER) in - let current()=match send(I.bootstrap_request() |> Result.get_ok)with - |Mailbox.Completed response->(match I.bootstrap response with - |Ok(W.Ready p)->List.find_map(fun r->if r.W.phase=W.Open then Some r.session else None)(W.residents p) - |_->None) - |_->None in - let session=ref None in - let deadline=Unix.gettimeofday()+.6. in - while !session=None && Unix.gettimeofday() Result.get_ok in - let id=W.request_id(uuid "9c000000-0000-4000-8000-000000009901") in - let request=I.request session ~request_id:id(W.Query(budget,W.Graph_info)) |> Result.get_ok in - ignore(send(Service.Client_command(Reconcile_authenticated_user{user_id=None}))); - let response=match send request with - |Mailbox.Completed response->response - |_->failwith "retired query used an uncorrelated transport failure instead of an owned authority result" in - require(I.reply session ~request_id:id(W.Query(budget,W.Graph_info))response=Ok(W.Answered(Error W.Revoked_session))) + Option.get !result + in + ignore (send (Service.Client_command (Restore_local_account { user_id = "user-1" }))); + let issuer = + Worker_effect_authority.issuer + ~managed_origin:"https://api.logseq.io" + ~user_id:"user-1" + ~runtime_epoch:(Mailbox.runtime_epoch client) + ~worker_generation:(Mailbox.worker_generation client) + |> Result.get_ok + in + let module I = (val issuer : Worker_effect_authority.ISSUER) in + let current () = + match send (I.bootstrap_request () |> Result.get_ok) with + | Mailbox.Completed response -> + (match I.bootstrap response with + | Ok (W.Ready p) -> + List.find_map + (fun r -> if r.W.phase = W.Open then Some r.session else None) + (W.residents p) + | _ -> None) + | _ -> None + in + let session = ref None in + let deadline = Unix.gettimeofday () +. 6. in + while !session = None && Unix.gettimeofday () < deadline do + session := current (); + Thread.delay 0.001 + done; + let session = Option.get !session in + let budget = + W.read_budget session ~maximum_items:1 ~maximum_bytes:65536 |> Result.get_ok + in + let id = W.request_id (uuid "9c000000-0000-4000-8000-000000009901") in + let request = + I.request session ~request_id:id (W.Query (budget, W.Graph_info)) |> Result.get_ok + in + ignore + (send (Service.Client_command (Reconcile_authenticated_user { user_id = None }))); + let response = + match send request with + | Mailbox.Completed response -> response + | _ -> + failwith + "retired query used an uncorrelated transport failure instead of an owned \ + authority result" + in + require + (I.reply session ~request_id:id (W.Query (budget, W.Graph_info)) response + = Ok (W.Answered (Error W.Revoked_session))) "authority result did not retain original query owner"; - let other=W.request_id(uuid "9c000000-0000-4000-8000-000000009902") in - require(match I.reply session ~request_id:other(W.Query(budget,W.Graph_info))response with - |Error(W.Protocol_mismatch _)->true|_->false) "foreign request forged a binding retirement proof"; - I.revoke(); - require(match I.reply session ~request_id:other(W.Query(budget,W.Graph_info))response with - |Error(W.Protocol_mismatch _)->true|_->false) "revoked local issuer bypassed exact proof correlation") + let other = W.request_id (uuid "9c000000-0000-4000-8000-000000009902") in + require + (match + I.reply session ~request_id:other (W.Query (budget, W.Graph_info)) response + with + | Error (W.Protocol_mismatch _) -> true + | _ -> false) + "foreign request forged a binding retirement proof"; + I.revoke (); + require + (match + I.reply session ~request_id:other (W.Query (budget, W.Graph_info)) response + with + | Error (W.Protocol_mismatch _) -> true + | _ -> false) + "revoked local issuer bypassed exact proof correlation") ;; let test_physical_outbox_failure_carries_unknown_reason () = @@ -4691,75 +8595,195 @@ let test_physical_outbox_failure_carries_unknown_reason () = let path = ref "" in let reject enabled = Test_support.set_physical_outbox_rejection !path ~enabled in let prepare support = - path := Filename.concat support ("logseq-db-worker/synced-graphs/" - ^ W.Uuid.to_string Test_support.graph_uuid ^ "/db.sqlite") in - with_observation_service ~prepare (fun client ~call ~send:_ ~wait:_ ~replies:_ ~notices:_ ~baseline ~owner ~registration -> - reject true; - let request_id = uuid "9c000000-0000-4000-8000-000000006001" in - let revision = match call (query_request client !baseline owner registration request_id - (Read.Outline_block {uuid=Test_support.authoritative_block_uuid;revision=None})) with - | Mailbox.Completed (Service.Coordination_queried {answer=Read.Query_answer (_,Read.Outline_block _,Ok (Worker_pure_data.V2_present_block {revision;_}));_}) -> revision - | _ -> failwith "physical failure source read failed" in - let request = mutation_request client !baseline owner registration - ~request_id:(uuid "9c000000-0000-4000-8000-000000006002") - ~mutation_id:(uuid "9c000000-0000-4000-8000-000000006003") - (W.Save_block {block=Test_support.authoritative_block_uuid;title="Private synthetic title"; - preconditions={blocks=[Test_support.authoritative_block_uuid,revision];pages=[];scopes=[]}}) in - (match call request with - | Mailbox.Completed (Service.Coordination_mutation_result {outcome=Wire.Unknown_with_cause detail;_}) -> - require (Diagnostic.source_class detail="sqlite-constraint-trigger-1811") - "physical SQLite trigger lost extended classification across actual worker"; - let encoded=Diagnostic.encode detail in - require (String.length encoded<=1024) "unknown cause exceeded bound"; - require (not (String.contains encoded '\n')) "unknown cause is not a bounded canonical carrier" - | _ -> failwith "physical failure must remain Unknown with its concrete safe cause"); - reject false; - require (match call request with - | Mailbox.Completed (Service.Coordination_mutation_result {outcome=Wire.Committed _;_}) -> true - | _ -> false) "same immutable intent did not recover after physical fault removal") + path + := Filename.concat + support + ("logseq-db-worker/synced-graphs/" + ^ W.Uuid.to_string Test_support.graph_uuid + ^ "/db.sqlite") + in + with_observation_service + ~prepare + (fun + client + ~call + ~send:_ + ~wait:_ + ~replies:_ + ~notices:_ + ~baseline + ~owner + ~registration + -> + reject true; + let request_id = uuid "9c000000-0000-4000-8000-000000006001" in + let revision = + match + call + (query_request + client + !baseline + owner + registration + request_id + (Read.Outline_block + { uuid = Test_support.authoritative_block_uuid; revision = None })) + with + | Mailbox.Completed + (Service.Coordination_queried + { answer = + Read.Query_answer + ( _ + , Read.Outline_block _ + , Ok (Worker_pure_data.V2_present_block { revision; _ }) ) + ; _ + }) -> revision + | _ -> failwith "physical failure source read failed" + in + let request = + mutation_request + client + !baseline + owner + registration + ~request_id:(uuid "9c000000-0000-4000-8000-000000006002") + ~mutation_id:(uuid "9c000000-0000-4000-8000-000000006003") + (W.Save_block + { block = Test_support.authoritative_block_uuid + ; title = "Private synthetic title" + ; preconditions = + { blocks = [ Test_support.authoritative_block_uuid, revision ] + ; pages = [] + ; scopes = [] + } + }) + in + (match call request with + | Mailbox.Completed + (Service.Coordination_mutation_result + { outcome = Wire.Unknown_with_cause detail; _ }) -> + require + (Diagnostic.source_class detail = "sqlite-constraint-trigger-1811") + "physical SQLite trigger lost extended classification across actual worker"; + let encoded = Diagnostic.encode detail in + require (String.length encoded <= 1024) "unknown cause exceeded bound"; + require + (not (String.contains encoded '\n')) + "unknown cause is not a bounded canonical carrier" + | _ -> + failwith "physical failure must remain Unknown with its concrete safe cause"); + reject false; + require + (match call request with + | Mailbox.Completed + (Service.Coordination_mutation_result { outcome = Wire.Committed _; _ }) -> + true + | _ -> false) + "same immutable intent did not recover after physical fault removal") ;; let () = - if Array.mem "--physical-outbox-cause-only" Sys.argv then ( - test_physical_outbox_failure_carries_unknown_reason (); exit 0); + List.iter + (fun terminal -> test_native_registration_network_epochs ~terminal ()) + [ "protocol"; "storage"; "crypto"; "integrity"; "authentication" ]; + if Array.mem "--terminal-recovery-only" Sys.argv then exit 0; + if Array.mem "--network-epochs-only" Sys.argv + then ( + test_native_service_cold_restore_waits_for_identity ~confirmation_first:true (); + test_native_service_cold_restore_waits_for_identity (); + test_native_registration_network_epochs ~confirmation_first:true (); + test_native_registration_network_epochs (); + print_endline "Native registration network epochs passed"; + exit 0); + if Array.mem "--network-scopes-only" Sys.argv + then ( + test_native_service_rejects_stale_network_commands (); + print_endline "Native Service network scope guards passed"; + exit 0); + test_native_service_cold_restore_waits_for_identity ~confirmation_first:true (); + test_native_service_cold_restore_waits_for_identity (); + test_native_registration_network_epochs ~confirmation_first:true (); + test_native_registration_network_epochs (); + test_native_service_rejects_stale_network_commands (); + if Array.mem "--source-recovery-only" Sys.argv + then ( + test_host_recovers_only_pending_import_sources (); + print_endline "Exact pending source evidence recovery passed"; + exit 0); + if Array.mem "--physical-outbox-cause-only" Sys.argv + then ( + test_physical_outbox_failure_carries_unknown_reason (); + exit 0); test_physical_outbox_failure_carries_unknown_reason (); - if Array.mem "--typed-read-only" Sys.argv then ( + if Array.mem "--typed-read-only" Sys.argv + then ( test_client_to_typed_read (); - print_endline "Worker coordination real typed client/read passed (1 test)"; exit 0); - test_cleanup_transport_causes(); + print_endline "Worker coordination real typed client/read passed (1 test)"; + exit 0); + test_cleanup_transport_causes (); if Array.mem "--cleanup-causes-only" Sys.argv then exit 0; - if Array.mem "--auth-session-only" Sys.argv then ( - test_restored_authentication_sessions(); - test_retired_query_has_correlated_response(); - print_endline "Restored same-user sessions and real authority retirement passed";exit 0); - if Array.mem "--read-revocation-only" Sys.argv then (test_retired_query_has_correlated_response();exit 0); - if Array.mem "--application-close-only" Sys.argv then (test_application_current_close ~replacement:false();test_application_current_close ~replacement:true();exit 0); - test_application_current_close ~replacement:false(); - test_application_current_close ~replacement:true(); - if Array.mem "--read-owner-only" Sys.argv then ( + if Array.mem "--auth-session-only" Sys.argv + then ( + test_restored_authentication_sessions (); + test_retired_query_has_correlated_response (); + print_endline "Restored same-user sessions and real authority retirement passed"; + exit 0); + if Array.mem "--read-revocation-only" Sys.argv + then ( + test_retired_query_has_correlated_response (); + exit 0); + if Array.mem "--application-close-only" Sys.argv + then ( + test_application_current_close ~replacement:false (); + test_application_current_close ~replacement:true (); + exit 0); + test_application_current_close ~replacement:false (); + test_application_current_close ~replacement:true (); + if Array.mem "--read-owner-only" Sys.argv + then ( test_read_completion_rechecks_graph_owner (); - print_endline "Real Service read completion owner race passed"; exit 0); - if Array.mem "--change-owner-only" Sys.argv then ( - test_service_changefeed_has_one_owner();print_endline "Real Service changefeed owner cases passed";exit 0); - if Array.mem "--read-boundaries-only" Sys.argv then ( - test_enriched_response_budget();test_encoded_utf8_budget_boundary();test_canonical_target_summary_query(); - print_endline "Migrated real typed read boundary cases passed";exit 0); - if Array.mem "--changes-application-only" Sys.argv then ( - test_changefeed_business_waits_for_ui_application(); print_endline "Real typed Changes application barrier passed"; exit 0); - if Array.mem "--source-recovery-only" Sys.argv then ( - test_host_recovers_only_pending_import_sources();print_endline "Exact pending source evidence recovery passed";exit 0); - test_host_recovers_only_pending_import_sources(); - if Array.mem "--observation-retirement-only" Sys.argv then ( - test_service_binding_retirement_reclaims_observation_history(); - print_endline "8200 actual Service observation binding retirements passed";exit 0); - test_service_binding_retirement_reclaims_observation_history(); - test_mixed_mailbox_requests_do_not_exhaust_history(); - if Array.exists((=)"--ownership-only")Sys.argv then (print_endline "4100 real mixed Mailbox rounds passed";exit 0); - test_revoked_hidden_lease_blocks_unregister_ack(); - if Array.exists((=)"--retirement-only")Sys.argv then (print_endline "Actual revoked hidden lease / unregister / retry passed";exit 0); - let host_only=Array.exists((=)"--host-handoff-only")Sys.argv in - test_real_host_close_keeps_other_resident_and_reopens(); - if host_only then (print_endline "Real host close/observe/consume/reopen and exclusive responses passed";exit 0); + print_endline "Real Service read completion owner race passed"; + exit 0); + if Array.mem "--change-owner-only" Sys.argv + then ( + test_service_changefeed_has_one_owner (); + print_endline "Real Service changefeed owner cases passed"; + exit 0); + if Array.mem "--read-boundaries-only" Sys.argv + then ( + test_enriched_response_budget (); + test_encoded_utf8_budget_boundary (); + test_canonical_target_summary_query (); + print_endline "Migrated real typed read boundary cases passed"; + exit 0); + if Array.mem "--changes-application-only" Sys.argv + then ( + test_changefeed_business_waits_for_ui_application (); + print_endline "Real typed Changes application barrier passed"; + exit 0); + test_host_recovers_only_pending_import_sources (); + if Array.mem "--observation-retirement-only" Sys.argv + then ( + test_service_binding_retirement_reclaims_observation_history (); + print_endline "8200 actual Service observation binding retirements passed"; + exit 0); + test_service_binding_retirement_reclaims_observation_history (); + test_mixed_mailbox_requests_do_not_exhaust_history (); + if Array.exists (( = ) "--ownership-only") Sys.argv + then ( + print_endline "4100 real mixed Mailbox rounds passed"; + exit 0); + test_revoked_hidden_lease_blocks_unregister_ack (); + if Array.exists (( = ) "--retirement-only") Sys.argv + then ( + print_endline "Actual revoked hidden lease / unregister / retry passed"; + exit 0); + let host_only = Array.exists (( = ) "--host-handoff-only") Sys.argv in + test_real_host_close_keeps_other_resident_and_reopens (); + if host_only + then ( + print_endline "Real host close/observe/consume/reopen and exclusive responses passed"; + exit 0); test_service_writer_fence_waits_for_canceled_protected_commit (); test_prepared_import_survives_worker_lifetime (); test_service_import_rejects_retired_registration (); diff --git a/test/wrapped_graph_key_crypto_test.swift b/test/wrapped_graph_key_crypto_test.swift index c448b539..d819cc8b 100644 --- a/test/wrapped_graph_key_crypto_test.swift +++ b/test/wrapped_graph_key_crypto_test.swift @@ -1,6 +1,7 @@ import Darwin import Foundation import Security +import OSLog private enum HarnessError: Error { case failed(String) @@ -49,6 +50,90 @@ private func makeFixture() throws -> (privateKey: Data, wrapped: String) { return (external, wrapped) } +private func rsaFormatTest() throws { + func der(_ tag: UInt8, _ contents: Data) -> Data { + let count = contents.count + let length: [UInt8] = count < 128 ? [UInt8(count)] + : count < 256 ? [0x81, UInt8(count)] + : [0x82, UInt8(count >> 8), UInt8(count & 255)] + return Data([tag] + length) + contents + } + let fixture = try makeFixture() + let version = Data([0x02, 0x01, 0x00]) + let rsaAlgorithm = Data([0x30, 0x0d, 0x06, 0x09, 0x2a, 0x86, 0x48, 0x86, + 0xf7, 0x0d, 0x01, 0x01, 0x01, 0x05, 0x00]) + let pkcs8 = der(0x30, version + rsaAlgorithm + der(0x04, fixture.privateKey)) + func verify(_ key: Data) throws { + _ = try JournalE2EECrypto.handle(JournalCryptoRequest( + operation: "installTestPrivateKey", origin: "https://example.invalid", + userId: "rsa-format-synthetic", privateKey: hex(key))) + _ = try JournalE2EECrypto.handle(JournalCryptoRequest( + operation: "verifyAndSaveWrappedGraphKey", origin: "https://example.invalid", + userId: "rsa-format-synthetic", graphId: "10000000-0000-4000-8000-000000000002", + encryptedGraphKey: fixture.wrapped)) + } + try verify(fixture.privateKey) + let started = Date() + try verify(pkcs8) + Thread.sleep(forTimeInterval: 2) + let store = try OSLogStore(scope: .currentProcessIdentifier) + let entries = try store.getEntries(at: store.position(date: started)) + let errors = entries.compactMap { $0 as? OSLogEntryLog }.filter { + $0.level == .error && ($0.subsystem.contains("security") || $0.composedMessage.contains("SecKey")) + } + print("synthetic-pkcs8 Security E count=\(errors.count)") + try require(errors.isEmpty, "valid PKCS8 emitted a Security error") + try verify(der(0x30, version + rsaAlgorithm + der(0x04, fixture.privateKey) + Data([0xa0, 0]))) + try verify(der(0x30, version + der(0x30, Data(rsaAlgorithm.dropFirst(2).dropLast(2))) + + der(0x04, fixture.privateKey))) + let attributeOID = Data([0x06, 0x02, 0x2a, 0x03]) + let attribute = der(0x30, attributeOID + der(0x31, der(0x0c, Data("synthetic".utf8)))) + func withAttributes(_ attributes: Data) -> Data { + der(0x30, version + rsaAlgorithm + der(0x04, fixture.privateKey) + der(0xa0, attributes)) + } + try verify(withAttributes(attribute)) + var nested = der(0x05, Data()) + for _ in 0..<20 { nested = der(0x30, nested) } + let invalidAttributes = [ + Data([0x30]), + der(0x30, attributeOID), + der(0x30, der(0x05, Data()) + der(0x31, der(0x05, Data()))), + der(0x30, Data([0x06, 0x01, 0x80]) + der(0x31, der(0x05, Data()))), + der(0x30, attributeOID + der(0x31, Data([0x30]))), + der(0x30, attributeOID + der(0x31, Data([0x0c, 0x81, 0x01, 0x61]))), + der(0x30, attributeOID + der(0x31, nested)), + attribute + Data([0]), + ] + for attributes in invalidAttributes { + do { + try verify(withAttributes(attributes)) + throw HarnessError.failed("malformed attributes accepted: \(hex(attributes))") + } catch let error as JournalE2EECryptoError { + try require(journalCryptoFailureCode(error) == "cryptoInvalidPrivateKey", + "malformed attributes lost their typed cause") + } + } + print("synthetic attributes: nonempty accepted; malformed rejected=\(invalidAttributes.count)") + var otherAlgorithm = rsaAlgorithm + otherAlgorithm[12] = 0x02 + let cases: [(Data, String)] = [ + (pkcs8 + Data([0]), "cryptoInvalidPrivateKey"), + (Data(pkcs8.dropLast()), "cryptoInvalidPrivateKey"), + (Data([0x30, 0x88, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff]), "cryptoInvalidPrivateKey"), + (der(0x30, version + otherAlgorithm + der(0x04, fixture.privateKey)), "cryptoUnsupportedPrivateKey"), + (der(0x30, Data([0x02, 0x01, 0x01]) + rsaAlgorithm + der(0x04, fixture.privateKey)), "cryptoUnsupportedPrivateKey"), + (der(0x30, version + rsaAlgorithm + der(0x04, Data([0x30, 0]))), "cryptoInvalidPrivateKey"), + ] + for (key, expected) in cases { + do { + try verify(key) + throw HarnessError.failed("malformed or unsupported RSA input was accepted") + } catch let error as JournalE2EECryptoError { + try require(journalCryptoFailureCode(error) == expected, "RSA format failure lost its typed cause") + } + } +} + private func unlockLogTest() throws { let capture = URL(fileURLWithPath: NSTemporaryDirectory()) .appendingPathComponent("journal-unlock-log-\(UUID().uuidString)") @@ -143,6 +228,7 @@ private enum WrappedGraphKeyCryptoTest { JournalE2EECrypto.resetTestMemoryPrivateKeys() } + try rsaFormatTest() try unlockLogTest() let malformed = "{\"operation\":\"deleteAccountSecrets\",\"origin\":\"\(origin)\",\"userId\":\"\(userID)\",\"iterations\":\"600000\"}" diff --git a/tool/build_simulator_network_fault.sh b/tool/build_simulator_network_fault.sh new file mode 100644 index 00000000..c9f132b8 --- /dev/null +++ b/tool/build_simulator_network_fault.sh @@ -0,0 +1,35 @@ +#!/bin/sh +set -eu +root=$(CDPATH='' cd -- "$(dirname -- "$0")/.." && pwd) +out=$(mktemp -d "${TMPDIR:-/tmp}/journal-network-fault.XXXXXXXX") +trap 'rm -f "$out/simulator_network_fault.dylib"; rmdir "$out"' EXIT +chmod 700 "$out" +sdk=$(xcrun --sdk iphonesimulator --show-sdk-path) +xcrun --sdk iphonesimulator clang \ + -target arm64-apple-ios26.0-simulator -isysroot "$sdk" \ + -std=c11 -Wall -Wextra -Werror -O2 -dynamiclib \ + -Wl,-no_uuid -Wl,-install_name,@rpath/simulator_network_fault.dylib \ + "$root/tool/simulator_network_fault.c" -o "$out/simulator_network_fault.dylib" +codesign --force --sign - --timestamp=none "$out/simulator_network_fault.dylib" +codesign --verify --strict "$out/simulator_network_fault.dylib" +file "$out/simulator_network_fault.dylib" +xcrun vtool -show-build "$out/simulator_network_fault.dylib" +xcrun otool -l "$out/simulator_network_fault.dylib" +xcrun nm -m "$out/simulator_network_fault.dylib" +shasum -a 256 "$root/tool/simulator_network_fault.c" "$out/simulator_network_fault.dylib" +if [ -n "${JOURNAL_TEST_DYLIB_DEST:-}" ]; then + python3 - "$out/simulator_network_fault.dylib" "$JOURNAL_TEST_DYLIB_DEST" <<'PY' +import os, shutil, stat, sys +source, destination = sys.argv[1:] +parent = os.path.dirname(destination) +info = os.stat(parent) +if (not os.path.isabs(destination) or os.path.realpath(parent) != parent + or not stat.S_ISDIR(info.st_mode) or info.st_uid != os.getuid() + or stat.S_IMODE(info.st_mode) != 0o700): + raise SystemExit("destination requires canonical private owned directory") +with open(source, "rb") as incoming: + descriptor = os.open(destination, os.O_WRONLY | os.O_CREAT | os.O_EXCL | os.O_NOFOLLOW, 0o600) + with os.fdopen(descriptor, "wb") as outgoing: + shutil.copyfileobj(incoming, outgoing) +PY +fi diff --git a/tool/simulator_network_fault.c b/tool/simulator_network_fault.c new file mode 100644 index 00000000..85766e44 --- /dev/null +++ b/tool/simulator_network_fault.c @@ -0,0 +1,314 @@ +/* Simulator-only runtime fault injection; never link into the app. */ +#include +#if !TARGET_OS_SIMULATOR +#error Simulator only +#endif +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#define CAPACITY 256 +/* Original-symbol calls in this image bypass its __interpose replacements. */ +#define INTERPOSE(replacement, original) \ + __attribute__((used, section("__DATA,__interpose"))) \ + static const struct { const void *new_function; const void *old_function; } \ + pair_##original = { (const void *)&replacement, (const void *)&original } + +static pthread_mutex_t lock = PTHREAD_MUTEX_INITIALIZER; +static struct { + int fd; + bool occupied; + unsigned in_flight; + uint64_t identity, stopped_generation; +} ledger[CAPACITY]; +static uint64_t next_identity, fault_generation, heartbeat; +static bool status_failed; + +static struct { + uint64_t shutdowns, rejected_connects, infrastructure_errors, shutdown_errors; + int error_code; + bool offline; +} counters; +static char flag_path[PATH_MAX]; +static int status_fd = -1; +static atomic_bool enabled, stopping; +static pthread_t monitor_thread; +static bool thread_started; + +static void infrastructure_error(int code) { + counters.infrastructure_errors++; + counters.error_code = code; +} + +static bool fault_present(void) { + struct stat info; + if (lstat(flag_path, &info) == 0) return true; + if (errno == ENOENT) return false; + int saved = errno; + if (counters.error_code != saved) infrastructure_error(saved); + return true; +} + +static int injected_socket(int domain, int type, int protocol) { + if (!atomic_load(&enabled) || (domain != AF_INET && domain != AF_INET6)) + return socket(domain, type, protocol); + pthread_mutex_lock(&lock); + unsigned slot; + for (slot = 0; slot < CAPACITY && ledger[slot].occupied; slot++) {} + if (slot == CAPACITY) { + infrastructure_error(ENOBUFS); + pthread_mutex_unlock(&lock); + errno = ENOBUFS; + return -1; + } + int descriptor = socket(domain, type, protocol); + int saved = errno; + if (descriptor >= 0) { + ledger[slot].fd = descriptor; + ledger[slot].occupied = true; + ledger[slot].in_flight = 0; + ledger[slot].identity = ++next_identity; + ledger[slot].stopped_generation = 0; + } + pthread_mutex_unlock(&lock); + errno = saved; + return descriptor; +} + +static int injected_close(int descriptor) { + if (!atomic_load(&enabled)) return close(descriptor); + pthread_mutex_lock(&lock); + int result = close(descriptor); + int saved = errno; + /* Darwin releases the descriptor even when close reports EINTR. */ + for (unsigned slot = 0; slot < CAPACITY; slot++) + if (ledger[slot].occupied && ledger[slot].fd == descriptor) + ledger[slot].occupied = false; + pthread_mutex_unlock(&lock); + errno = saved; + return result; +} + +static void observe_fault(bool offline) { + if (offline && !counters.offline) fault_generation++; + counters.offline = offline; +} + +static void reconcile_socket(unsigned slot) { + if (!counters.offline || !ledger[slot].occupied || + ledger[slot].stopped_generation == fault_generation) return; + struct sockaddr_storage address; + socklen_t length = sizeof(address); + if (getpeername(ledger[slot].fd, (struct sockaddr *)&address, &length) != 0) { + if (errno != ENOTCONN && errno != EINVAL) infrastructure_error(errno); + return; + } + if (shutdown(ledger[slot].fd, SHUT_RDWR) == 0) { + counters.shutdowns++; + if (ledger[slot].in_flight == 0) + ledger[slot].stopped_generation = fault_generation; + } else { + counters.shutdown_errors++; + infrastructure_error(errno); + } +} + +static bool begin_connect(int descriptor, uint64_t *identity) { + int saved = errno; + *identity = 0; + if (!atomic_load(&enabled)) return false; + bool network = false; + pthread_mutex_lock(&lock); + bool offline = fault_present(); + observe_fault(offline); + for (unsigned slot = 0; slot < CAPACITY; slot++) { + if (!ledger[slot].occupied || ledger[slot].fd != descriptor) continue; + network = true; + if (!offline) { + ledger[slot].in_flight++; + ledger[slot].stopped_generation = 0; + *identity = ledger[slot].identity; + } + break; + } + if (!network) { + struct sockaddr_storage address; + socklen_t length = sizeof(address); + if (getsockname(descriptor, (struct sockaddr *)&address, &length) == 0 && + (address.ss_family == AF_INET || address.ss_family == AF_INET6)) { + network = true; + infrastructure_error(ENOTSUP); + } + } + if (network && offline) counters.rejected_connects++; + pthread_mutex_unlock(&lock); + errno = network && offline ? ENETDOWN : saved; + return network && offline; +} + +static void finish_connect(int descriptor, uint64_t identity) { + if (!identity) return; + int saved = errno; + pthread_mutex_lock(&lock); + bool offline = fault_present(); + observe_fault(offline); + for (unsigned slot = 0; slot < CAPACITY; slot++) { + if (!ledger[slot].occupied || ledger[slot].fd != descriptor || + ledger[slot].identity != identity) continue; + ledger[slot].in_flight--; + ledger[slot].stopped_generation = 0; + reconcile_socket(slot); + break; + } + pthread_mutex_unlock(&lock); + errno = saved; +} + +static int injected_connect(int descriptor, const struct sockaddr *address, socklen_t length) { + uint64_t identity; + if (begin_connect(descriptor, &identity)) return -1; + int result = connect(descriptor, address, length); + finish_connect(descriptor, identity); + return result; +} + +static int injected_connectx(int descriptor, const sa_endpoints_t *endpoints, + sae_associd_t association, unsigned int flags, + const struct iovec *data, unsigned int count, + size_t *written, sae_connid_t *connection) { + uint64_t identity; + if (begin_connect(descriptor, &identity)) return -1; + int result = connectx(descriptor, endpoints, association, flags, data, count, written, connection); + finish_connect(descriptor, identity); + return result; +} + +static void *monitor(void *unused) { + (void)unused; + while (!atomic_load(&stopping)) { + char status[768]; + pthread_mutex_lock(&lock); + bool offline = fault_present(); + observe_fault(offline); + for (unsigned slot = 0; slot < CAPACITY; slot++) reconcile_socket(slot); + heartbeat++; + int length = snprintf(status, sizeof(status), + "{\"schema\":2,\"capacity\":256,\"offline\":%s,\"shutdowns\":%llu," + "\"heartbeat\":%llu,\"fault_generation\":%llu,\"terminal_error\":%s," + "\"rejected_connects\":%llu,\"infrastructure_errors\":%llu," + "\"shutdown_errors\":%llu,\"error_code\":%d}\n", + counters.offline ? "true" : "false", + (unsigned long long)counters.shutdowns, + (unsigned long long)heartbeat, (unsigned long long)fault_generation, + counters.infrastructure_errors ? "true" : "false", + (unsigned long long)counters.rejected_connects, + (unsigned long long)counters.infrastructure_errors, + (unsigned long long)counters.shutdown_errors, counters.error_code); + pthread_mutex_unlock(&lock); + ssize_t written = pwrite(status_fd, status, (size_t)length, 0); + int failure = written != length ? (written < 0 ? errno : EIO) : 0; + if (!failure && ftruncate(status_fd, length) != 0) failure = errno; + if (failure) { + pthread_mutex_lock(&lock); + infrastructure_error(failure); + pthread_mutex_unlock(&lock); + if (!status_failed) { + const char message[] = "simulator_network_fault: terminal status write failure; reject acceptance; monitoring continues\n"; + (void)write(STDERR_FILENO, message, sizeof(message) - 1); + status_failed = true; + } + } + struct timespec delay = { .tv_sec = 0, .tv_nsec = 100000000 }; + nanosleep(&delay, NULL); + } + return NULL; +} + +static bool private_paths(const char *flag, const char *status) { + char parent[PATH_MAX], other[PATH_MAX], canonical[PATH_MAX]; + if (strlen(flag) >= sizeof(parent) || strlen(status) >= sizeof(other)) return false; + strcpy(parent, flag); + strcpy(other, status); + char *flag_name = strrchr(parent, '/'); + char *status_name = strrchr(other, '/'); + if (!flag_name || !status_name || !flag_name[1] || !status_name[1] || + strcmp(flag_name + 1, ".") == 0 || strcmp(flag_name + 1, "..") == 0 || + strcmp(status_name + 1, ".") == 0 || strcmp(status_name + 1, "..") == 0) return false; + *flag_name = '\0'; + *status_name = '\0'; + struct stat info; + return strcmp(parent, other) == 0 && realpath(parent, canonical) != NULL && + strcmp(parent, canonical) == 0 && stat(parent, &info) == 0 && + S_ISDIR(info.st_mode) && info.st_uid == getuid() && (info.st_mode & 077) == 0; +} + +static void before_fork(void) { + pthread_mutex_lock(&lock); + infrastructure_error(ENOTSUP); +} + +static void after_fork_parent(void) { pthread_mutex_unlock(&lock); } + +static void after_fork_child(void) { + atomic_store(&enabled, false); + thread_started = false; + if (status_fd >= 0) close(status_fd); + status_fd = -1; + pthread_mutex_unlock(&lock); + const char message[] = "simulator_network_fault: fork unsupported; child injection disabled\n"; + (void)write(STDERR_FILENO, message, sizeof(message) - 1); +} + +__attribute__((constructor)) static void start(void) { + const char *flag = getenv("JOURNAL_TEST_NETWORK_FAULT_FLAG"); + const char *status = getenv("JOURNAL_TEST_NETWORK_FAULT_STATUS"); + if (!flag && !status) return; + if (!flag || !status || flag[0] != '/' || status[0] != '/' || + strcmp(flag, status) == 0 || !private_paths(flag, status)) goto failed; + memcpy(flag_path, flag, strlen(flag) + 1); + /* Require a fresh status file; never overwrite another task's data. */ + status_fd = open(status, O_WRONLY | O_CREAT | O_EXCL | O_NOFOLLOW | O_CLOEXEC, 0600); + if (status_fd < 0) goto failed; + if (pthread_atfork(before_fork, after_fork_parent, after_fork_child) != 0) { + close(status_fd); + status_fd = -1; + goto failed; + } + atomic_store(&enabled, true); + int result = pthread_create(&monitor_thread, NULL, monitor, NULL); + if (result != 0) { + atomic_store(&enabled, false); + close(status_fd); + status_fd = -1; + goto failed; + } + thread_started = true; + return; +failed:; + const char message[] = "simulator_network_fault: initialization failed; injection not ready\n"; + (void)write(STDERR_FILENO, message, sizeof(message) - 1); +} + +__attribute__((destructor)) static void stop(void) { + if (!thread_started) return; + atomic_store(&stopping, true); + pthread_join(monitor_thread, NULL); + close(status_fd); +} + +INTERPOSE(injected_socket, socket); +INTERPOSE(injected_close, close); +INTERPOSE(injected_connect, connect); +INTERPOSE(injected_connectx, connectx); diff --git a/tool/simulator_network_fault.md b/tool/simulator_network_fault.md new file mode 100644 index 00000000..fc6af925 --- /dev/null +++ b/tool/simulator_network_fault.md @@ -0,0 +1,178 @@ +# Simulator runtime network fault injection + +This is acceptance-only runtime fault injection, not an ordinary production run. +No app source, binary, bundle, native inputs, TLS, credentials, or signatures are +changed. Only the main operator may launch the owned App A with this library. +Do not launch, terminate, attach, or manipulate the primary Simulator while its +owner has a pending mutation. No global network settings or dependencies change. + +## Build and inspect (no Simulator interaction) + +From the repository root: + +```sh +sh tool/build_simulator_network_fault.sh + +``` + +The helper builds arm64 iOS 26 Simulator code and ad-hoc signs only the separate +dylib. `__DATA,__interpose` is the standard dyld replacement/original pair pattern: +original-symbol calls from the interposer image remain original calls. It avoids +`dlsym`, allocation, lazy lookup, and recursive file-close resolution inside hooks. +SDK `sys/socket.h` supplies the exact `connectx` signature. This requires dyld +interposition to be honored; compilation/signature checks do not prove loading. + +The helper uses a new private temporary directory, inspects and verifies the dylib, +and deletes its own build products on exit. Optional JOURNAL_TEST_DYLIB_DEST copies +the signed dylib exclusively (O_EXCL | O_NOFOLLOW) into a canonical owned mode-700 +parent; it never overwrites a file. Runtime loading and acceptance remain pending. + +## Build-only revision evidence + +At HEAD `4a85af1ef08a2153a806ecea605f5695e38e10ba`, the revised source compiled +successfully for `arm64-apple-ios26.0-simulator` with `-Wall -Wextra -Werror`. +Shell syntax, ad-hoc signing and strict signature verification passed; otool/nm +inspection completed. The helper removed its temporary dylib and directory. + +- C SHA-256: `df8c712bc84e54d79bb7d678fb14127155eb74b44bba2e4855ac5caf7c51da72`. +- Helper SHA-256: `4a0be9b08bfd94ca12bdc712b52da21a16940527f21acc45a5f7875599760f4f`. +- Built dylib SHA-256: `2fbc330c3a9dd16937a9c31e7164b601512841a04782b849923612ce4da479f1`. + +No runtime smoke, Simulator operation or fault injection was performed. Main must +execute the heartbeat and actual-transport acceptance recipe below after re-review. + +## Main-owner-only launch template (NOT executed by the implementation task) + +After independent review and when the main owner can safely restart App A, obtain +its existing data container with `simctl get_app_container` using its full owned +Simulator UDID, not `booted`. Do not use App B's container. Choose a new private, +canonical task directory inside App A's data container, outside its `.app` bundle. +Set `A_CONTAINER` and `A_UDID` explicitly after checking ownership; do not infer +the full UDID from the abbreviated `9733B6DF...` identifier. + +```sh +set -eu +: "${A_CONTAINER:?set owned App A data container}" "${A_UDID:?set full owned UDID}" +TASK_DIR=$(python3 - "$A_CONTAINER" <<'PY' +import os, sys, tempfile +container = os.path.realpath(sys.argv[1]) +parent = os.path.join(container, "tmp") +if os.path.realpath(parent) != parent or os.stat(parent).st_uid != os.getuid(): + raise SystemExit("expected owned canonical container tmp") +path = tempfile.mkdtemp(prefix="network-fault-", dir=parent) +os.chmod(path, 0o700) +print(path) +PY +) +JOURNAL_TEST_DYLIB_DEST="$TASK_DIR/simulator_network_fault.dylib" \ +sh tool/build_simulator_network_fault.sh +# Status must not already exist. Flag absent starts online. +SIMCTL_CHILD_DYLD_INSERT_LIBRARIES="$TASK_DIR/simulator_network_fault.dylib" \ +SIMCTL_CHILD_JOURNAL_TEST_NETWORK_FAULT_FLAG="$TASK_DIR/offline" \ +SIMCTL_CHILD_JOURNAL_TEST_NETWORK_FAULT_STATUS="$TASK_DIR/status.json" \ +xcrun simctl launch "$A_UDID" com.logseq.journal +``` + +These are command-local launch variables, not `launchctl setenv` or production +settings. The process needs both explicit absolute paths. A fresh status file is +created exclusively, without following a final symlink, mode 0600. Keep all path +components in the operator-owned private directory; no shared or symlink paths. +The injector enforces a canonical shared parent directory owned by the process user +with no group/other permissions. No library copy into the bundle, re-signing of the +App, protection bypass, or attach. + +Main must capture process stderr through its launch/supervisor stream. Before EACH +case, require two valid schema-2/capacity-256 snapshots from this fresh process, +with heartbeat strictly increasing within 2 seconds, expected offline state, +terminal_error:false, infrastructure_errors:0 and shutdown_errors:0. Continue +checking at least every 500 ms and through the final case evidence. Missing, +unparseable or non-advancing heartbeat for 2 seconds, any counter error, process +exit, or terminal stderr diagnostic FAILS the case; a stale file is never readiness. +Retry incomplete concurrent JSON reads only within that deadline. Check heartbeat +again after flag transitions and after each case. A flag transition must also show +the expected offline state and, on entry, increased fault_generation. Heartbeat is +a monotonic per-process counter, not file mtime or wall clock. Do not reuse a status +file across launches. A pre-case check alone cannot validate the whole case. + +Create the ephemeral signal exclusively; never overwrite status or another file: + + +```sh +python3 - "$TASK_DIR/offline" <<'PY' +import os, sys +os.close(os.open(sys.argv[1], os.O_WRONLY | os.O_CREAT | os.O_EXCL | os.O_NOFOLLOW, 0o600)) +PY +# Wait for status.json offline:true, then exercise the actual owned App A. +# Remove the flag only when the acceptance case calls for recovery: +rm "$TASK_DIR/offline" +``` + +The monitor polls every 100 ms. It calls `shutdown(SHUT_RDWR)`, never `close`, on +registered connected IPv4/IPv6 sockets per fault generation; application ownership stays +intact. New IPv4/IPv6 `connect` and `connectx` attempts return -1/ENETDOWN while the +flag exists. AF_UNIX remains untouched. Removing the flag restores forwarding; +shutdown sockets do not resurrect, so the app must reconnect normally. The ledger +is process-local, mutex-protected, limited to 256 live sockets, cleared on close. +Overflow refuses socket creation with ENOBUFS and increments infrastructure errors +rather than silently permitting untracked traffic. Status contains only bounded +numeric counters and booleans, never addresses, tokens, payloads, or graph data. +Only the monitor writes status, every poll with an advancing heartbeat; read snapshots +with retry if concurrent writing yields incomplete JSON. Successful shutdowns and +rejected connects are distinct evidence; shutdown errors are never hidden. + +The monitor has one thread, one status fd and a bounded ledger; unload joins after +its polling sleep and closes only its own status fd. There is no service/cache. +A blocked filesystem can delay join; use only the local Simulator data filesystem. +Fork handlers prevent an inherited locked ledger: a fork records ENOTSUP in the +parent, disables injection in the child, and emits a constant child diagnostic. +Fork is unsupported for acceptance, not silently represented as injected coverage. +Initialization failures report a constant stderr message and no valid readiness +status. Status-write failure latches a terminal acceptance error, emits one constant +stderr diagnostic and retries status at the bounded polling rate WITHOUT exiting +the physical monitor or disabling injection. If writes recover, the sticky error +counter remains visible. If the fd stays broken, no file-based error delivery is +promised: main must reject missing heartbeat advancement and supervise stderr. +The App is NOT stopped, signaled or claimed process-fail-closed. Any infrastructure +error invalidates acceptance even when physical shutdown continues. + +Real connect/connectx runs outside the mutex. A bounded per-socket in-flight count +and descriptor identity protect completion reconciliation from slot/fd reuse. +After the original returns, current fault state is sampled under the ledger lock +and connected sockets are actually shut down. The monitor continues checking +pending/nonblocking connections; unconnected sockets are not completed coverage. +Only successful shutdown with no in-flight calls marks a generation covered; +failed shutdowns remain retryable. A later connection completion clears coverage. +Flag creation itself is not instantaneous network isolation; require actual main +transport smoke evidence after the monitor observes it. + +## Required acceptance evidence and limits + +The main owner must prove actual dyld loading first, then actual server connection +failure, persistent offline retry exhaustion and conflict behavior in App A, while +App B stays online and its server checkpoint advances. Retain the pending mutation +and its ID; after flag removal prove recovery of that same ID, without duplication. +Bind results to app/native provenance, HEAD, tool hashes and this injected launch. +Counters alone are not server-failure or sync-recovery proof. + +Coverage is specifically sockets created via interposed `socket`, closed via +interposed `close`, and connections via interposed `connect`/`connectx`. Direct +syscalls, private/nocancel variants, duplicated/received/accepted descriptors, +connections predating library initialization, and out-of-process system services +are not fully covered. An intercepted offline connect on an unregistered network +fd is still rejected, but records ENOTSUP because shutdown coverage is incomplete. Unconnected UDP `sendto`/`sendmsg` is not blocked by these hooks; +shutdown can fail on such sockets. DNS delegated to a resolver service is not +process-local traffic. Do not claim whole-App offline, QUIC/DNS isolation, or use +this for acceptance if the actual app transport takes an uncovered path. A stricter +transport-specific extension needs review and actual-app proof, not a test-for-test. +There is no Simulator launch/attach/UI/kill or actual fault verification in the +implementation/build task. Dyld eligibility and all runtime acceptance remain pending. + +## Cleanup (main owner, after preserving evidence and the mutation) + +Remove the flag, recover and verify sync first. At an owner-approved restart, +launch App A without any of the three `SIMCTL_CHILD_*` variables. If the operator +exported them in a shell, unset them before that launch. After the injected process +has exited and evidence is saved, delete only this case's status/flag/dylib and new +task directory. Do not delete graph/container data. Remove the workspace build +artifact when no longer needed; keep these reusable tool sources. No commit or +production rollout before independent review. diff --git a/tool/sync_acceptance_fixture/README.md b/tool/sync_acceptance_fixture/README.md new file mode 100644 index 00000000..85062666 --- /dev/null +++ b/tool/sync_acceptance_fixture/README.md @@ -0,0 +1,83 @@ +# Task-owned plaintext sync fixtures + +This standalone helper is restricted to fresh fixture data. It does not link App, +change production contracts, control simulators, call RSA/key endpoints, or use +CLI sync upload. The API origin and Cognito protocol come from the existing +`test_managed_sync_e2e.ml` and `managed_sync_e2e_support.ml` harness. + +## Verification evidence + +Each run records creation acknowledgements, assigned-UUID preflights, upload +receipts, catalog readiness, snapshot cursors and downloaded-frame re-admission +in the private 0600 registry. Graph identities and measured quantities are +run-specific evidence, not hard-coded acceptance targets. Keep the registry and +generated frames outside Git. + +A builtin-only graph can have a computed zero checksum. The helper still derives +and checks it through the real public-library activation and fresh remote read; +it never substitutes a constant. Native fresh-read evidence is not Simulator UI +evidence. + +## Safety boundary + +- Local CLI subprocesses receive only a system/opam environment allowlist. CLI + routing variables, secrets, and proxy variables are absent. Every graph command + has explicit `--root-dir`, `--config`, and unique `--graph` arguments. +- Each CLI workspace is private 0700 under `_build/plain-fixture-*`; `cli.edn` + points to an absolute absent auth file. Creation does not enable sync or login. + Export is SQLite. Only the exact scoped CLI worker is stopped; auth remains absent. +- The native helper copies only the newly exported fixture into an exclusive fresh + destination. It restores through the real SQLite adapter, changes the three + graph identity/remote/encryption KVs using pure `Datascript.db_with`, and persists + through existing storage callbacks. No user SQLite data is patched with SQL. +- Both local preflights finish before any remote POST. Each server acknowledgement + is persisted and fsynced immediately. The assigned UUID is then embedded and + the full preflight repeated before upload. +- HTTP credentials travel only in private stdin pipes, never argv/files/logs. + Curl uses real CA validation, ignores curlrc/proxies, and has no automatic retry. +- A successful provisioning run creates exactly two new graphs and uploads a + snapshot only to each acknowledged UUID. Other graph access is limited to + catalog metadata; fixture downloads never read an existing graph. +- Cleanup requires the exact creation-acknowledged UUID, exact nonce/name/role, + the same authenticated creator subject hash, and a fresh exact catalog match; + manager membership alone never authorizes deletion. Run deletion only after + the main owner has completed UI acceptance and fixture cleanup. +- Unknown creation outcomes are not blindly retried or adopted. Exact nonce-name + catalog matches are recorded as indeterminate for manual resolution. On later + failure, only acknowledged fixtures still catalog-unready may be auto-deleted; + server-ready fixtures remain for Main. + +## Commands + +Run from the repository root with the prepared opam environment: + +```sh +dune build tool/sync_acceptance_fixture/fixture.exe +python3 tool/sync_acceptance_fixture/provision.py --create-two-new-plaintext-graphs +``` + +Creation is one-shot: the nonce is durably recorded before setup or remote writes. +Any recorded nonce or indeterminate creation refuses further provisioning, even +when no creation acknowledgement reached the client. Reconcile that original +nonce manually; do not reset the registry or create a new nonce to retry an +unknown result. It requires +`LOGSEQ_ACCOUNT` and `LOGSEQ_ACCOUNT_PASSWORD` only for the private Cognito request, +not for local CLI operations. Without an explicit orchestrator action flag, the +Python tool performs no network operation; the native executable is local-only. + +`--verify-recorded-owned-fixtures` downloads and validates only recorded fixtures. +It deliberately refuses overwriting prior `remote.transit` evidence. Keep each +run's existing results; do not rerun creation to repeat verification. + +After Main completes UI case cleanup, the guarded deletion action is: + +```sh +python3 tool/sync_acceptance_fixture/provision.py --delete-recorded-owned-fixtures +``` + +That action reauthenticates, checks the registry nonce and each creation ACK against +fresh catalog metadata, and verifies exact deletion receipts. It is provided for +Main's later cleanup, not executed during fixture setup. Retain the registry until +both remote deletions are confirmed. Main owns all simulator interaction, plain +sync and capacity/replace UI acceptance, final artifact cleanup, and independent +review before any commit. Native fresh-read evidence is not simulator UI evidence. diff --git a/tool/sync_acceptance_fixture/dune b/tool/sync_acceptance_fixture/dune new file mode 100644 index 00000000..d7a4e074 --- /dev/null +++ b/tool/sync_acceptance_fixture/dune @@ -0,0 +1,6 @@ +(executable + (name fixture) + (modules fixture) + (libraries datascript_ocaml datascript-ocaml-native logseq_db_storage + logseq_db_types logseq_overlay_db melange-transit-core melange-transit-native + sqlite3 unix yojson eio eio_main)) diff --git a/tool/sync_acceptance_fixture/fixture.ml b/tool/sync_acceptance_fixture/fixture.ml new file mode 100644 index 00000000..251118c3 --- /dev/null +++ b/tool/sync_acceptance_fixture/fixture.ml @@ -0,0 +1,457 @@ +module Storage = Logseq_db_storage.Logseq_sqlite_storage +module D = Logseq_overlay_db.Database +module T = Logseq_overlay_db.Types +module G = Logseq_db_types.Graph_types + +let require condition message = if not condition then failwith message + +let ok label = function + | Ok value -> value + | Error message -> failwith (label ^ ": " ^ message) +;; + +let snapshot_error = function + | T.Snapshot_parse_error message -> "parse: " ^ message + | Snapshot_crypto_error message -> "crypto: " ^ message + | Snapshot_commit_persistence_failed message -> "persistence: " ^ message + | Invalid_snapshot_path -> "invalid snapshot path" + | Invalid_expected_rows -> "invalid expected rows" + | Snapshot_input_too_large -> "snapshot input too large" + | Mirror_exists -> "mirror exists" + | Stale_snapshot_inspection -> "stale snapshot inspection" + | Snapshot_limit_exceeded -> "snapshot limit exceeded" + | Snapshot_preparation_canceled -> "snapshot preparation canceled" + | Snapshot_commit_stale -> "snapshot commit stale" + | Snapshot_commit_consumed -> "snapshot commit consumed" + | Snapshot_state_error reason -> + (match reason with + | Snapshot_root_unavailable -> "snapshot root unavailable" + | Snapshot_next_after_persist_or_commit -> "next after persist/commit" + | Snapshot_crypto_batch_pending -> "crypto batch pending" + | Snapshot_supply_after_persist_or_commit -> "supply after persist/commit" + | Snapshot_persist_awaiting_crypto -> "persist awaiting crypto" + | Snapshot_checkpoint_invalid reason -> + "invalid checkpoint: " + ^ + (match reason with + | `Schema_negative -> "negative schema" + | `Checksum_invalid -> "invalid checksum" + | `Unknown -> "unknown")) + | Snapshot_crypto_result_error reason -> + (match reason with + | Crypto_result_missing_item _ -> "crypto result missing item" + | Crypto_result_extra_item _ -> "crypto result extra item" + | Crypto_result_duplicate_item _ -> "crypto result duplicate item" + | Crypto_result_reordered -> "crypto result reordered" + | Crypto_result_stale -> "crypto result stale" + | Crypto_result_limit_exceeded -> "crypto result limit exceeded") +;; + +let snapshot_ok label result = Result.map_error snapshot_error result |> ok label + +let mirror_error = function + | T.Invalid_application_support_directory -> "invalid support directory" + | Mirror_location_unavailable message -> "mirror location: " ^ message + | Mirror_inspection_failed message -> "mirror inspection: " ^ message +;; + +let open_error = function + | T.Mirror_absent -> "mirror absent" + | Invalid_graph_name -> "invalid graph name" + | Ownership_conflict -> "ownership conflict" + | Attachment_stale -> "attachment stale" + | Restore_failed message -> "restore: " ^ message + | Corrupt_authoritative_store message -> "authoritative store: " ^ message + | Corrupt_outbox message -> "outbox: " ^ message + | Corrupt_mutation_receipt message -> "mutation receipt: " ^ message + | Open_resource_failed message -> "resource: " ^ message +;; + +let read_error = function + | T.Database_closed -> "database closed" + | Snapshot_released -> "snapshot released" + | Snapshot_generation_invalidated -> "snapshot generation invalidated" + | Read_limit_exceeded -> "read limit exceeded" + | Stale_read_cursor -> "stale read cursor" + | Invalid_read_request message -> "invalid request: " ^ message + | Fatal_read_state message -> "fatal read: " ^ message +;; + +let storage label = function + | Ok value -> value + | Error error -> failwith (Storage.diagnostic_error ~operation:label error) +;; + +let canonical_path path = + require + (not (Filename.is_relative path) + && not (List.exists (fun part -> part = ".." || part = ".") (String.split_on_char '/' path))) + "explicit path must be absolute without traversal"; + let rec no_symlink current = + require ((Unix.lstat current).Unix.st_kind <> Unix.S_LNK) "symlink in explicit path"; + let parent = Filename.dirname current in + if parent <> current then no_symlink parent + in + no_symlink path; + require (Unix.realpath path = path) "explicit path must be canonical" +;; + +let work_root path = + canonical_path path; + require + (Filename.dirname path = "/Users/rcmerci/Library/Application Support/Journal isolated evidence") + "work root is outside isolated evidence"; + let stat = Unix.lstat path in + require + (stat.Unix.st_kind = Unix.S_DIR && stat.Unix.st_uid = Unix.getuid () + && stat.Unix.st_perm land 0o7777 = 0o700) + "work root must be private and owned"; + path +;; + +let owned_path ?root path = + let parent = Unix.realpath (Filename.dirname path) in + (match root with + | None -> + let prefix = Unix.realpath "_build" ^ "/plain-fixture-" in + require (String.starts_with ~prefix parent) "not a task-owned fixture directory" + | Some root -> + canonical_path (Filename.dirname path); + require (path = Filename.concat parent (Filename.basename path)) "noncanonical fixture path"; + require (Filename.basename path <> "." && Filename.basename path <> "..") "invalid fixture filename"; + let name = Filename.basename parent in + require + (Filename.dirname parent = root + && String.length name = 28 + && String.sub name 0 14 = "plain-fixture-" + && String.for_all + (fun character -> (character >= '0' && character <= '9') || (character >= 'a' && character <= 'f')) + (String.sub name 14 12) + && name.[26] = '-' && (name.[27] = 'A' || name.[27] = 'B')) + "not a task-owned fixture directory"); + let stat = Unix.stat parent in + require + (stat.Unix.st_perm land (if Option.is_some root then 0o7777 else 0o777) = 0o700 + && stat.Unix.st_uid = Unix.getuid ()) + "fixture directory is not private and owned"; + let resolved = Filename.concat parent (Filename.basename path) in + (match Unix.lstat resolved with + | stat -> + require + (stat.Unix.st_kind = Unix.S_REG && stat.Unix.st_uid = Unix.getuid ()) + "fixture path is not an owned regular file" + | exception Unix.Unix_error (Unix.ENOENT, _, _) -> ()); + resolved +;; + +let rewrite source destination graph_id = + require (not (Sys.file_exists destination)) "destination already exists"; + let input = open_in_bin source in + let output = + open_out_gen [ Open_wronly; Open_creat; Open_excl; Open_binary ] 0o600 destination + in + Fun.protect + ~finally:(fun () -> + close_in_noerr input; + close_out_noerr output) + (fun () -> + let buffer = Bytes.create 65536 in + let rec copy () = + let count = Stdlib.input input buffer 0 (Bytes.length buffer) in + if count > 0 + then ( + Stdlib.output output buffer 0 count; + copy ()) + in + copy ()); + let source_connection = Storage.open_database destination |> storage "source open" in + let source_callbacks = Storage.connection_callbacks source_connection in + Fun.protect + ~finally:(fun () -> Storage.close source_callbacks |> storage "source close") + (fun () -> + let database = + Storage.restore_database source_connection |> storage "source restore" + in + let kv ident value = + Datascript.Entity + { db_id = Some (Datascript.Temp_id ident) + ; attrs = + [ "db/ident", Datascript.One_value (Keyword ident) + ; "kv/value", Datascript.One_value value + ] + } + in + let database = + Datascript.db_with + [ kv "logseq.kv/graph-uuid" (Uuid graph_id) + ; kv "logseq.kv/graph-rtc-e2ee?" (Bool false) + ; kv "logseq.kv/graph-remote?" (Bool true) + ] + database + in + let facts = Datascript.datoms database Datascript.Eavt () |> List.of_seq in + let kv_value ident = + let entity = + match + List.find_opt + (fun (fact : Datascript.datom) -> + fact.a = "db/ident" && fact.v = Datascript.Keyword ident) + facts + with + | Some entity -> entity + | None -> failwith ("missing KV ident: " ^ ident) + in + (List.find + (fun (fact : Datascript.datom) -> fact.e = entity.e && fact.a = "kv/value") + facts) + .v + in + require (kv_value "logseq.kv/graph-uuid" = Uuid graph_id) "graph identity mismatch"; + require + (kv_value "logseq.kv/graph-rtc-e2ee?" = Bool false) + "encryption flag mismatch"; + require (kv_value "logseq.kv/graph-remote?" = Bool true) "remote flag mismatch"; + let schema = + match kv_value "logseq.kv/schema-version" with + | Map entries -> + let number key = + match List.assoc (Datascript.Keyword key) entries with + | Datascript.Int64 value -> Int64.to_int value + | _ -> failwith "schema component is not an integer" + in + let major = number "major" + and minor = number "minor" in + require (major = 65 && minor >= 33) "schema is below supported minimum"; + Printf.sprintf "%d.%d" major minor + | _ -> failwith "schema is not a version map" + in + let callbacks = source_callbacks in + callbacks.begin_staging () |> ok "begin staging"; + ignore (Datascript.store ~storage:callbacks.storage database); + let batch = + callbacks.finish_staging + None + [ Datascript.Storage.tail_address, Datascript.Storage_tail [] ] + |> ok "finish staging" + in + Storage.commit_batch callbacks batch |> storage "commit destination"; + schema) +;; + +let frame database_path snapshot_path = + let module Json = Transit_core.Json in + let module Codec = Transit_native.Transit.Json in + let connection = Sqlite3.db_open ~mode:`READONLY database_path in + let statement = + Sqlite3.prepare connection "SELECT addr, content, addresses FROM kvs ORDER BY addr" + in + let rec collect reversed = + match Sqlite3.step statement with + | ROW -> + let address = Sqlite3.column_int statement 0 in + let content = + match Sqlite3.column statement 1 with + | TEXT value -> value + | _ -> failwith "native destination content is not text" + in + let addresses = + match Sqlite3.column statement 2 with + | NULL -> Json.Null + | TEXT value -> Json.String value + | _ -> failwith "native destination addresses are not text/null" + in + collect (Json.Array [ Json.Int address; Json.String content; addresses ] :: reversed) + | DONE -> List.rev reversed + | status -> failwith ("snapshot read: " ^ Sqlite3.Rc.to_string status) + in + let rows = + Fun.protect + ~finally:(fun () -> + ignore (Sqlite3.finalize statement); + require (Sqlite3.db_close connection) "close SQLite") + (fun () -> collect []) + in + let payload = Codec.to_string ~mode:Codec.Verbose (Json.Array rows) in + let length = String.length payload in + require (length + 4 <= 1_000_000) "snapshot exceeds single request limit"; + let prefix = + Bytes.init 4 (fun index -> Char.chr ((length lsr ((3 - index) * 8)) land 255)) + in + let channel = + open_out_gen [ Open_wronly; Open_creat; Open_excl; Open_binary ] 0o600 snapshot_path + in + Fun.protect + ~finally:(fun () -> close_out_noerr channel) + (fun () -> + output_bytes channel prefix; + output_string channel payload); + List.length rows, length + 4 +;; + +let activate root graph_id path expected_rows expected = + Unix.mkdir root 0o700; + let limits : T.capability_limits = + { response_budget_bytes = 8 * 1024 * 1024 + ; outbox_max_records = 4096 + ; outbox_max_bytes = 8 * 1024 * 1024 + ; change_max_items = 4096 + ; change_max_bytes = 8 * 1024 * 1024 + ; dispatcher_capacity = 128 + ; wire_batch_max_bytes = 8 * 1024 * 1024 + } + in + let dependencies = + D.dependencies ~epoch_ms:(fun () -> 0L) ~monotonic_ns:(fun () -> 0L) ~limits + |> Result.map_error (function + | T.Non_positive_limit name -> "non-positive limit: " ^ name + | Inconsistent_limits name -> "inconsistent limits: " ^ name) + |> ok "dependencies" + in + let graph_id = G.Uuid.of_string graph_id |> ok "UUID" in + let inspection = + D.inspect_mirror ~application_support_directory:root ~graph_id + |> Result.map_error mirror_error + |> ok "inspect" + in + let prepared = + D.prepare_snapshot_activation + dependencies + inspection + ~path + ~applied_server_cursor: + (T.Server_cursor.of_int64 0L + |> Result.map_error (fun `Negative_cursor -> "negative cursor") + |> ok "cursor") + ~expected_checksum:expected + ~expected_rows + |> snapshot_ok "snapshot preparation" + in + Fun.protect + ~finally:(fun () -> D.cancel_snapshot_activation prepared) + (fun () -> + let rec plaintext () = + match + D.next_snapshot_unprotection_batch prepared |> snapshot_ok "plaintext batch" + with + | None -> () + | Some request -> + D.supply_snapshot_unprotection_batch + prepared + ~request + ~plaintexts:(D.unprotection_ciphertexts request) + |> snapshot_ok "plaintext supply"; + plaintext () + in + plaintext (); + let inspection = + D.commit_snapshot_activation prepared |> snapshot_ok "snapshot activation" + in + match D.mirror_presence inspection with + | T.Available { checksum = Some checksum; _ } -> + Eio.Switch.run (fun sw -> + let database = + D.open_ ~sw dependencies inspection ~graph_name:"Owned acceptance fixture" + |> Result.map_error open_error + |> ok "open activated graph" + in + Fun.protect + ~finally:(fun () -> + D.close database + |> Result.map_error (function + | T.Close_failed message -> "close: " ^ message + | Fatal_close message -> "fatal close: " ^ message) + |> ok "close activated graph") + (fun () -> + let snapshot = + D.current_snapshot database + |> Result.map_error read_error + |> ok "current snapshot" + in + Fun.protect + ~finally:(fun () -> D.release_snapshot snapshot) + (fun () -> + let info = + D.graph_info snapshot + |> Result.map_error read_error + |> ok "graph info" + in + require + (G.Uuid.to_string info.graph_uuid = G.Uuid.to_string graph_id) + "activated identity mismatch"; + require + (info.schema.major = 65 && info.schema.minor >= 33) + "activated schema mismatch"))); + checksum + | _ -> failwith "activated checksum unavailable") +;; + +let () = + try + Eio_main.run (fun _ -> + let root, arguments = + if Array.length Sys.argv >= 3 && Sys.argv.(1) = "--work-root" + then Some (work_root Sys.argv.(2)), Array.sub Sys.argv 2 (Array.length Sys.argv - 2) + else None, Sys.argv + in + let owned_path = owned_path ?root in + require + (Array.length arguments = 5) + "usage: fixture [--work-root ROOT] SOURCE DESTINATION UUID SNAPSHOT | --verify-frame PATH UUID ROWS"; + if arguments.(1) = "--verify-frame" + then ( + let path = owned_path arguments.(2) in + let graph_id = arguments.(3) in + let rows = int_of_string arguments.(4) in + let checksum = activate (path ^ ".fresh-read") graph_id path rows None in + let repeated = + activate (path ^ ".fresh-verify") graph_id path rows (Some checksum) + in + require (T.Checksum.equal checksum repeated) "remote snapshot checksum mismatch"; + print_endline + (Yojson.Safe.to_string + (`Assoc + [ "graph_id", `String graph_id + ; "checksum", `String (T.Checksum.to_string checksum) + ; "fresh_read_open_current", `Bool true + ]))) + else ( + let source = owned_path arguments.(1) in + let destination = owned_path arguments.(2) in + let graph_id = arguments.(3) in + ignore (G.Uuid.of_string graph_id |> ok "UUID"); + let snapshot = owned_path arguments.(4) in + let schema = rewrite source destination graph_id in + let rows, bytes = frame destination snapshot in + let checksum = activate (snapshot ^ ".activation") graph_id snapshot rows None in + let repeated = + activate (snapshot ^ ".verification") graph_id snapshot rows (Some checksum) + in + require (T.Checksum.equal checksum repeated) "checksum parity"; + let inspected = + D.inspect_snapshot_storage + ~check_current:(fun () -> ()) + ~scratch_bytes:16_777_216L + ~encrypted:false + ~path:destination + |> ok "storage inspector" + in + require + (T.Checksum.equal checksum inspected) + "storage inspector checksum mismatch"; + print_endline + (Yojson.Safe.to_string + (`Assoc + [ "graph_id", `String graph_id + ; "schema", `String schema + ; "rows", `Int rows + ; "bytes", `Int bytes + ; "checksum", `String (T.Checksum.to_string checksum) + ; "encrypted", `Bool false + ; "cursor", `Int 0 + ; "public_activation_verified", `Bool true + ])))) + with + | exception_value -> + prerr_endline (Printexc.to_string exception_value); + exit 1 +;; diff --git a/tool/sync_acceptance_fixture/provision.py b/tool/sync_acceptance_fixture/provision.py new file mode 100644 index 00000000..b1f7d0b7 --- /dev/null +++ b/tool/sync_acceptance_fixture/provision.py @@ -0,0 +1,371 @@ +#!/usr/bin/env python3 +"""Provision only new task-owned plaintext acceptance graphs, with explicit consent.""" +import argparse +import base64 +import hashlib +import gzip +import io +import json +import os +from pathlib import Path +import re +import secrets +import stat +import subprocess +import sys +import urllib.parse +import uuid + +REPO = Path(__file__).resolve().parents[2] +REGISTRY = REPO / "_build/owned-plain-fixtures.json" +ORIGIN = "https://api.logseq.io" +PREFIX = "SYNC-ACCEPT-PLAIN-20261119-" +SAFE_ENV = {key: os.environ[key] for key in ( + "PATH", "HOME", "TMPDIR", "OPAM_SWITCH_PREFIX", "CAML_LD_LIBRARY_PATH", "OCAMLPATH" +) if key in os.environ} + + +def configured_paths(parser, options): + if (options.registry is None) != (options.work_root is None): + parser.error("--registry and --work-root must be supplied together") + if options.work_root is None: + return REGISTRY, REPO / "_build" + values = (options.registry, options.work_root) + paths = [Path(value) for value in values] + for value, path in zip(values, paths): + if str(path) != value or str(path.resolve()) != value: + parser.error("explicit paths must be canonical") + if not path.is_absolute() or ".." in path.parts: + parser.error("explicit paths must be absolute without parent traversal") + if any(parent.is_symlink() for parent in (path, *path.parents)): + parser.error("explicit paths must not contain symlinks") + registry_path, work_root = [path.resolve() for path in paths] + evidence = (Path.home() / "Library/Application Support/Journal isolated evidence").resolve() + if work_root.parent != evidence or registry_path.parent != work_root: + parser.error("work root must be a task directory directly under Journal isolated evidence; registry must be directly inside it") + if registry_path == REGISTRY.resolve(): + parser.error("explicit registry must not be the legacy registry") + if not work_root.exists(): + work_root.mkdir(mode=0o700) + status = work_root.lstat() + if (not stat.S_ISDIR(status.st_mode) or status.st_uid != os.getuid() + or stat.S_IMODE(status.st_mode) != 0o700): + parser.error("work root must be a private owned 0700 directory") + return registry_path, work_root + + +def fixture_base(record, registry, work_root): + stored = Path(record["fixture_directory"]) + base = stored if stored.is_absolute() else REPO / stored + expected = work_root / ("plain-fixture-" + registry["nonce"] + "-" + record["role"]) + if (".." in stored.parts or base != expected or base.resolve() != expected + or base.is_symlink() or base.parent.resolve() != work_root): + raise RuntimeError("fixture directory escaped owned scope") + status = base.lstat() + if (not stat.S_ISDIR(status.st_mode) or status.st_uid != os.getuid() + or stat.S_IMODE(status.st_mode) != 0o700): + raise RuntimeError("fixture directory must be private and owned") + return base + + +def recorded_preflight(registry, work_root, verify): + if not re.fullmatch(r"[0-9a-f]{12}", registry["nonce"]): + raise RuntimeError("invalid registry nonce") + for record in registry["fixtures"]: + if (record["role"] not in ("A", "B") + or record["name"] != PREFIX + registry["nonce"] + "-" + record["role"]): + raise RuntimeError("registry nonce/role mismatch") + uuid.UUID(record["id"]) + acknowledgement = record.get("creation_ack") + if (not acknowledgement or acknowledgement.get("graph-id") != record["id"] + or acknowledgement.get("graph-e2ee?") is not False + or record["creator_subject_sha256"] != registry["creator_subject_sha256"]): + raise RuntimeError("ownership guard failed") + base = fixture_base(record, registry, work_root) + path = base / "remote.transit" + if verify and not record.get("deleted") and (path.exists() or path.is_symlink()): + raise RuntimeError("remote verification already exists") + + +def save(registry, registry_path): + temporary = registry_path.with_name(registry_path.name + ".pending") + descriptor = os.open(temporary, os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0o600) + with os.fdopen(descriptor, "w") as output: + json.dump(registry, output, indent=2) + output.write("\n") + output.flush() + os.fsync(output.fileno()) + os.replace(temporary, registry_path) + directory = os.open(registry_path.parent, os.O_RDONLY) + try: + os.fsync(directory) + finally: + os.close(directory) + + +def run(arguments, data=None): + result = subprocess.run(arguments, input=data, capture_output=True, env=SAFE_ENV, + cwd=REPO, timeout=90) + if result.returncode: + raise RuntimeError(f"subprocess failed: {Path(arguments[0]).name}, exit {result.returncode}") + return result.stdout + + +def cli(base, name, command): + return run(["logseq", "--root-dir", str(base / "cli"), "--config", + str(base / "cli.edn"), "--graph", name, *command]) + + +def local(base, name): + (base / "cli").mkdir(mode=0o700) + config = base / "cli.edn" + config.write_text('{:auth-path ' + json.dumps(str(base / "absent-auth.json")) + '}\n') + config.chmod(0o600) + try: + cli(base, name, ["graph", "create"]) + cli(base, name, ["graph", "export", "--type", "sqlite", "--file", str(base / "source.sqlite")]) + finally: + cli(base, name, ["server", "stop"]) + if (base / "absent-auth.json").exists(): + raise RuntimeError("isolated auth file unexpectedly exists") + + +def native_command(work_root): + command = [str(REPO / "_build/default/tool/sync_acceptance_fixture/fixture.exe")] + if work_root != REPO / "_build": + command.extend(["--work-root", str(work_root)]) + return command + + +def preflight(base, identifier, label, work_root): + response = run([*native_command(work_root), + str(base / "source.sqlite"), str(base / (label + ".sqlite")), + identifier, str(base / (label + ".transit"))]) + report = json.loads(response) + if not report["public_activation_verified"] or report["encrypted"] or report["bytes"] > 1_000_000: + raise RuntimeError("public snapshot preflight failed") + return report + + +def authenticate(): + payload = json.dumps({"AuthFlow": "USER_PASSWORD_AUTH", + "ClientId": "69cs1lgme7p8kbgld8n5kseii6", + "AuthParameters": {"USERNAME": os.environ["LOGSEQ_ACCOUNT"], + "PASSWORD": os.environ["LOGSEQ_ACCOUNT_PASSWORD"]}}).encode() + response = run(["/usr/bin/curl", "--disable", "--silent", "--show-error", + "--fail-with-body", "--noproxy", "*", "--connect-timeout", "15", "--max-time", "60", + "--header", "content-type: application/x-amz-json-1.1", + "--header", "x-amz-target: AWSCognitoIdentityProviderService.InitiateAuth", + "--data-binary", "@-", "https://cognito-idp.us-east-1.amazonaws.com/"], payload) + token = json.loads(response)["AuthenticationResult"]["IdToken"] + payload_part = token.split(".")[1] + subject = json.loads(base64.urlsafe_b64decode(payload_part + "=" * (-len(payload_part) % 4)))["sub"] + return token, hashlib.sha256(subject.encode()).hexdigest() + + +def request(token, method, path, body=None, snapshot=None, raw=False): + if not path.startswith("/graphs") and not path.startswith("/sync/"): + raise RuntimeError("endpoint is outside fixture protocol") + config = {"url": ORIGIN + path, "request": method, + "header": "Authorization: Bearer " + token} + lines = [key + " = " + json.dumps(value) for key, value in config.items()] + if body is not None: + lines.extend(['header = "Content-Type: application/json"', + "data-binary = " + json.dumps(json.dumps(body))]) + if snapshot is not None: + lines.extend(['header = "Content-Type: application/transit+json"', + "data-binary = " + json.dumps("@" + str(snapshot))]) + result = subprocess.run(["/usr/bin/curl", "--disable", "--silent", "--show-error", "--noproxy", "*", + "--compressed", "--max-filesize", "2000000", + "--connect-timeout", "15", "--max-time", "60", "--config", "-", + "--write-out", "\n%{http_code}"], + input=("\n".join(lines) + "\n").encode(), capture_output=True, + env=SAFE_ENV, timeout=70) + if result.returncode: + raise RuntimeError("HTTP transport outcome indeterminate; no automatic retry") + response, status = result.stdout.rsplit(b"\n", 1) + if not 200 <= int(status) < 300: + raise RuntimeError("HTTP status " + status.decode()) + if raw: + if response.startswith(b"\x1f\x8b"): + with gzip.GzipFile(fileobj=io.BytesIO(response)) as stream: + response = stream.read(1_000_001) + if len(response) > 1_000_000: + raise RuntimeError("remote frame exceeds bounded fixture size") + return response + return json.loads(response) + + +def catalog(token): + return request(token, "GET", "/graphs")["graphs"] + + +def own_match(record, graphs, creator): + if (not record.get("creation_ack") or record["creator_subject_sha256"] != creator + or record["creation_ack"].get("graph-id") != record["id"] + or record["creation_ack"].get("graph-e2ee?") is not False + or record["role"] not in ("A", "B") + or not record["name"].endswith("-" + record["role"]) + or not re.fullmatch(PREFIX + r"[0-9a-f]{12}-[AB]", record["name"])): + raise RuntimeError("ownership guard failed") + matches = [graph for graph in graphs if graph["graph-id"] == record["id"] + and graph["graph-name"] == record["name"]] + if len(matches) != 1: + raise RuntimeError("exact acknowledged UUID/name absent or ambiguous") + return matches[0] + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + actions = parser.add_mutually_exclusive_group(required=True) + actions.add_argument("--create-two-new-plaintext-graphs", action="store_true") + actions.add_argument("--verify-recorded-owned-fixtures", action="store_true") + actions.add_argument("--delete-recorded-owned-fixtures", action="store_true") + parser.add_argument("--registry", help="Private ownership registry inside --work-root (requires both options)") + parser.add_argument("--work-root", help="Private task directory under Journal isolated evidence (requires --registry)") + options = parser.parse_args() + os.umask(0o077) + registry_path, work_root = configured_paths(parser, options) + if not registry_path.exists() and not registry_path.is_symlink() and options.create_two_new_plaintext_graphs: + initial = {"temporary": True, "scope": "task-created plaintext acceptance graphs only", "fixtures": []} + if options.work_root is not None: + initial["work_root"] = str(work_root) + save(initial, registry_path) + status = registry_path.lstat() + if (registry_path.is_symlink() or not stat.S_ISREG(status.st_mode) + or status.st_uid != os.getuid() or stat.S_IMODE(status.st_mode) != 0o600 + or status.st_nlink != 1): + raise RuntimeError("ownership registry must be private and owned") + registry = json.loads(registry_path.read_text()) + if options.work_root is not None and registry.get("work_root") != str(work_root): + raise RuntimeError("registry work root does not match configuration") + if options.verify_recorded_owned_fixtures or options.delete_recorded_owned_fixtures: + recorded_preflight(registry, work_root, options.verify_recorded_owned_fixtures) + token, creator = authenticate() + if registry["creator_subject_sha256"] != creator: + raise RuntimeError("authenticated creator does not match registry") + for record in registry["fixtures"]: + if record.get("deleted"): + continue + if record["name"] != PREFIX + registry["nonce"] + "-" + record["role"]: + raise RuntimeError("registry nonce/role mismatch") + graph = own_match(record, catalog(token), creator) + record["cleanup_guard_verified"] = True + if options.delete_recorded_owned_fixtures: + receipt = request(token, "DELETE", "/graphs/" + record["id"]) + if receipt.get("deleted") is not True or receipt.get("graph-id") != record["id"]: + raise RuntimeError("deletion receipt mismatch") + record["deleted"] = True + else: + if graph["graph-e2ee?"] is not False or graph["graph-ready-for-use?"] is not True: + raise RuntimeError("fixture is not ready plaintext") + base = fixture_base(record, registry, work_root) + path = base / "remote.transit" + if path.exists() or path.is_symlink(): + raise RuntimeError("remote verification already exists") + frame = request(token, "GET", "/sync/" + record["id"] + "/snapshot/stream", raw=True) + with path.open("xb") as output: + output.write(frame) + path.chmod(0o600) + report = json.loads(run([*native_command(work_root), + "--verify-frame", str(path), record["id"], str(record["rows"])])) + if report["checksum"] != record["checksum"]: + raise RuntimeError("fresh remote checksum differs from upload") + record["remote_fresh_read"] = report + record["remote_frame_sha256"] = hashlib.sha256(path.read_bytes()).hexdigest() + save(registry, registry_path) + print(json.dumps({"id": record["id"], "cleanup_guard_verified": True, + "deleted": record.get("deleted", False), + "fresh_read": record.get("remote_fresh_read")})) + return + if registry["fixtures"] or "nonce" in registry or "indeterminate_creation" in registry: + raise RuntimeError("registry records a creation attempt; refusing new provisioning; reconcile its existing nonce and outcomes manually") + nonce = secrets.token_hex(6) + registry.update(nonce=nonce, temporary=True, origin=ORIGIN, + head=run(["git", "rev-parse", "HEAD"]).decode().strip()) + save(registry, registry_path) + bases = [] + for role in ("A", "B"): + name = PREFIX + nonce + "-" + role + base = work_root / ("plain-fixture-" + nonce + "-" + role) + base.mkdir(mode=0o700) + bases.append((base, name, role)) + local(base, name) + report = preflight(base, "92000000-0000-4000-8000-000000000001", "preflight", work_root) + registry.setdefault("local_preflights", []).append({"name": name, **report}) + save(registry, registry_path) + token, creator = authenticate() + registry["creator_subject_sha256"] = creator + save(registry, registry_path) + existing = catalog(token) + if any(graph["graph-name"] in [name for _, name, _ in bases] for graph in existing): + raise RuntimeError("nonce name already exists; no adoption or mutation allowed") + try: + for base, name, role in bases: + try: + ack = request(token, "POST", "/graphs", { + "graph-name": name, "schema-version": "65", "graph-e2ee?": False, + "graph-ready-for-use?": False}) + except Exception: + matches = [graph for graph in catalog(token) if graph["graph-name"] == name] + registry["indeterminate_creation"] = {"name": name, + "matching_ids": [graph["graph-id"] for graph in matches]} + save(registry, registry_path) + raise + identifier = ack["graph-id"] + record = {"id": identifier, "name": name, "role": role, "creation_ack": ack, + "creator_subject_sha256": creator, "ready": False, + "fixture_directory": str(base) if options.work_root is not None else str(base.relative_to(REPO))} + registry["fixtures"].append(record) + save(registry, registry_path) + uuid.UUID(identifier) + if ack.get("graph-e2ee?") is not False or ack.get("graph-ready-for-use?") is not False: + raise RuntimeError("creation acknowledgement flags are not plaintext/unready") + report = preflight(base, identifier, "upload", work_root) + record.update(report) + save(registry, registry_path) + checksum = report["checksum"].removeprefix("checksum:v1:") + if not re.fullmatch(r"[0-9a-f]{16}", checksum): + raise RuntimeError("invalid checksum wire representation") + own_match(record, catalog(token), creator) + record["upload_receipt"] = request(token, "POST", "/sync/" + identifier + + "/snapshot/upload?reset=true&finished=true&checksum=" + + urllib.parse.quote(checksum, safe=""), snapshot=base / "upload.transit") + save(registry, registry_path) + graph = own_match(record, catalog(token), creator) + record["catalog"] = {key: graph[key] for key in ( + "graph-id", "graph-name", "schema-version", "graph-e2ee?", "graph-ready-for-use?", "role")} + if graph["graph-e2ee?"] is not False or graph["graph-ready-for-use?"] is not True: + raise RuntimeError("uploaded graph not catalog-ready plaintext") + record["ready"] = True + record["snapshot_metadata"] = request(token, "GET", "/sync/" + identifier + "/snapshot/download") + record["snapshot_metadata"] = {key: value for key, value in record["snapshot_metadata"].items() + if key not in ("url", "download-url")} + save(registry, registry_path) + print(json.dumps({key: record[key] for key in ( + "id", "name", "ready", "schema", "checksum", "bytes", "rows", "encrypted")})) + except Exception: + for record in registry["fixtures"]: + if record.get("ready") or record.get("deleted"): + continue + try: + graph = own_match(record, catalog(token), creator) + if graph["graph-ready-for-use?"]: + record["cleanup_deferred"] = "server-ready; retained for Main" + else: + receipt = request(token, "DELETE", "/graphs/" + record["id"]) + record["deleted"] = receipt.get("deleted") is True + record["cleanup_guard_verified"] = True + save(registry, registry_path) + except Exception: + record["cleanup_indeterminate"] = True + save(registry, registry_path) + raise + + +if __name__ == "__main__": + try: + main() + except Exception as error: + print("fixture provisioning stopped: " + str(error), file=sys.stderr) + sys.exit(1) diff --git a/tool/sync_acceptance_probe/README.md b/tool/sync_acceptance_probe/README.md new file mode 100644 index 00000000..b5180d05 --- /dev/null +++ b/tool/sync_acceptance_probe/README.md @@ -0,0 +1,112 @@ +# Separate Simulator sync-network probe + +This diagnostic entry registers the real `Application.For_testing.app_with_service` +with the production facade and its production overlay, auth, calendar and platform +dependencies. Only the facade's Registration network capability is wrapped. +`app/native_embed.ml` and the ordinary service never read the probe environment. +No SyncService, DSL or Execute public API changes are involved. + +## Build and ownership gate + +Arrange independent read-only review of the final diff **before producing a new +Simulator App or doing UI acceptance**. Source/object compilation is not runtime +acceptance. The main acceptance owner alone creates/installs/launches the new +Simulator; do not reuse ordinary A/B devices or bundles. + +Use the maintained producer/importer, not raw object injection. After review: + +```sh +BUILD=.agents/skills/journal-build/scripts/build_journal_apple.sh +LUI="$HOME/.opam/logseq-journal-lui/.opam-switch/sources/lui/platform/apple" +bash "$BUILD" macos --producer-only --sync-network-probe \ + --switch logseq-journal-lui --lui-package "$LUI" --cache-dir "$EVIDENCE/build-cache" +# ENVELOPE is producer.envelope.path from this run's manifest, not an old receipt. +bash "$BUILD" ios-simulator --sync-network-probe --experimental-host-restamp \ + --native-manifest "$ENVELOPE" --lui-package "$LUI" --cache-dir "$EVIDENCE/build-cache" +``` + +Select the existing prepared locked LUI path from the current switch/manifest if +it differs; never move it, re-pin dependencies, or edit immutable old evidence. +Unset ambient tool lookup overrides as required by the builder. `$EVIDENCE` must +be an owned external directory. These build commands never install or launch. + +Alternatively the authorized main owner can run the maintained +`.agents/skills/journal-build/scripts/run_isolated_simulator.py` with its existing +required evidence/switch/LUI/runtime/device-type arguments plus +`--sync-network-probe`. It passes the role to both builder invocations, verifies +current sources, creates a fresh device, and records the private control paths in +`session.json.network_probe`. Do not run this install/launch route merely to +validate the tooling. + +The explicit role selects only `tool/sync_acceptance_probe/native_probe.exe.o`. +Role/target, native probe sources and current Dune/lock configuration are bound +in producer evidence. Cross-role imports fail closed. The default production +target and commit-based build-configuration guard are unchanged. Probe builds +use the real maintained Swift sources, the `logseqjournal` auth redirect scheme, +bundle ID `com.logseq.journal.sync-probe`, and `JournalSyncNetworkProbe=true` in +the packaged plist. Both manifests record `production_acceptance=false` and keep +`cross_abi_validated=false`. Host restamping is experimental, not verified iOS +cross-compilation or ordinary production acceptance. + +## Launch inputs and signal protocol + +For a manual launch by the main owner, create a **fresh** directory in the new +probe App's data-container `tmp` directory. Resolve its absolute canonical path, +keep mode 0700 and current-user ownership, and pass only that path as +`SIMCTL_CHILD_JOURNAL_SYNC_NETWORK_PROBE_DIRECTORY` to `simctl launch`. +The test entry reads `JOURNAL_SYNC_NETWORK_PROBE_DIRECTORY`; ordinary entries do +not read it. No account/password/graph secret environment is inspected by this +probe. Login remains the real App UI's responsibility. + +The instance exclusively creates `status.json` (0600, O_CREAT|O_EXCL). O_EXCL +also rejects a pre-existing final symlink; the retained descriptor is never +reopened through a pathname. Existing status files reject reuse: use a fresh +private directory for every service instance/launch. The directory contains only +local control/status, no graph data, credentials or payloads. + +- Online: the `offline` signal file is absent. +- Offline: create an empty, owned mode-0600 regular `offline` file exclusively. +- Online again: remove only that owned signal file. +- Wait for the observed status transition before changing the signal again. + Rapid unobserved create/remove pulses are not a supported fault command. + +The switch-owned fiber samples every 100ms. An offline transition physically +shuts down both directions of every tracked live stream; it does **not** close +those resources. New stream connects fail with an explicit injected +`Unix.ENETDOWN` backend inside an Eio network connection error. A connect that +suspends across a fault generation is shut down and rejected before return, +even when online has already resumed. Original networking, flows and TLS remain +real outside faults. Original flow operations are forwarded unchanged; the +close hook removes its own ledger entry and release hook. The ledger is bounded +to 256 simultaneous streams; overflow is a diagnostic error, not a silently +untracked connection. Switch release removes entries, cancels/joins the monitor, +and closes its status descriptor. There is no global cache or production state. + +This is **Registration stream fault injection**, not the OS radio being offline. +It covers actual sync/WebSocket and HTTP assets using that capability. DNS, +UDP/QUIC, Cognito/CFNetwork and other capabilities are not claimed offline. +No DYLD/SPI injection, host firewall, fd scan, daemon, re-signing of existing Apps, +or host protection change is used. + +## Main acceptance status gates + +The bounded numeric schema is exactly: + +```json +{"schema":1,"heartbeat":1,"offline":0,"generation":0,"live_flows":0,"connect_failures":0,"shutdowns":0,"errors":0} +``` + +All counters saturate at 1,000,000,000; `live_flows` is 0..256 and `offline` is +0 or 1. Reads can race a small in-place status write: retry an incomplete JSON +read within the supervision window; never accept an old sample as liveness. +After the file first appears, require heartbeat advancement within **500ms** +throughout the acceptance case; stale/missing status or `errors != 0` invalidates +the case. Cold App startup has its own explicit timeout, not an assumed heartbeat. + +For offline acceptance require `offline=1` and an increased `generation`. +To claim an existing live connection was cut, require a positive shutdown delta; +to claim a new connect was refused require a positive `connect_failures` delta. +For recovery require `offline=0`, another generation increment, a live heartbeat +and actual UI-observed synchronization. Counter changes alone are not evidence +of persistent offline/retry/conflict behavior or an actual server pass. Main must +run the actual App UI cases; no unit tests of this diagnostic substitute for them. diff --git a/tool/sync_acceptance_probe/dune b/tool/sync_acceptance_probe/dune new file mode 100644 index 00000000..be3d16dc --- /dev/null +++ b/tool/sync_acceptance_probe/dune @@ -0,0 +1,9 @@ +(executable + (name native_probe) + (modules native_probe test_net) + (libraries app lui logseq_db_worker.lui eio eio.unix unix) + (link_flags + (:standard + (:include ../../app/native_static_link_flags.sexp))) + (modes + (native object))) diff --git a/tool/sync_acceptance_probe/native_probe.ml b/tool/sync_acceptance_probe/native_probe.ml new file mode 100644 index 00000000..8ad83b3f --- /dev/null +++ b/tool/sync_acceptance_probe/native_probe.ml @@ -0,0 +1,8 @@ +let () = + let directory = Sys.getenv "JOURNAL_SYNC_NETWORK_PROBE_DIRECTORY" in + let service = + Logseq_db_worker_lui.Logseq_db_worker_lui_service.For_testing.service_with_network + (Test_net.wrap ~directory) + in + Journal_bridge.register (Application.For_testing.app_with_service service) +;; diff --git a/tool/sync_acceptance_probe/test_net.ml b/tool/sync_acceptance_probe/test_net.ml new file mode 100644 index 00000000..6eaa0ad8 --- /dev/null +++ b/tool/sync_acceptance_probe/test_net.ml @@ -0,0 +1,174 @@ +module Journal_worker = Logseq_db_worker_lui.Journal_worker + +let bounded_increment value = if !value < 1_000_000_000 then incr value + +let private_directory path = + let canonical = Unix.realpath path in + let stat = Unix.lstat path in + if + canonical <> path + || stat.Unix.st_kind <> Unix.S_DIR + || stat.Unix.st_perm land 0o7777 <> 0o700 + || stat.Unix.st_uid <> Unix.getuid () + then invalid_arg "Probe directory must be canonical, owned and mode 0700"; + canonical +;; + +let wrap ~directory ~sw ~clock (original : Journal_worker.net) : Journal_worker.net = + let directory = private_directory directory in + let status = + Unix.openfile + (Filename.concat directory "status.json") + [ Unix.O_WRONLY; Unix.O_CREAT; Unix.O_EXCL; Unix.O_CLOEXEC ] + 0o600 + in + Eio.Switch.on_release sw (fun () -> Unix.close status); + Unix.fchmod status 0o600; + let offline = ref false in + let generation = ref 0 in + let heartbeat = ref 0 in + let connect_failures = ref 0 in + let shutdowns = ref 0 in + let errors = ref 0 in + let flows = Hashtbl.create 256 in + let next_token = ref 0L in + let fail_network () = + bounded_increment connect_failures; + raise + (Eio.Net.err + (Connection_failure + (Refused (Eio_unix.Unix_error (Unix.ENETDOWN, "sync-network-probe", ""))))) + in + let shutdown flow = + match Eio.Flow.shutdown flow `All with + | () -> bounded_increment shutdowns + | exception Eio.Io _ -> bounded_increment errors + | exception Unix.Unix_error _ -> bounded_increment errors + in + let read_control () = + let path = Filename.concat directory "offline" in + match Unix.lstat path with + | stat -> + if + stat.Unix.st_kind <> Unix.S_REG + || stat.Unix.st_uid <> Unix.getuid () + || stat.Unix.st_perm land 0o7777 <> 0o600 + || stat.Unix.st_size <> 0 + then ( + bounded_increment errors; + true) + else true + | exception Unix.Unix_error (Unix.ENOENT, _, _) -> false + | exception Unix.Unix_error _ -> + bounded_increment errors; + true + in + let publish () = + let text = + Printf.sprintf + "{\"schema\":1,\"heartbeat\":%d,\"offline\":%d,\"generation\":%d,\"live_flows\":%d,\"connect_failures\":%d,\"shutdowns\":%d,\"errors\":%d}\n" + !heartbeat + (if !offline then 1 else 0) + !generation + (Hashtbl.length flows) + !connect_failures + !shutdowns + !errors + in + ignore (Unix.lseek status 0 Unix.SEEK_SET); + let bytes = Bytes.of_string text in + let rec write offset = + if offset < Bytes.length bytes + then ( + let count = Unix.write status bytes offset (Bytes.length bytes - offset) in + if count = 0 then failwith "Probe status write stopped" else write (offset + count)) + in + write 0; + Unix.ftruncate status (Bytes.length bytes) + in + let reconcile () = + let requested = read_control () in + if requested <> !offline + then ( + offline := requested; + bounded_increment generation; + if requested + then ( + let live = Hashtbl.fold (fun token flow live -> (token, flow) :: live) flows [] in + List.iter + (fun (token, flow) -> + match Hashtbl.find_opt flows token with + | Some current when current == flow -> shutdown flow + | _ -> ()) + live)) + in + reconcile (); + publish (); + Eio.Fiber.fork_daemon ~sw (fun () -> + while true do + Eio.Time.Mono.sleep clock 0.1; + reconcile (); + bounded_increment heartbeat; + publish () + done; + `Stop_daemon); + let module Network = struct + type t = Journal_worker.net + type tag = [ `Generic ] + + let connect net ~sw address = + reconcile (); + if !offline then fail_network (); + if Hashtbl.length flows >= 256 || !next_token = Int64.max_int + then ( + bounded_increment errors; + fail_network ()); + let started = !generation in + let flow = Eio.Net.connect ~sw net address in + reconcile (); + if !offline || started <> !generation + then ( + shutdown flow; + fail_network ()); + if Hashtbl.length flows >= 256 || !next_token = Int64.max_int + then ( + bounded_increment errors; + shutdown flow; + fail_network ()); + let token = !next_token in + next_token := Int64.succ token; + Hashtbl.add flows token flow; + let release () = + match Hashtbl.find_opt flows token with + | Some current when current == flow -> Hashtbl.remove flows token + | _ -> () + in + let hook = Eio.Switch.on_release_cancellable sw release in + let (Eio.Resource.T (resource, handlers)) = flow in + let original_close = Eio.Resource.get handlers Eio.Resource.Close in + let close resource = + release (); + Eio.Switch.remove_hook hook; + original_close resource + in + Eio.Resource.T + ( resource + , Eio.Resource.handler + (Eio.Resource.H (Eio.Resource.Close, close) :: Eio.Resource.bindings handlers) + ) + ;; + + let listen net ~reuse_addr ~reuse_port ~backlog ~sw address = + Eio.Net.listen net ~reuse_addr ~reuse_port ~backlog ~sw address + ;; + + let datagram_socket net ~reuse_addr ~reuse_port ~sw address = + Eio.Net.datagram_socket net ~reuse_addr ~reuse_port ~sw address + ;; + + let getaddrinfo net ~service host = Eio.Net.getaddrinfo net ~service host + let getnameinfo net address = Eio.Net.getnameinfo net address + end + in + Eio.Resource.T (original, Eio.Net.Pi.network (module Network)) +;; diff --git a/tool/test_asset_interop.py b/tool/test_asset_interop.py index bb23c5b3..7e0ab80e 100644 --- a/tool/test_asset_interop.py +++ b/tool/test_asset_interop.py @@ -23,11 +23,11 @@ record = {"commands": [], "sources": {}, "result": "incomplete"} for source in [crypt, upstream / "deps/db/src/logseq/db/sqlite/util.cljs", root / "swift/JournalE2EECrypto.swift", - root / "logseq_sync/lib/effect_runner/protocol/asset_codec.ml", + root / "logseq_sync/lib/native_crypto/asset_codec.ml", root / "logseq_sync/test/asset_native_interop.ml", root / "logseq_sync/test/asset_adapter_memory.ml", - root / "logseq_sync/lib/effect_runner/platform/platform_crypto.ml", - root / "logseq_sync/lib/effect_runner/platform/platform_crypto_stubs.c", + root / "logseq_sync/lib/native_crypto/platform_crypto.ml", + root / "logseq_sync/lib/native_crypto/platform_crypto_stubs.c", root / "tool/asset_interop.cljs", Path(__file__).resolve()]: record["sources"][str(source)] = hashlib.sha256(source.read_bytes()).hexdigest() record["upstreamCommit"] = subprocess.check_output(