fix(replication): converge delete-marker purge retries and MRF healing

[julienlau]

Follow-up to #162; completes the delete-marker convergence work (context: #152, #153, and docs/site-replication/delete-marker-convergence.md in this PR).

Problem

#162 fixed the first-attempt classification of delete-marker purges (DeleteObjectHandler, heal and resync paths now send the purge as a version purge) and made MRF drops observable. Three defects remained on the delete-marker path, and they re-create the non-convergence loop described in #153 whenever a delivery fails once (the norm with N>2 sites):

1. Purge retries short-circuited by creation status. In replicateDeleteToTarget, the "already replicated" early-out tested PrevReplicationStatus (the marker's creation status) even for a purge, so a purge whose first delivery to a site failed was never retried — that site kept the marker forever. Only a manual mc admin replicate resync bypassed it.
2. Purge outcome recorded in the wrong field. On RemoveObject success for a marker-path purge (VersionID == ""), the code set ReplicationStatus = Completed but never VersionPurgeStatus = VersionPurgeComplete: the source's PurgeTargets stayed PENDING forever, the scanner re-queued the purge every cycle, and the marker version never left xl.meta.
3. MRF dropped every delete-marker entry. queueMRFHeal calls GetObjectInfo(VersionID=<marker version>), which deterministically answers 405 MethodNotAllowed; the entry was silently continued, so the fastest retry path never ran for deletes.

Residual effect: stuck PENDING states drove perpetual heal/resync traffic whose per-marker probes are answered 405 — the storm from #153 (measured there at ~4,100 req/s, 96.8% answered 405 in an 84M-version mesh).

Fix

In replicateDeleteToTarget, a marker-path purge is identified precisely:

isDMPurge := dobj.VersionID == "" && dobj.DeleteMarkerVersionID != "" && !rinfo.VersionPurgeStatus.Empty()

• the early-out is guarded with !isDMPurge and the marker's creation status is pinned (rinfo.ReplicationStatus = rinfo.PrevReplicationStatus) so purges neither get suppressed by nor overwrite it;
• the offline / error / success branches record purge outcomes in VersionPurgeStatus whenever dobj.VersionID != "" || isDMPurge — the composite reaches VersionPurgeComplete, the source rewrite takes the xlMetaV2.DeleteVersion removal path and the marker version is dropped;
• queueMRFHeal accepts the 405 (it carries a fully populated ObjectInfo) and routes the entry through QueueReplicationHeal → the (now fixed) delete path.

405 remains "marker already replicated" for creations only — same semantics #162 preserved; what disappears is the unbounded repetition driven by states that could never advance.

Tests

• TestReplicateDeleteMarkerTargetSemantics/marker-path_purge_* (new): purge in the legacy shape with PrevReplicationStatus=COMPLETED, VersionPurgeStatus=PENDING must not short-circuit; success yields VersionPurgeComplete with the creation status preserved; rejection yields VersionPurgeFailed.
• TestReplicateDeleteMarkerPurge/recover_legacy_true (reworked): a legacy-shaped purge now converges on the first proper delivery — marker version dropped on source and target, nothing further scheduled — instead of documenting the pre-fix PENDING state and recovering via a heal re-queue.
• Full TestReplication|TestReplicate|TestMRF|TestResync|TestSiteResync surface (23 tests) plus delete-handler tests pass; go build ./..., go vet, gofmt clean.

Docs

docs/site-replication/delete-marker-convergence.md: port of the convergence write-up with silo line references, a status note on what #162 covered, the applied fix, reproduction (3 sites) and known remaining limitations (wire tombstones, non-attempted targets, replica markers, per-code 405 metrics, MRF backoff re-enqueue — follow-ups from #152/#153).

Fixes the convergence loop reported in #153 (retry/purge-status side; the observability suggestions from #152/#153 are listed as follow-ups in the doc).

Companion write-up

Blog post covering the investigation and both fixes (#162, #184), to be published at: https://pepitedata.com/minio-silo-delete-marker-convergence-fix/