Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 7 additions & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -151,7 +151,8 @@ Keep the format `major.minor.micro`: the Makefile splits it on the dots and `mak
- A **read-only** participant (`Role=source`, `Participant::readOnly`) is the reverse: never written to. It can win like any side, but is never sent a version, so its recorded generation stays the one it really holds; a change of its own that lost is recorded (tag only) so it is not raised again. Being behind is its normal state, and a change it makes from an older generation than the 3DS's is a conflict with the 3DS, by the generation rule below. `syncEntry()` leaves it out of an upload-only entry, and `runSync()` does not offer it `legacyUpload()`.
- **Generations, not timestamps, say who is behind.** The manifest keeps the 3DS's own copy under `local|<path>` (mtime, size, `generation`), and each remote's record holds the generation it last received. The generation goes up whenever the 3DS's copy changes. A remote whose tag moved is *changed*; one whose generation is older is *behind* and is sent the current version — unless it is both, which is a conflict with the 3DS, because its edit started from a version the 3DS has since replaced. Without generations a remote that was offline for a run looks like it made the change it missed.
- **Manifests from before the n-way engine** have no `local|` entry and no `"gen"`. The newest remote record becomes the 3DS baseline, and a record without a generation counts as holding the baseline's when its recorded mtime/size match it (`sameStat()`), which makes the first run after an upgrade decide what the pairwise engine would have. `"gen"` is only written when non-zero, so such a manifest round-trips byte for byte, and an older 3DSync ignores both additions.
- Change detection compares the local mtime **and** size against the `local|` record, plus each remote's `tag`. `SyncProvider::localTag()` is the content-based fallback for stale mtimes and is only meaningful where the remote's tag is derivable from file contents (Drive's MD5, Dropbox's `content_hash`); it returns "" elsewhere. It also settles a remote that changed to the very bytes the 3DS has (no conflict, nothing copied) — which is what lets a remote added with copies already on it avoid a conflict per file. Size+mtime remotes cannot prove that and still ask.
- Change detection compares the local mtime **and** size against the `local|` record, plus each remote's `tag`. `SyncProvider::localTag()` is the content-based fallback for stale mtimes and is only meaningful where the remote's tag is derivable from file contents (Drive's MD5, Dropbox's `content_hash`); it returns "" elsewhere. It also settles a remote that changed to the very bytes the 3DS has (no conflict, nothing copied) — which is what lets a remote added with copies already on it avoid a conflict per file. Size+mtime remotes cannot prove that from the listing; for a *conflict* the engine then compares the bytes (see the next point).
- **A conflict is checked against the bytes before anyone is asked.** With two or more sides changed and the 3DS holding a copy, `sameAsLocalCopy()` in `syncengine.cpp` takes each changed remote without a content hash: a known `RemoteFileInfo::size` that differs from the 3DS's settles it; otherwise, within `SyncEntry::compareUpTo` (`[Sync] CompareUpTo=` in KB, default 4096, 0 = no limit (`COMPARE_UP_TO_UNLIMITED`), parsed by `parseCompareUpTo()` in `remoteconfig.cpp`; a `SyncEntry`'s own default of 0 never compares, which keeps the other engine tests free of comparison downloads; `[Sync] CompareConflicts=false` (`parseCompareConflicts()`) sets that 0 whatever `CompareUpTo=` says), it `download()`s the remote copy to `<save>.compare<side>.3dstmp` and `compareFiles()` (`utils/fsutil`) it. A match is `sameAsLocal` — out of the prompt, recorded as in step, nothing copied. A copy that differs is kept as `Side::fetched`: if that side wins the prompt, `downloadFrom()` moves it into place with `replaceLocalFile()` instead of downloading it again, and `FetchedCopies` removes every unused one however `syncFile()` returns. A failed or over-size check leaves the conflict as it was. FTP (`SIZE`), SMB and WebDAV (`getcontentlength`) fill `size`; Drive and Dropbox never need it. `recurse_dir()` skips names ending in `.3dstmp`, so a comparison or download cut short is never synced as a save.
- **Path names keep their spelling on the remote** (#71). INIReader lowercases every key, so `configuredPaths()` gives a `SyncEntry` both: `remoteName`, the folder as written (`INIReader::GetSpelling()`), which the providers, the log and the picker get; and `name`, lowercased, which is what `Only=` (`takesPart()`), `[Filter]` (`forEntry()`) and Drive's `Folder.<name>=` (`_pathFolders`, looked up through `asciiLower()`) compare. Versions before this one sent the lowercased name, so a case-sensitive server may hold `gba` where the INI says `GBA`: for a name with capitals, `resolveRoot()` in `syncengine.cpp` asks `SyncProvider::findRoot()` — which never creates anything — for the name as written, then lowercased, and only calls `ensureRoot()` (as written) when neither exists. FTP (a name-only listing, refused = missing), WebDAV (Depth 0 `PROPFIND`, 404 expected) and SMB (`smb2_stat()`) implement it; Dropbox and Drive compare names ignoring case and keep the default, which finds nothing. A lowercase name costs no extra request.
- **The extension filter** (`[Filter]`, `ExtensionFilter` and `PathFilters` in `sync/extfilter.h`) is per path and defaults to all files. `Extensions=` is the list for every path; a key naming a path (`roms/nds=dsv`, lowercased like every INIReader key) replaces it for that one, `*` included. `parsePathFilters()` collects keys naming no configured path in `unknown`, which `configIssues()` turns into a startup issue; a path named `extensions` cannot have its own line. It is applied twice on purpose: `getConfiguredSyncPaths()` drops non-matching local files, because a provider's `legacyUpload()` only ever sees `localFiles`; and `syncEntry()` filters both `localFiles` and every remote listing, so a remote-only file it excludes is never downloaded. Filtered files are skipped *before* the decision table, so their manifest entries are neither updated nor removed — widening the filter again must not look like a first sync and raise a conflict for every file. The menu card shows `PathFilters::describe()` under FILES (it is in `signature()`): the default's list, or "per path" once any path has its own. A run prints an entry's filter under that entry's header, when it has one. List separators are commas and whitespace, never `;` (inih's inline comment); the configurator refuses a `;` for that reason.
- Manifest keys are `Participant::keyPrefix + "|" + localPath`, the prefix being `manifestScope()`: `SyncProvider::manifestPrefix()` for an unnamed section, plus `#<label>` for a labelled one. Changing a provider's prefix orphans every existing entry for that remote, which makes the next sync treat every file as unseen — so keep them stable.
Expand Down Expand Up @@ -344,6 +345,10 @@ RemoteName=/LocalPath
[Filter] ; optional
Extensions=sav, bin ; every path; commas/spaces; empty, missing or * = all files
RemoteName=dsv ; this path instead of Extensions=; * = all files

[Sync] ; optional
CompareUpTo=4096 ; KB: largest file downloaded to check a conflict; 0 = no limit
CompareConflicts=true ; false: never download to check, always ask
```

---
Expand All @@ -363,6 +368,7 @@ RemoteName=dsv ; this path instead of Extensions=; * = all files
- **Test connection** only exists where a browser can reach the service: Dropbox and Drive (token refresh + "who am I", and the Drive folder ID), and WebDAV (`PROPFIND` from the page). SMB and FTP are not HTTP, so their settings point at a debug sync on the 3DS instead — do not add a button that cannot work. A WebDAV test that fails at the network level is reported as "could not check" rather than as wrong settings: browsers give the same error for an unreachable host and for a server that refuses cross-origin requests, and most WebDAV servers do the latter.
- Every typed value goes through `iniValueProblem()`: inih ends a value at a `;` that follows whitespace and trims both ends, so such a value is refused with a reason rather than silently cut on the 3DS. Optional keys are written only when filled — an empty `User=` overrides the FTP default of `anonymous`, which the app applies only when the key is missing. Pasted forms (`\\nas\share\dir`, `ftps://user@host:port/dir`, a URL without a scheme) are split into the fields on blur.
- A step's title reopens it only when it lies before the open step (`markReachableSteps()` in `index.js`, kept current by a `MutationObserver` on the steps' `active` class). MStepper's own `stepTitleNavigation` stays off: it would also jump forward, past the validation in Continue and Confirm, and step 3's preview is only built by Confirm.
- **Conflicts** in step 2 (`#conflicts-section`, shown only when an SMB, FTP or WebDAV remote is configured, since Drive and Dropbox compare by hash): `#compare-conflicts` is `CompareConflicts=` and `#compare-up-to` is `CompareUpTo=` in KB, blank for the 3DS's default of 4096, `0` for no limit, refused by Confirm unless a whole number. Both live in `config.sync` (`{ compare, compareUpTo, other }`, `other` being the `[Sync]` lines the page has no field for); `toIni()` writes `[Sync]` only when it says something, and `fromIni()` reads it back the way the 3DS does (an unknown `CompareConflicts=` is on, and reported).
- WebDAV and FTP have a "the server uses its own certificate" checkbox (`webdav-insecure`, `ftp-insecure`) that writes `VerifyTLS=false`; `inifile.js` maps the key back to it on import (`CHECKBOX_VALUES`), and keeps a value the box cannot show as it was.
- **Anonymous FTP needs a base directory.** With User blank, `anonymous` or `ftp`, an empty (or `/`) Path is refused: stock anonymous servers only accept uploads in one folder such as `pub`, and without `Path=` the first sync on the 3DS fails when it cannot create its folder there. A named user may leave it blank.
- A field's error replaces its hint (`index.css`, `.remote-field.invalid ~ .helper-text`); Materialize would otherwise draw the error over the visible white hint.
Expand Down
27 changes: 25 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -432,8 +432,8 @@ If more than one side changed the file, one prompt lists them all.

"Changed" means the local mtime/size, or the remote tag, differs from the last manifest entry.
On the first sync of a file (no manifest entry yet), a file on one side only is copied to the
others. A file already on both is a conflict, unless Google Drive or Dropbox can prove it holds
the same bytes as the 3DS.
others. A file already on both is a conflict, unless the remote holds the same bytes as the 3DS
(see [Conflicts that are the same file](#conflicts-that-are-the-same-file)).

A remote file whose name could reach outside the synced folder (one with a `..`
segment, an empty segment or a backslash) is never downloaded or compared. The
Expand All @@ -445,6 +445,29 @@ save whose mtime never moves (some emulators never update it) is still detected,
and a remote that already holds the same bytes as the 3DS is not a conflict.
On SMB, FTP and WebDAV an unchanged mtime *and* size is taken as unchanged.

### Conflicts that are the same file

A conflict is often no conflict at all: the same save reached both sides
separately, say a remote added with the saves already on it, or a file copied
back by hand. Google Drive and Dropbox prove that with their content hash.
SMB, FTP and WebDAV cannot, so before asking, 3DSync downloads the remote copy
and compares it with the 3DS's byte for byte. If they match, nothing is asked
and nothing is copied. A different size settles it without a download. If they
differ and you keep the remote's version, the copy already fetched is used, so
the file is not downloaded twice.

Comparing costs one download per conflicting file, so it is limited by size:

```ini
[Sync]
CompareUpTo=4096 ; KB, the default (4 MB); 0 = no limit
CompareConflicts=false ; never compare: every such conflict is asked about
```

A larger file is asked about as before. Saves are almost always far below 4 MB.
With `0` every conflicting file is compared, however large: on a slow server a
ROM can take minutes, though START stops it.

### Subfolder structure on the remote

Bidirectional paths preserve the relative directory structure.
Expand Down
23 changes: 23 additions & 0 deletions docs/configurator.html
Original file line number Diff line number Diff line change
Expand Up @@ -547,6 +547,29 @@ <h4 class="white-text">File Types</h4>
the one above; <strong>*</strong> syncs every file in that folder.</p>
<div id="folder-filters" class="folder-filters"></div>
</div>
<div id="conflicts-section" class="conflicts-section">
<h4 class="white-text">Conflicts</h4>
<p class="white-text panel-intro">When a save changed both on the 3DS and on an SMB, FTP or WebDAV
remote, 3DSync first downloads the remote copy to see whether it is the same file. It only asks
you when the two really differ.</p>
<p class="compare-option">
<label>
<input type="checkbox" class="filled-in" id="compare-conflicts" checked>
<span class="white-text" style="font-size:0.88em">Check whether both copies are the same
before asking</span>
</label>
</p>
<div class="row">
<div class="input-field col s12">
<input id="compare-up-to" class="white-text" type="text" inputmode="numeric"
autocomplete="off" spellcheck="false">
<label for="compare-up-to" class="white-text">Largest file to check, in KB (optional)</label>
<span class="helper-text white-text" data-error="">Blank: <strong>4096</strong> (4 MB),
enough for any save. <strong>0</strong>: any size, which can take minutes for a ROM on a
slow server.</span>
</div>
</div>
</div>
<br>
<div class="step-actions">
<a href="#" class="waves-effect waves-dark btn-flat previous-step white-text">Back</a>
Expand Down
9 changes: 9 additions & 0 deletions docs/static/css/index.css
Original file line number Diff line number Diff line change
Expand Up @@ -603,6 +603,15 @@ main {
margin: 0.2rem 0 0.8rem 0;
}

/* Step 2: [Sync] CompareConflicts= and CompareUpTo=, for SMB, FTP and WebDAV. */
.conflicts-section {
display: none;
}

.compare-option {
margin: 0.2rem 0 0.4rem 0;
}

/* Step 2: the Drive folder chosen per folder, for 3DSync's own client */
.drive-folders-section {
display: none;
Expand Down
25 changes: 24 additions & 1 deletion docs/static/js/index.js
Original file line number Diff line number Diff line change
Expand Up @@ -122,7 +122,8 @@ $(function () {
// What the steps build, and what the file is written from (IniFile in
// inifile.js describes it): config.remotes as they are saved in step 1,
// folders and file types as step 2 confirms them.
let config = { remotes: [], folders: [], filter: '', extras: [] };
let config = { remotes: [], folders: [], filter: '', sync: { compare: true, compareUpTo: '', other: [] },
extras: [] };
let draft = null;
let draftSnapshot = '';
let nextUid = 1;
Expand Down Expand Up @@ -436,6 +437,9 @@ $(function () {
if (f.filter) folderFilters[f.name.toLowerCase()] = f.filter;
});
$('#file-filter').val(config.filter);
$('#compare-conflicts').prop('checked', config.sync.compare !== false);
$('#compare-up-to').val(config.sync.compareUpTo);
updateCompareUpTo();
M.updateTextFields();
renderRemotes();
renderRouting();
Expand Down Expand Up @@ -1301,6 +1305,12 @@ $(function () {
});
}

// The size only matters while the check is on.
function updateCompareUpTo() {
$('#compare-up-to').prop('disabled', !$('#compare-conflicts').prop('checked'));
}
$('#compare-conflicts').on('change', updateCompareUpTo);

$('#file-filter').on('input', function () {
$('#folder-filters input.folder-filter').attr('placeholder', $(this).val().trim() || 'all files');
});
Expand Down Expand Up @@ -1331,6 +1341,10 @@ $(function () {
function renderRouting() {
let names = currentFolderNames();
$('#routing-section').toggle(config.remotes.length > 1);
// Drive and Dropbox prove a copy is the same with a content hash.
$('#conflicts-section').toggle(config.remotes.some(function (r) {
return r.type === 'smb' || r.type === 'ftp' || r.type === 'webdav';
}));
let $table = $('<table></table>');
let $head = $('<tr></tr>').append($('<th>Folder</th>'));
config.remotes.forEach(function (remote) {
Expand Down Expand Up @@ -1718,9 +1732,18 @@ $(function () {
if (filterProblem) error = true;
renderRouting();
if (folderFiltersProblem()) error = true;
// The 3DS takes a whole number of KB and nothing else.
let $compareUpTo = $('#compare-up-to');
$compareUpTo.val($compareUpTo.val().trim());
let compareProblem = /^\d*$/.test($compareUpTo.val()) ? null
: 'A whole number of KB, e.g. 4096; 0 for any size';
setFieldError($compareUpTo, compareProblem);
if (compareProblem) error = true;
folders.forEach(function (f) { f.filter = (folderFilters[f.name.toLowerCase()] || '').trim(); });
config.folders = folders;
config.filter = $filter.val();
config.sync = { compare: $('#compare-conflicts').prop('checked'), compareUpTo: $compareUpTo.val(),
other: (config.sync && config.sync.other) || [] };
let routing = error ? null : routingProblem();
$('#routing-note').text(routing || '').toggleClass('warn', !!routing);
if (routing) {
Expand Down
Loading
Loading