Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -135,6 +135,7 @@ Keep the format `major.minor.micro`: the Makefile splits it on the dots and `mak
- All HTTP goes through the `Curl` wrapper (`source/utils/curl.h`). Never call libcurl directly.
- **Certificates are checked on every TLS connection.** `Curl::_applyDefaults()` sets `VERIFYPEER`/`VERIFYHOST` and hands mbedTLS the roots from `romfs/cacert.pem` through `CURLOPT_SSL_CTX_FUNCTION` (`_sslContext()`), parsed once per run (`caChain()`): libcurl would otherwise reparse the file per connection, and FTPS opens one per transfer. It also clears `CAINFO`/`CAPATH`, because devkitPro's libcurl defaults to `sdmc:/config/ssl/cacert.pem`, which most cards lack. The callback runs after libcurl's own `mbedtls_ssl_conf_ca_chain()` (checked with `objdump` on libcurl 8.4: the indirect call follows it in `mbed_connect_common`), so its chain wins; recheck after a libcurl bump. Only WebDAV and FTP may turn checking off (`VerifyTLS=false` → `Curl::setVerifyTls(false)`, parsed in `remoteconfig.cpp`). A certificate failure (curl 60, or 77 for a missing bundle) is fatal for that remote, never retried, and explained by `Curl::printCertificateHelp()` — including the 3DS date, since an unset clock is the likeliest cause. Update the bundle with curl's published file and checksum (`https://curl.se/ca/cacert.pem` and `.sha256`).
- **Remote names never leave the sync folder.** `syncEntry()` drops every listed relPath that `isSafeRelativePath()` (`utils/pathutil`) refuses — a `..`, `.` or empty segment, a backslash, a control character — before anything is decided, says so once per remote and entry, and adds a `problems` entry. Providers need not check this themselves, but must not "fix" such a name into something that passes.
- **A transfer that stops receiving gives up after 60 s** (`CURLOPT_LOW_SPEED_LIMIT`/`_TIME` in `Curl::_applyDefaults()`, `STALL_SECONDS`). Wi-Fi dropping mid-file sends no error — the connection just goes quiet — and libcurl's only other limit is the 50 s connect timeout, so a ROM download once waited until START. The timeout is `CURLE_OPERATION_TIMEDOUT`, which every provider retries; a retry must discard what a stalled download already wrote (`rewindDownloadFile()` — FTP did not, and appended to the half file) and FTP opens a fresh connection for it.
- `Curl::perform()` returns 0 on network success. **HTTP status is not checked by `perform()`** — callers must call `getStatusCode()`. `CURLOPT_FAILONERROR` is intentionally disabled so error response bodies are captured.
- Every provider owns a `_performWithRetry()` that maps transport errors to *transient* (retry with back-off), *fatal* (`_fatalError`, stop the run) or *per-file* (log and skip). Never call `_curl.perform()` directly from a provider method — the retry helper is also where the server `Date` header is captured.
- libcurl options are sticky on a handle. Providers that issue more than one shape of request call `Curl::reset()` first (via their own `_prepare()`); otherwise `NOBODY`, `UPLOAD`, `WILDCARDMATCH` or a custom method leaks into the next call.
Expand Down Expand Up @@ -169,7 +170,7 @@ Keep the format `major.minor.micro`: the Makefile splits it on the dots and `mak
- A body streamed to a temp file is not in `getResponse()` — a failed download writes the API error into the temp file, so call `debugFileBody()` before removing it.
- `debugRedact()` masks bearer headers and the OAuth token/secret fields, and every dump goes through it. Debug output is what ends up in a bug report, and once opened the credentials are plaintext in memory: when adding a dump of anything new that could carry a secret, extend `debugRedact()` and its tests rather than filtering at the call site.
- `debug.cpp` is free of `<3ds.h>`, libcurl and libsmb2, so the sync engine and the host tests can both log.
- **The session log.** While debug mode is on, every byte written to stdout is also kept in `debug.cpp` (`debugLogAppend()`), and the end of `runSync()` offers to write it to `/3ds/3DSync/logs/debug-<stamp>.log`. The capture is fed by the stdout devoptab `frontend::init()` installs (`logWrite()` in `ui/frontend.cpp`), the same one that feeds the on-screen log — so a plain `printf` lands in both with no extra call, and nothing has to be routed through `debugf()` to be reportable. Nothing may call `consoleInit()` any more: it would put libctru's console devoptab back over it. Only the first 96 KB and the last 48 KB are kept, and what fell out in between is counted and stated in the file. `debugLogSave()` strips ANSI sequences and runs `debugRedact()` over the whole text — a token can straddle two console writes, so redacting per write would miss it.
- **The session log.** Every run, debug mode or not, writes every byte of stdout to `/3ds/3DSync/logs/debug-<stamp>.log.part` (`sync-<stamp>` for a normal run) on the SD card as it goes (`debugLogStart()` before the run, `debugLogAppend()` per write), and the end of every run (`offerLogSave()`) asks `frontend::askSaveLog()` whether to keep it: `debugLogKeep()` renames it to `.log`, `debugLogReset()` deletes it. Nothing is trimmed and nothing is held in memory — the old in-memory head and tail lost the middle of exactly the long runs a report needed — and a crash leaves the `.part` up to that point. The capture is fed by the stdout devoptab `frontend::init()` installs (`logWrite()` in `ui/frontend.cpp`), the same one that feeds the on-screen log — so a plain `printf` lands in both with no extra call, and nothing has to be routed through `debugf()` to be reportable. Nothing may call `consoleInit()` any more: it would put libctru's console devoptab back over it. Each *complete line* is stripped of ANSI sequences and passed through `debugRedact()` before it reaches the card, so the file never holds a token, even mid-run: a token can straddle two console writes, which is why it is redacted per line, never per write (a line without a break is written once it passes 64 KB). `debugLogAppend()` must never print — stdout is what feeds it.
- **`Curl` closes every socket through `_closeGracefully()`** (`CURLOPT_CLOSESOCKETFUNCTION`, set in `_applyDefaults()` so `reset()` keeps it): `shutdown(SHUT_WR)`, drain until the peer's FIN or 1 s, then `closesocket()`. A plain close on the 3DS ends the connection with a RST, and an FTP upload in stream mode has no end of file but the client's close — vsftpd answered every upload with "426 Failure reading network stream", sometimes with the tail of the file lost. Do not remove it as a no-op: HTTP does not need it, FTP does. libcurl does not call it for sockets it *accepted*, i.e. the data connection of `Mode=active`. A TLS upload also needs a `close_notify` before that FIN, which libcurl 8.4's mbedTLS backend never sends and vsftpd 3.0.5 demands (`strict_ssl_read_eof`, 1 in its `tunables.c` despite the man page) — see `utils/tlsclose.cpp` below.
- **`utils/tlsclose.cpp` hooks libcurl's TLS teardown at link time.** `BUILD_FLAGS_CXX` in the Makefile passes `-Wl,--wrap=mbedtls_ssl_setup -Wl,--wrap=mbedtls_ssl_config_free` (make_base hands `CXX_FLAGS` to the link too; it has no other way to add a linker option). The setup wrapper records which `mbedtls_ssl_context` belongs to which config; the config-free wrapper, which libcurl's `mbedtls_close()` reaches while the context and its socket are still open, sends `mbedtls_ssl_close_notify()` first. It depends on libcurl 8.4's close order and mbedTLS 2.28's public `ssl->state`: after a devkitPro libcurl or mbedTLS bump, check `objdump -d` that `mbedtls_close` still calls `__wrap_mbedtls_ssl_config_free`, and drop the file once libcurl sends close_notify itself (curl ≥ 8.10). Not host-testable; the host tests do not link it.
- **A request body read from a file must be seekable.** `setUserPassword()` defaults to `CURLAUTH_ANY`, so libcurl sends an authenticated request once without credentials, gets the 401, and sends it again — rewinding the body through `_seek_callback()`, which `setUploadFile()` and `setReadData()` install. Without it every WebDAV `PUT` failed with curl 65 ("rewinding of the data stream failed") until a retry got through. `WebDavRemote` keeps that probe to the first request of a run: `_learnAuthMethods()` reads `getAuthOffered()` (`CURLINFO_HTTPAUTH_AVAIL`, which still holds the 401's methods after the retry succeeded) and, when Basic is the *only* one offered, passes `CURLAUTH_BASIC` from then on, which libcurl sends unasked. A server that also offers Digest keeps the challenge on every request — choosing Basic there would put the password in the clear unasked.
Expand All @@ -192,7 +193,7 @@ They build for the **host**, not the 3DS, and run in the `Unit Tests` job of `pr
- `ui/canvas`, `ui/logbuffer`, `ui/screens` and `ui/touchinput` are host units too: `test_ui.cpp` covers the scrollback (colours, split escapes, wrap, trimming, the view holding still while scrolled back), the tap/repeat/drag rules, and that every screen's buttons are on screen, apart, and wide enough for their label. `make -C tests preview` renders every screen to `tests/build/preview/*.png` — look at those after any layout change, there is no other way to see one without a console.
- Response parsing lives in `modules/remoteparse.cpp`, so a provider's `list()` is a request plus a call into a testable function. New protocol parsing belongs there, not inline in the provider.
- `utils/console.h` supplies the `CONSOLE_*` macros off-console, so host-portable code must include it instead of `<3ds.h>`.
- `utils/debug.cpp` is in the host `UNITS` because the engine, the manifest and `fsutil` all log through it. `test_debug.cpp` covers `debugRedact()`, which is the one part of debug output that has to be right: everything else is only printed, but a redaction that misses puts a token on screen. The `debuglog` suite covers the same risk one step further out — the capture must stay empty outside debug mode, stay bounded on a long run, and reach the SD card with the colour codes gone and the tokens masked.
- `utils/debug.cpp` is in the host `UNITS` because the engine, the manifest and `fsutil` all log through it. `test_debug.cpp` covers `debugRedact()`, which is the one part of debug output that has to be right: everything else is only printed, but a redaction that misses puts a token on screen. The `debuglog` suite covers the same risk one step further out — the capture must stay empty outside a run, keep every line of a long run, reach the SD card with the colour codes gone and the tokens masked (also a token split over two writes), and be deleted when declined.
- The hash tests need mbedtls on the host (`libmbedtls-dev`). Without it the suite still builds and runs, and the runner prints what was left out — deliberately loud rather than a silent pass.

The framework is `tests/framework.h`, about eighty lines: the build image has no gtest and vendoring one for this is more dependency than it is worth. `TEST(suite, name)` registers a case; `CHECK`/`CHECK_EQ`/`CHECK_STR_EQ` record a failure and keep going, so one broken case reports every mismatch rather than only the first.
Expand Down
18 changes: 11 additions & 7 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ label says which, so either works:
| Choosing paths | Tap a path to tick or untick it, or move with **D-pad ↑/↓** and press **A**. **Y** ticks all (or clears all), **B** goes back. The choice lasts until 3DSync is closed; a remote that none of the ticked paths go to is not connected |
| During a sync | **START** stops after the current file |
| Conflict | The versions that changed are listed; **A** keeps the highlighted one (the 3DS's at first). Tap a version, or move to it with **R** or **D-pad ↑/↓**. **X** skips the file, **L** applies the answer to every remaining conflict, **START** stops. The log still scrolls with ←/→, the Circle Pad and the scroll strip |
| End of a debug run | **A** save the log, **B** discard it |
| End of a run | **A** save its log, **B** discard it |
| Problems in 3DSync.ini, at startup | **A** continue to the menu, **START** exit; the list scrolls like the log |

The log can be scrolled at any time, including while a sync is running:
Expand Down Expand Up @@ -514,12 +514,16 @@ The rest is not filtered — a debug run prints one line per file examined. The
log on the top screen can be scrolled back while the run is going (see
[Controls](#controls)), but for a bug report a file beats a photo.

So a debug run does not leave it at the screen: when the sync ends, 3DSync
offers to write everything it printed to the SD card as
`/3ds/3DSync/logs/debug-<date>-<time>.log`. Press **A** to save it or **B** to
discard it. That file is the whole run, colour codes removed and secrets masked
— attach it to the issue instead of photographing the screen. A very long run
keeps its beginning and its end and notes in the middle how much was dropped.
So a run does not leave it at the screen: when the sync ends, 3DSync offers to
write everything it printed to the SD card, as
`/3ds/3DSync/logs/debug-<date>-<time>.log` after a debug run and
`sync-<date>-<time>.log` after a normal one. Press **A** to save it or **B** to
discard it. That file is the whole run, however long, colour codes removed and
secrets masked — attach it to the issue instead of photographing the screen.
It is written as the run goes (as `…log.part`, deleted if you discard it), so
if 3DSync crashes, the log up to that point is still on the card. A normal
run's log holds only what the screen showed; for a report, a debug run says far
more.

The log holds the same text the screen did, which includes the local paths you
sync. Worth a look before sharing it, and it stays on your SD card until you
Expand Down
59 changes: 35 additions & 24 deletions source/main.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -183,15 +183,19 @@ static void findLocalFiles(ConfiguredPath &path)
}

// ---------------------------------------------------------------------------
// offerDebugLogSave — end of a debug run: keep the transcript or drop it
// offerLogSave — end of a run: keep the transcript or drop it
// ---------------------------------------------------------------------------
// The whole point of debug mode is to produce something that can be attached
// to a bug report, and a screen that has already scrolled is not that. The
// save is offered rather than done, because the transcript is a plaintext file
// on a shared SD card and the run it came from listed every save path on the
// console — asking is what makes writing it the user's decision.
// A screen that has already scrolled is no bug report, and a run that went
// wrong without debug mode is worth keeping too. Every run is written to
// "<name>.part" as it goes (debugLogStart(), redacted line by line), so
// nothing is lost however long it runs, and a crash still leaves the log up
// to that point. Keeping it is offered rather than done, because the
// transcript sits on a shared SD card and lists every save path on the
// console: declined, the file is deleted.
// ---------------------------------------------------------------------------
static std::string debugLogPath()
static std::string g_logPath; // where a kept log goes, named when the run starts

static std::string logPath(bool debug)
{
time_t now = time(NULL);
struct tm *lt = (now > 0) ? localtime(&now) : NULL;
Expand All @@ -202,34 +206,37 @@ static std::string debugLogPath()
if (lt == NULL || strftime(stamp, sizeof(stamp), "%Y%m%d-%H%M%S", lt) == 0)
snprintf(stamp, sizeof(stamp), "unknown-time");

return std::string("/3ds/3DSync/logs/debug-") + stamp + ".log";
return std::string("/3ds/3DSync/logs/") + (debug ? "debug-" : "sync-") + stamp + ".log";
}

static void offerDebugLogSave()
static void offerLogSave()
{
if (!debugEnabled() || debugLogSize() == 0)
if (debugLogSize() == 0)
{
debugLogReset();
return;
}

// The question itself is on the bottom screen; the log keeps a line so
// the saved file and the screen agree about what was asked.
unsigned kb = (unsigned)((debugLogSize() + 1023) / 1024);
printf("\n--- Debug log ---\n");
printf("%u KB captured%s.\n", kb,
debugLogDropped() > 0 ? ", middle of the run trimmed" : "");
printf("\n--- %s ---\n", debugEnabled() ? "Debug log" : "Log");
printf("%u KB captured.\n", kb);
if (debugLogFailed())
printf(CONSOLE_RED "Writing it to the SD card failed part\n"
"way; it ends early.\n" CONSOLE_RESET);

if (!frontend::askSaveDebugLog(kb, debugLogDropped() > 0))
if (!frontend::askSaveLog(kb))
{
printf("Debug log discarded.\n");
debugLogReset();
printf("Log discarded.\n");
return;
}
std::string path = debugLogPath();
// The write itself prints, and the stdout hook would fold that into the
// very text being written; it is already snapshotted by then, so the
// file just ends a line or two before the screen does.
if (debugLogSave(path))
printf(CONSOLE_GREEN "Saved to\n %s\n" CONSOLE_RESET, path.c_str());
// What is printed from here on is no longer captured.
if (debugLogKeep(g_logPath))
printf(CONSOLE_GREEN "Saved to\n %s\n" CONSOLE_RESET, g_logPath.c_str());
else
printf(CONSOLE_RED "Could not write\n %s\n" CONSOLE_RESET, path.c_str());
printf(CONSOLE_RED "Could not write\n %s\n" CONSOLE_RESET, g_logPath.c_str());
}

// ---------------------------------------------------------------------------
Expand Down Expand Up @@ -778,7 +785,7 @@ static void runSync(const INIReader &reader, const std::vector<bool> &selected)
frontend::setLastResult(summaryResult(summary), true);
}

offerDebugLogSave();
offerLogSave();
}

// ---------------------------------------------------------------------------
Expand Down Expand Up @@ -919,7 +926,7 @@ static void syncWorker(void *arg)
printf("Can't load configuration\n");
debugConfigSummary(reader);
frontend::setLastResult("configuration not loaded", false);
offerDebugLogSave();
offerLogSave();
}
}

Expand Down Expand Up @@ -987,6 +994,10 @@ int main(int argc, char **argv)
// Per run, not per session: syncing twice should not offer the first
// run's transcript along with the second's.
debugLogReset();
g_logPath = logPath(debugEnabled());
if (!debugLogStart(g_logPath + ".part"))
printf(CONSOLE_RED "The log cannot be written to the SD\n"
"card; this run keeps none.\n" CONSOLE_RESET);

SyncRequest request = {&reader, &selected};
frontend::runWorker(syncWorker, &request);
Expand Down
5 changes: 5 additions & 0 deletions source/modules/ftpremote.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -213,6 +213,11 @@ int FtpRemote::_performWithRetry(FtpRequest request, FILE *uploadFile)
debugf(" curl %d on %s\n", res, _curl.getURL().c_str());
if (attempt < 2)
{
// A download that stalled part way has written what arrived;
// the retry starts the file again, on a new connection: the
// old one may be the one that went quiet. reset() clears it.
_curl.rewindDownloadFile();
_curl.setFreshConnect(true);
svcSleepThread(2000000000LL);
continue;
}
Expand Down
Loading
Loading