From 17b13e120301cb841af69f97ee737348724b4a0c Mon Sep 17 00:00:00 2001 From: michaelvillani Date: Sun, 8 Nov 2026 21:07:29 +0000 Subject: [PATCH] fix: a stalled transfer gives up; every run's log can be kept whole - A download from a phone's FTP server waited until START: Wi-Fi dropped mid-file, which sends no error, and libcurl had only a connect timeout. A transfer now gives up after 60 s without a byte (LOW_SPEED_LIMIT/TIME in Curl::_applyDefaults()); every provider already retries a timeout. - FTP's retry did not discard what a stalled download had written and appended the next attempt to the half file; it now rewinds the temp file and retries on a fresh connection. - The debug log kept the first 96 KB and the last 48 KB, so a long run lost the part a report needed. Every run is now written to /3ds/3DSync/logs/{debug,sync}-.log.part as it goes, each line stripped of colour codes and redacted before it reaches the card, and renamed to .log when kept or deleted when not. Nothing is held in memory, and a crash leaves the log up to that point. - Normal runs offer to keep their log too, with the same question as a debug run. Co-Authored-By: Claude Opus 5.5 --- AGENTS.md | 5 +- README.md | 18 ++-- source/main.cpp | 59 +++++++------ source/modules/ftpremote.cpp | 5 ++ source/ui/frontend.cpp | 7 +- source/ui/frontend.h | 4 +- source/ui/screens.cpp | 11 +-- source/ui/screens.h | 3 +- source/utils/curl.cpp | 10 +++ source/utils/debug.cpp | 158 ++++++++++++++++++----------------- source/utils/debug.h | 34 ++++---- tests/preview.cpp | 3 + tests/test_debug.cpp | 99 ++++++++++++---------- 13 files changed, 237 insertions(+), 179 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index f0df07b..1c2531f 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -135,6 +135,7 @@ Keep the format `major.minor.micro`: the Makefile splits it on the dots and `mak - All HTTP goes through the `Curl` wrapper (`source/utils/curl.h`). Never call libcurl directly. - **Certificates are checked on every TLS connection.** `Curl::_applyDefaults()` sets `VERIFYPEER`/`VERIFYHOST` and hands mbedTLS the roots from `romfs/cacert.pem` through `CURLOPT_SSL_CTX_FUNCTION` (`_sslContext()`), parsed once per run (`caChain()`): libcurl would otherwise reparse the file per connection, and FTPS opens one per transfer. It also clears `CAINFO`/`CAPATH`, because devkitPro's libcurl defaults to `sdmc:/config/ssl/cacert.pem`, which most cards lack. The callback runs after libcurl's own `mbedtls_ssl_conf_ca_chain()` (checked with `objdump` on libcurl 8.4: the indirect call follows it in `mbed_connect_common`), so its chain wins; recheck after a libcurl bump. Only WebDAV and FTP may turn checking off (`VerifyTLS=false` → `Curl::setVerifyTls(false)`, parsed in `remoteconfig.cpp`). A certificate failure (curl 60, or 77 for a missing bundle) is fatal for that remote, never retried, and explained by `Curl::printCertificateHelp()` — including the 3DS date, since an unset clock is the likeliest cause. Update the bundle with curl's published file and checksum (`https://curl.se/ca/cacert.pem` and `.sha256`). - **Remote names never leave the sync folder.** `syncEntry()` drops every listed relPath that `isSafeRelativePath()` (`utils/pathutil`) refuses — a `..`, `.` or empty segment, a backslash, a control character — before anything is decided, says so once per remote and entry, and adds a `problems` entry. Providers need not check this themselves, but must not "fix" such a name into something that passes. +- **A transfer that stops receiving gives up after 60 s** (`CURLOPT_LOW_SPEED_LIMIT`/`_TIME` in `Curl::_applyDefaults()`, `STALL_SECONDS`). Wi-Fi dropping mid-file sends no error — the connection just goes quiet — and libcurl's only other limit is the 50 s connect timeout, so a ROM download once waited until START. The timeout is `CURLE_OPERATION_TIMEDOUT`, which every provider retries; a retry must discard what a stalled download already wrote (`rewindDownloadFile()` — FTP did not, and appended to the half file) and FTP opens a fresh connection for it. - `Curl::perform()` returns 0 on network success. **HTTP status is not checked by `perform()`** — callers must call `getStatusCode()`. `CURLOPT_FAILONERROR` is intentionally disabled so error response bodies are captured. - Every provider owns a `_performWithRetry()` that maps transport errors to *transient* (retry with back-off), *fatal* (`_fatalError`, stop the run) or *per-file* (log and skip). Never call `_curl.perform()` directly from a provider method — the retry helper is also where the server `Date` header is captured. - libcurl options are sticky on a handle. Providers that issue more than one shape of request call `Curl::reset()` first (via their own `_prepare()`); otherwise `NOBODY`, `UPLOAD`, `WILDCARDMATCH` or a custom method leaks into the next call. @@ -169,7 +170,7 @@ Keep the format `major.minor.micro`: the Makefile splits it on the dots and `mak - A body streamed to a temp file is not in `getResponse()` — a failed download writes the API error into the temp file, so call `debugFileBody()` before removing it. - `debugRedact()` masks bearer headers and the OAuth token/secret fields, and every dump goes through it. Debug output is what ends up in a bug report, and once opened the credentials are plaintext in memory: when adding a dump of anything new that could carry a secret, extend `debugRedact()` and its tests rather than filtering at the call site. - `debug.cpp` is free of `<3ds.h>`, libcurl and libsmb2, so the sync engine and the host tests can both log. - - **The session log.** While debug mode is on, every byte written to stdout is also kept in `debug.cpp` (`debugLogAppend()`), and the end of `runSync()` offers to write it to `/3ds/3DSync/logs/debug-.log`. The capture is fed by the stdout devoptab `frontend::init()` installs (`logWrite()` in `ui/frontend.cpp`), the same one that feeds the on-screen log — so a plain `printf` lands in both with no extra call, and nothing has to be routed through `debugf()` to be reportable. Nothing may call `consoleInit()` any more: it would put libctru's console devoptab back over it. Only the first 96 KB and the last 48 KB are kept, and what fell out in between is counted and stated in the file. `debugLogSave()` strips ANSI sequences and runs `debugRedact()` over the whole text — a token can straddle two console writes, so redacting per write would miss it. + - **The session log.** Every run, debug mode or not, writes every byte of stdout to `/3ds/3DSync/logs/debug-.log.part` (`sync-` for a normal run) on the SD card as it goes (`debugLogStart()` before the run, `debugLogAppend()` per write), and the end of every run (`offerLogSave()`) asks `frontend::askSaveLog()` whether to keep it: `debugLogKeep()` renames it to `.log`, `debugLogReset()` deletes it. Nothing is trimmed and nothing is held in memory — the old in-memory head and tail lost the middle of exactly the long runs a report needed — and a crash leaves the `.part` up to that point. The capture is fed by the stdout devoptab `frontend::init()` installs (`logWrite()` in `ui/frontend.cpp`), the same one that feeds the on-screen log — so a plain `printf` lands in both with no extra call, and nothing has to be routed through `debugf()` to be reportable. Nothing may call `consoleInit()` any more: it would put libctru's console devoptab back over it. Each *complete line* is stripped of ANSI sequences and passed through `debugRedact()` before it reaches the card, so the file never holds a token, even mid-run: a token can straddle two console writes, which is why it is redacted per line, never per write (a line without a break is written once it passes 64 KB). `debugLogAppend()` must never print — stdout is what feeds it. - **`Curl` closes every socket through `_closeGracefully()`** (`CURLOPT_CLOSESOCKETFUNCTION`, set in `_applyDefaults()` so `reset()` keeps it): `shutdown(SHUT_WR)`, drain until the peer's FIN or 1 s, then `closesocket()`. A plain close on the 3DS ends the connection with a RST, and an FTP upload in stream mode has no end of file but the client's close — vsftpd answered every upload with "426 Failure reading network stream", sometimes with the tail of the file lost. Do not remove it as a no-op: HTTP does not need it, FTP does. libcurl does not call it for sockets it *accepted*, i.e. the data connection of `Mode=active`. A TLS upload also needs a `close_notify` before that FIN, which libcurl 8.4's mbedTLS backend never sends and vsftpd 3.0.5 demands (`strict_ssl_read_eof`, 1 in its `tunables.c` despite the man page) — see `utils/tlsclose.cpp` below. - **`utils/tlsclose.cpp` hooks libcurl's TLS teardown at link time.** `BUILD_FLAGS_CXX` in the Makefile passes `-Wl,--wrap=mbedtls_ssl_setup -Wl,--wrap=mbedtls_ssl_config_free` (make_base hands `CXX_FLAGS` to the link too; it has no other way to add a linker option). The setup wrapper records which `mbedtls_ssl_context` belongs to which config; the config-free wrapper, which libcurl's `mbedtls_close()` reaches while the context and its socket are still open, sends `mbedtls_ssl_close_notify()` first. It depends on libcurl 8.4's close order and mbedTLS 2.28's public `ssl->state`: after a devkitPro libcurl or mbedTLS bump, check `objdump -d` that `mbedtls_close` still calls `__wrap_mbedtls_ssl_config_free`, and drop the file once libcurl sends close_notify itself (curl ≥ 8.10). Not host-testable; the host tests do not link it. - **A request body read from a file must be seekable.** `setUserPassword()` defaults to `CURLAUTH_ANY`, so libcurl sends an authenticated request once without credentials, gets the 401, and sends it again — rewinding the body through `_seek_callback()`, which `setUploadFile()` and `setReadData()` install. Without it every WebDAV `PUT` failed with curl 65 ("rewinding of the data stream failed") until a retry got through. `WebDavRemote` keeps that probe to the first request of a run: `_learnAuthMethods()` reads `getAuthOffered()` (`CURLINFO_HTTPAUTH_AVAIL`, which still holds the 401's methods after the retry succeeded) and, when Basic is the *only* one offered, passes `CURLAUTH_BASIC` from then on, which libcurl sends unasked. A server that also offers Digest keeps the challenge on every request — choosing Basic there would put the password in the clear unasked. @@ -192,7 +193,7 @@ They build for the **host**, not the 3DS, and run in the `Unit Tests` job of `pr - `ui/canvas`, `ui/logbuffer`, `ui/screens` and `ui/touchinput` are host units too: `test_ui.cpp` covers the scrollback (colours, split escapes, wrap, trimming, the view holding still while scrolled back), the tap/repeat/drag rules, and that every screen's buttons are on screen, apart, and wide enough for their label. `make -C tests preview` renders every screen to `tests/build/preview/*.png` — look at those after any layout change, there is no other way to see one without a console. - Response parsing lives in `modules/remoteparse.cpp`, so a provider's `list()` is a request plus a call into a testable function. New protocol parsing belongs there, not inline in the provider. - `utils/console.h` supplies the `CONSOLE_*` macros off-console, so host-portable code must include it instead of `<3ds.h>`. -- `utils/debug.cpp` is in the host `UNITS` because the engine, the manifest and `fsutil` all log through it. `test_debug.cpp` covers `debugRedact()`, which is the one part of debug output that has to be right: everything else is only printed, but a redaction that misses puts a token on screen. The `debuglog` suite covers the same risk one step further out — the capture must stay empty outside debug mode, stay bounded on a long run, and reach the SD card with the colour codes gone and the tokens masked. +- `utils/debug.cpp` is in the host `UNITS` because the engine, the manifest and `fsutil` all log through it. `test_debug.cpp` covers `debugRedact()`, which is the one part of debug output that has to be right: everything else is only printed, but a redaction that misses puts a token on screen. The `debuglog` suite covers the same risk one step further out — the capture must stay empty outside a run, keep every line of a long run, reach the SD card with the colour codes gone and the tokens masked (also a token split over two writes), and be deleted when declined. - The hash tests need mbedtls on the host (`libmbedtls-dev`). Without it the suite still builds and runs, and the runner prints what was left out — deliberately loud rather than a silent pass. The framework is `tests/framework.h`, about eighty lines: the build image has no gtest and vendoring one for this is more dependency than it is worth. `TEST(suite, name)` registers a case; `CHECK`/`CHECK_EQ`/`CHECK_STR_EQ` record a failure and keep going, so one broken case reports every mismatch rather than only the first. diff --git a/README.md b/README.md index 84bd50a..6377a88 100644 --- a/README.md +++ b/README.md @@ -40,7 +40,7 @@ label says which, so either works: | Choosing paths | Tap a path to tick or untick it, or move with **D-pad ↑/↓** and press **A**. **Y** ticks all (or clears all), **B** goes back. The choice lasts until 3DSync is closed; a remote that none of the ticked paths go to is not connected | | During a sync | **START** stops after the current file | | Conflict | The versions that changed are listed; **A** keeps the highlighted one (the 3DS's at first). Tap a version, or move to it with **R** or **D-pad ↑/↓**. **X** skips the file, **L** applies the answer to every remaining conflict, **START** stops. The log still scrolls with ←/→, the Circle Pad and the scroll strip | -| End of a debug run | **A** save the log, **B** discard it | +| End of a run | **A** save its log, **B** discard it | | Problems in 3DSync.ini, at startup | **A** continue to the menu, **START** exit; the list scrolls like the log | The log can be scrolled at any time, including while a sync is running: @@ -514,12 +514,16 @@ The rest is not filtered — a debug run prints one line per file examined. The log on the top screen can be scrolled back while the run is going (see [Controls](#controls)), but for a bug report a file beats a photo. -So a debug run does not leave it at the screen: when the sync ends, 3DSync -offers to write everything it printed to the SD card as -`/3ds/3DSync/logs/debug--