HyperTransfer

English | 中文

A teaching-first FTP server: a half-sync/half-async network service taken apart so each piece can be explained, with every security decision covered by a test you can run yourself.

This is a rewrite of the 2013 HyperTransfer. The original was a student project with good instincts and dangerous execution: a remote heap overflow, arbitrary file read/write, connections identified by raw file descriptors, sessions that were never freed. The two documents that explain the whole story are docs/DESIGN-NOTES.md (what was wrong and why) and docs/VERIFICATION.md (how each fix is proven).

 ┌──────────────┐   command lines   ┌──────────────────┐  transfer job  ┌──────────────┐
 │ reactor      │ ────────────────▶ │ session pool     │ ─────────────▶ │ io pool      │
 │ thread:epoll │ ◀──────────────── │ FSM, PBKDF2      │ ◀───────────── │ file I/O     │
 └──────────────┘   reply text      └──────────────────┘   data chunks  └──────────────┘
        ▲                                                                      │
        └───────────────────── data socket writes ◀────────────────────────────┘

Why it is worth reading

Every deliberate choice below maps to a concrete engineering problem:

Choice	Problem it addresses
core/ never touches a socket	~90% of protocol logic is unit-testable in milliseconds — no server, no ports, no sleep
Filesystem and passive ports are interfaces	Tests inject in-memory fakes; security checks exist in exactly one place
Connection identity is a conn_id, not an fd	fds get reused by the kernel; using one as identity is a guaranteed bug
One line per command in a dispatch table	The original's 12-level if/else chain silently broke anonymous login
Sharding by connection id	A session is only ever touched by one thread, so it needs no lock
Bounded in-flight chunks	Backpressure: a slow disk or slow client cannot blow up memory
exit() only in main()	A library that exits turns one client's input into a whole-server outage

Quick start

Dependencies: a C++20 compiler and CMake ≥ 3.20. No third-party libraries.

Platform	Toolchain	What gets built
Linux	GCC 13+ / Clang 16+	Everything, including io/ and the server binary
Windows	MSVC (VS 2022 Build Tools) + Windows SDK	core + tests only; io/ is epoll-based and does not build on Windows
macOS	—	core compiles, io/ is unsupported

On Windows, CMake passes /utf-8 to MSVC. This is not optional: the sources are UTF-8 while MSVC defaults to the system code page (GBK on Chinese Windows), and Chinese comments otherwise trigger C4819 and corrupt the parse of the following code. See docs/VERIFICATION.md §4.3.

git clone https://github.com/Tianjian/HyperTransfer.git
cd HyperTransfer
cmake -S . -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build --parallel
ctest --test-dir build --output-on-failure

On Linux an end-to-end test is registered automatically: it starts a real server and drives it with Python's ftplib — login, list, upload/download 200 KB of random data, rename, delete, and two path-traversal attempts that must be rejected.

Run it

# 1. pick a directory to export
sudo mkdir -p /srv/hypertransfer && sudo chown "$USER" /srv/hypertransfer

# 2. create an account (the password is read from the TTY, never from argv)
./build/hyper_transfer_server --hash-password alice --permissions all \
    | grep '^alice:' > conf/users.conf

# 3. point root at /srv/hypertransfer in conf/ftp.conf, then:
./build/hyper_transfer_server --check --config conf/ftp.conf
./build/hyper_transfer_server --config conf/ftp.conf

Connect with any client: lftp -u alice ftp://127.0.0.1:2121, FileZilla, curl.

Repository layout

src/core/          Portable C++20, no sockets — the whole protocol lives here
  command.*        Line splitting with an explicit length limit (fixes the heap overflow)
  session.*        FTP state machine: table-driven dispatch, permissions, REST, PASV
  path_jail.*      Virtual path -> real path; the only security boundary
  filesystem.*     Filesystem abstraction + RealFileSystem
  listing.*        Pure LIST / MLSD / NLST formatting
  crypto.*         SHA-256 / HMAC / PBKDF2 / Base64, with standard test vectors
  user_store.*     Hashed passwords, constant-time comparison, duplicate rejection
  config.*         key=value parsing with typed validation (legacy keys accepted)

src/io/            Platform-specific
  epoll_reactor.*  ~150 lines of event loop
  thread_pool.*    Connection-sharded pools (portable, and unit-tested)
  server.*         Passive mode, backpressure, timeouts, graceful shutdown

tests/             114 unit tests on Windows / 115 on Linux, plus an end-to-end script
scripts/           One-command verification (Linux/WSL2) and a transfer benchmark
docs/              Architecture, design notes, verification, performance, config, migration
conf/              Example configuration and password file format

Supported commands

USER PASS QUIT NOOP SYST FEAT OPTS UTF8 PWD CWD CDUP TYPE MODE STRU PASV EPSV LIST NLST MLSD MLST RETR STOR APPE SIZE MDTM REST DELE MKD RMD RNFR RNTO ABOR STAT

Not implemented: active mode (PORT/EPRT, answered with an explicit 502) and FTPS. If you need encryption on the wire, use SFTP.

Capability limits — please read before using this for anything real

This is a teaching project, not a production server:

1. Plaintext on the wire. FTP does not encrypt credentials or data. Same as the original; this rewrite does not change it. Use SFTP or HTTPS if you need that.
2. No chroot. path_jail guarantees a virtual path cannot escape root (including through symlinks), but the process still has whatever filesystem permissions it was started with. For stronger isolation use systemd's ProtectSystem=strict / PrivateTmp=yes, or run it in a container.
3. Single node, no persisted state. No crash-safe resume bookkeeping, no replication, no quotas.
4. IPv4 only. EPSV is implemented, but the listening socket is AF_INET.

If you just need a working file service, use sftpgo, proftpd or vsftpd. If you want to understand how a network server is actually put together, this repository is organised for that.

Testing

ctest --test-dir build --output-on-failure     # everything
./build/ht_tests path_jail                      # only tests whose name matches
cmake -S . -B build-asan -DHT_ENABLE_SANITIZERS=ON && cmake --build build-asan

Some tests are deliberately kept as regressions for real defects in the original:

Test	Original defect
splitter_rejects_overlong_line_without_newline	remote heap overflow in command accumulation
user_store_requires_exact_name_match	prefix-matching usernames (sh matched share)
session_anonymous_login_works_when_enabled	broken if/else chain made anonymous login fail
session_cannot_navigate_above_root	strstr path checks plus unhandled ..
path_jail_blocks_symlink_escape	symlink escape
config_rejects_out_of_range_port	end_port = 65536 sat in the repo for a decade
session_repeated_pasv_closes_previous_listener	repeated PASV leaked listening fds

Verified locally on MSVC 19.44 (Debug/Release, /W4 /WX) and GCC 13.3 (Debug/Release, -Werror), and in CI on GCC, Clang and MSVC, plus ASan/UBSan. A 600 MiB transfer benchmark lives in docs/PERFORMANCE.md.

Documentation

The detailed documents are currently written in Chinese; English translations are welcome. One line each:

Document	Contents
ARCHITECTURE	Thread roles, a full RETR sequence, why sharding instead of locking, backpressure
DESIGN-NOTES	15 sections: what the original did, why it was wrong, what replaced it, how it is tested
VERIFICATION	How to prove each fix, plus the bugs that only a real Linux build and MSVC exposed
PERFORMANCE	600 MiB benchmark, bottleneck analysis, honest measurement limits
CONFIGURATION	Every key, password file format, permission bits, systemd unit
MIGRATION	Old file → new file, old key → new key, behaviour changes

License

MIT © 2013-2026 Tianjian.

Use it, change it, ship it commercially, redistribute it — just keep the copyright and license notice. No warranty.