Emulate Common TLS Fingerprints + HTTP/SOCK5 Proxy Support

[Silvenga]

As noted here #446 (comment) - I suspect we are being blocked automatically by Fastly bot detection.

So as a fix for the current TLS block, this pr switches outgoing HTTP requests to use wreq - designed to emulate popular browsers fingerprints (TLS, JA3/JA4, and HTTP/2). I opted to using a small pool of popular browsers/os, one randomly used.

Existing hyper responses handing was preserved with a compatibility layer. This introduces using the BoringSSL TLS stack (built from source by rust, so it introduced some build complexities).

I also added proxy support since it was a wreq option (#458) and moved the tests to be consistently in a test mod.

[ButteredCats]

Successfully lets me fetch pages from Reddit where I'm usually blocked. However, all images meant to be proxied from Reddit except emojis are giving a 307 redirect to their Reddit counterpart which results in them not loading.

[jnobbe]

Pardon my ignorance, but does this need to be built or can it be referenced in a compose file? @Silvenga @ButteredCats

[Silvenga]

Give me a bit, I broke something in a recent commit. This would be building the Dockerfile.ubuntu.

[Silvenga]

Sorry about that, should be good now @ButteredCats @jnobbe

[ButteredCats]

All good lol. I'll apply the patch on one of my servers and let you know how it goes!

[jnobbe]

@Silvenga , this is my first time doing a build instead of pulling an image.

I was able to build using docker buildx build -f ./Dockerfile.ubuntu -t Silvenga/redlib .

I swapped the new image name into my compose and I'm now getting an error about an env not being found in the build directory. I don't know if this is due to a (likely) oversight on my part.

Also, in the readme I saw you added info about boringssl. Is that included as a part of what you've done, does it need to be done on the host, or something else?

[dave0003]

This is working for me so far (pr509 is still working for me, also).

[dave0003]

I was also able to use a socks proxy by adding something like this to my systemd service file under [service]:

Environment="https_proxy=socks5h://127.0.0.1:8430"

Very nice to have socks support, thank you.

[jnobbe]

OK, I found the solution to the env error and successfully pulled and built the PR. I'll report back any issues @Silvenga .

[ButteredCats]

Works for sure and I think this is the way to go! Still regularly getting ratelimited but I don't think there's a good fix for that.

[oifj34f34f]

Hi! I want to make some changes to redlib, so I cloned the repository (this fork) via VSCode (Dev Containers extension), but I'm getting:

0.299 (!) The 'moby' option is not supported on Debian 'trixie' because 'moby-cli' and related system packages have been removed from that distribution.
0.299 (!) To continue, either set the feature option '"moby": false' or use a different base image (for example: 'debian:bookworm' or 'ubuntu-24.04').
0.299 ERROR: Feature "Docker (Docker-in-Docker)" (ghcr.io/devcontainers/features/docker-in-docker) failed to install! Look at the documentation at https://github.com/devcontainers/features/tree/main/src/docker-in-docker for help troubleshooting this error.

What am I doing wrong? I'm using Docker (WSL2) on Windows 11.

Actually, disabling the moby option helps:

{
    "name": "Rust",
    "image": "mcr.microsoft.com/devcontainers/rust:dev-trixie",
    "features": {
        "ghcr.io/devcontainers/features/docker-in-docker:2": {
            "moby": false
        }
    },
    "portsAttributes": {
        "8080": {
            "label": "redlib",
            "onAutoForward": "notify"
        }
    },
    "postCreateCommand": "sudo apt-get update && sudo apt-get install -y git build-essential cmake libclang-dev",
    "postStartCommand": "cargo build"
}

But can we do this by default? Or is the issue on my end (even though I haven't changed anything)? What is the correct way?

[Silvenga]

Hi! I want to make some changes to redlib, so I cloned the repository (this fork) via VSCode (Dev Containers extension), but I'm getting:

TBH, I don't use dev-containers, so I'm not sure - I just kind of guessed at what the updates needed to be hoping someone that uses them would come by 😁. Regarding trixie, I just took the liberty in applying security updates, so I also updated the image OS's to latest LTS - it's not strictly needed, of course those other solutions also seem legit.

[erusc]

Just out of curiosity, wouldn't it appear suspicious to emulate Windows or regular browsers?
Because when visiting via a desktop browser, there's no request (as shown in DevTools) to OAuth endpoints like /auth/v2/oauth/access-token/loid.

[Silvenga]

@erusc so my logic is that it's Fasty just doing bot detection - and not someone actively trying to block Redlib.

I don't think it would be worth it to Reddit to have an engineer manually annoy Redlib users, it's just not cost effective. Engineering doesn't want to block us, leadership does. It would be vastly more cost effective to pay the engineers to build generic blocking solutions that automatically adapt - or even better, outsource this to their CDN (Fastly).

So let's focus on Fastly bot detection. Fastly is looking for bots - typically programs using the standard libraries without customization. Now, Fastly isn't going to just have engineers go out and find the FPs of hundreds of developer libraries - what they are going to look at are FPs that are both somewhat common, yet not incredibly common (like from legit browsers they manually test). It's "machine learning" - getting signals and running a statistical analysis for outliers and blocking based on what they believe is a reasonable risk (unlikely to block someone legitimate).

Fastly isn't going to look at the context of the request (a mobile request), it's just looking at the surface signals (is this request statistically likely to be from a bot?). Would it be suspicious to a human? Yep! To a generic statistical engine? Highly unlikely.

Ultimately, we want to blend into the crowd here. The OS really just modifies the user-agent string, iirc and sets the other common headers like sec-ch-ua. We are TLS FPs right now, the OS emulation is just part of the package that also future proofs us against future advancements in bot detection (e.g., looking at the headers, looking at the case of the headers, order of the headers, etc.).

I picked Windows because that's just the most popular client out there. Android was added to increase evasion while ideally having a high chance of different Redlib instances picking the same device/browser to emulate (for privacy of long running instances, which felt likely).