diff --git a/CHANGELOG.md b/CHANGELOG.md index 9905896..d62e8cb 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,6 +8,95 @@ itself is built on: a change to what the gateway is believed to accept carries the probe that established it and the date it was run. "The vendor's documentation says so" is not one of those, and an entry that rests on it says so outright. +## Unreleased + +**PyPI serves 0.1.3 and `__version__` in this tree is also `0.1.3`**, so the two +are different libraries answering the same number. That is deliberate - bumping +the version is step 1 of the release procedure and nothing here releases without +an instruction - and it is written down because the last time it happened nobody +wrote it down and two trees called themselves 0.1.1 for five days. Anything +quoted about what `nodemaven` does has to say which tree it was read from until +this is released. + +### `pip install nodemaven[requests]` + +The quickstart imported `requests` and `pip install nodemaven` did not install +it. Reported on 2026-09-10 by an outside developer reading 0.1.3 from PyPI, and +reproduced the same day in a clean venv holding only `nodemaven==0.1.3`: +`import requests` after the documented install is a `ModuleNotFoundError`, which +is the first thing that happens to anyone who copies the README. + +The fix is an optional extra, declared in `[project.optional-dependencies]`, and +**`requests` is still not a dependency of this package** - `grep -rn "import +requests" src/` finds nothing, and it will go on finding nothing. +`Proxy.requests()` is a dict of strings and `check.connect()` writes its own +CONNECT by hand, for the reason under 0.1.3 below: an HTTP library throws away +the status line, the reason phrase and the headers, which on this gateway are +the diagnosis. Making `requests` a hard dependency would have cost the one +property that separates this package from the vendor's own client, to fix one +code block. + +Two things worth keeping from how the fix was chosen. `pip install requirements` +was considered and does not exist - `pypi.org/pypi/requirements/json` answers +**404**, measured 2026-09-10, so that spelling breaks a step earlier with a less +legible error - and `-r requirements.txt` names a file that an installer coming +from PyPI never has. And **an undeclared extra is not an error**: pip answers +`nodemaven[nosuchthing]` with a warning and installs the package without it, so +the documented line would have gone on failing while the README looked fixed. +That is why `test_readme.py` now reads `pyproject.toml` and asserts every extra +the documentation names is declared. + +`test_every_third_party_import_is_named_in_an_install_line` is the general +version: every `import` in every Python block in every document this suite +checks has to be either the standard library, `nodemaven` itself, or named in a +`pip install` line in the same file. It is what would have caught this before +the release rather than after it. + +### The README was split, and nothing in it was deleted + +984 lines, over half of them reference and measurement, so a reader arriving +from PyPI met the API reference before the second example. The measurements +moved rather than shrank - the argument they make only works at full length, and +summarising a measurement turns it into an opinion: + +- [`docs/api-reference.md`](https://github.com/nodemaven/nodemaven-python/blob/main/docs/api-reference.md) + - what every public name takes, returns and raises. +- [`docs/validation.md`](https://github.com/nodemaven/nodemaven-python/blob/main/docs/validation.md) + - the five status codes a wrong value comes back as, the fold list, why a + separator is refused, why session ids are hexadecimal. +- [`docs/observed-behavior.md`](https://github.com/nodemaven/nodemaven-python/blob/main/docs/observed-behavior.md) + - what the account API does that its own specification does not say. + +Every link out of the README is absolute and points at `blob/main`, because +**PyPI resolves nothing relative**: a `](docs/validation.md)` resolves against +`pypi.org/project/nodemaven/` and 404s there, which is a production link going +nowhere. That rule was written in three HTML comments in the README and enforced +by nobody; it is now two tests. + +**The split is the moment those tests were most likely to stop testing +anything.** Every one of them scanned `README.md` by name, and content moving +out from under a by-name scan is exactly the shape a test takes when it goes +green for the wrong reason. So `test_readme.py` now names its corpus rather than +globbing it - a new file under `docs/` fails the suite until somebody decides +what checks it is owed - the per-file checks say which file they mean, and every +scan asserts it found something first. Each new test was checked by breaking +what it guards and confirming it fails: renaming the extra, removing the install +line, making a link relative, deleting a `docs/` link, and renaming a +` ```python ` fence to ` ```py `. + +### `api.py`'s module docstring was wrong, and shipped that way in 0.1.3 + +Its "What is still not measured" block said the six write endpoints had never +been called and that the page-number base was unknown. Both had been settled on +2026-09-09, by `--phase 11` and `--phase 9`, whose findings are written into the +method docstrings **of the same file** - `Paging.first_cursor` has carried the +page base since, and `_refuse_unnumbered` was deleted with it. + +Corrected in place with the note, because the failure is worth more than the +correction: a module docstring is a summary of the module and reads as the +authoritative statement of what is known, so it was believed over the code it +summarises. The check is `grep` in the source and it was not run. + ## 0.1.3 - 2026-09-09 17:42 Uploaded by `publish.yml` through PyPI Trusted Publishing, from the `v0.1.3` diff --git a/README.md b/README.md index af787ad..edab4b2 100644 --- a/README.md +++ b/README.md @@ -28,9 +28,14 @@ + the order, which is why this comment does. -[Quickstart](#quickstart) · [Reference](#reference) · [Parameters](#parameters) · [Errors](#errors) · [Sticky sessions](#sticky-sessions) · [Why the validation is the point](#why-the-validation-is-the-point) · [Asking the gateway](#asking-the-gateway) · [Account API](#account-api) · [What it does not do](#what-this-library-does-not-do) · [Other gateways](#other-gateways) · [Docs](https://docs.nodemaven.com?utm_source=github&utm_medium=sdk_python&utm_campaign=readme) + Kept as one line rather than the vertical list an outside review asked for, + 2026-09-10, and the reason is where it renders: this block is the PyPI page + above the install command, and twelve bullets there push `pip install` off + the first screen. A nav is the one list a reader is meant to scroll past. --> + +[Quickstart](#quickstart) · [Parameters](#parameters) · [Sticky sessions](#sticky-sessions) · [Errors](#errors) · [Account API](#account-api) · [Other gateways](#other-gateways) · [Documentation](#documentation) @@ -39,17 +44,28 @@ the input that gateway would mishandle, and hands the result to whatever HTTP client you already use. Two calls do reach the network, both by name and neither on import: -[`proxy.check()`](#asking-the-gateway) opens one CONNECT and tells you what the -gateway said about it, and [`Client`](#account-api) talks to the dashboard API. +`proxy.check()` opens one CONNECT and tells you what the gateway said about it, +and [`Client`](#account-api) talks to the dashboard API. **Works with** requests · httpx · aiohttp · Playwright · Patchright · Puppeteer · curl - and anything else that takes a proxy URL, because that is all it hands back. +## Install + ``` pip install nodemaven ``` +Nothing else is required. The only dependency is `tomli`, and only on Python +3.10 and older, where the standard library has no TOML parser. + + + ## Quickstart `login` and `password` are the **Proxy Username and Proxy Password** assigned under @@ -59,48 +75,75 @@ credentials in the username at all; both are described in [authentication methods](https://docs.nodemaven.com/en/articles/9979031-authentication-methods). ```python -import requests from nodemaven import Proxy proxy = Proxy(login="your-login", password="your-password", country="us", filter="medium") -r = requests.get("https://api.ipify.org", proxies=proxy.requests()) -print(r.text) +print(proxy.check()) ``` ``` -203.0.113.42 +200 Connection established via gate.nodemaven.com:8080 in 0.42s, exit 203.0.113.7 ``` -**If that is not your own address, it worked.** If it is your own, the request -never went through the proxy. + -### Why not just write the URL yourself? +That is one CONNECT and no traffic through the tunnel. **A refusal is a return +value, not an exception**, because the status code is the thing you came for and +raising would bury it in a traceback: -Because the gateway does not tell you when you get it wrong. A misspelt -parameter is not refused - the tunnel opens, the setting is dropped, and the -traffic you are paying to route through a medium-quality US pool goes out -wherever the gateway felt like: +``` +407 Proxy Authentication Required via gate.nodemaven.com:8080 in 0.19s +usually NOT your credentials, despite what the status says. A value the gateway +will not take on `country`, `filter`, `ttl`, `type` or `speed` answers 407, and +so does a wrong password. Check the values before the password - and check the +case of `ttl`, which is the one value that is case-sensitive: `10M` is refused +where `10m` is accepted. +``` + +That second paragraph is data in the gateway definition, not a string in this +library, because what a status code means is per-gateway. + +`check()` tunnels to `api.ipify.org:443` by default and whatever you name will +see a TCP connection from your exit address, so the target is a parameter rather +than a constant. The timeout defaults to 15 seconds, because one of this +gateway's measured reactions is no reply for about 20 seconds. ```python -# by hand: a typo the gateway answers 200 to, and never mentions again -"http://user-country-us-filtr-medium:pass@gate.nodemaven.com:8080" +proxy.check(target="example.com:443", timeout=15.0) +``` -# with this library: refused before anything is sent -Proxy(login="user", password="pass", country="us", filtr="medium") +### Sending traffic through it + +The example below uses `requests`, which this package does **not** install - it +has no HTTP client of its own and does not want one. Install it alongside: + +``` +pip install nodemaven[requests] +``` + +```python +import requests +from nodemaven import Proxy + +proxy = Proxy(login="your-login", password="your-password", + country="us", filter="medium") + +r = requests.get("https://api.ipify.org", proxies=proxy.requests()) +print(r.text) ``` ``` -ParamError: NodeMaven does not know the parameter 'filtr': it is answered with -200 and dropped, so the connection would succeed and your setting would NOT be -applied. Known: ['city', 'country', 'filter', 'ipv4', 'isp', 'region', 'sid', -'speed', 'ttl', 'type'] +203.0.113.42 ``` -That is the whole reason the package exists, and -[why the validation is the point](#why-the-validation-is-the-point) has the -measurements behind it. +**If that is not your own address, it worked.** If it is your own, the request +never went through the proxy. Credentials can come from the environment instead, so nothing is in your source: @@ -109,7 +152,7 @@ Credentials can come from the environment instead, so nothing is in your source: proxy = Proxy(country="us", filter="medium") ``` -The same identity, for other clients: +### The same identity, for other clients ```python proxy.url() # http://user:pass@gate.nodemaven.com:8080 - httpx, aiohttp, curl @@ -122,6 +165,10 @@ proxy.server # host:port, no credentials With Playwright, Patchright or Puppeteer: +``` +pip install playwright +``` + ```python from playwright.sync_api import sync_playwright @@ -130,170 +177,32 @@ with sync_playwright() as p: context = browser.new_context(proxy=proxy.playwright()) ``` -## Reference +## Why not just write the URL yourself? - - -What every public name takes, returns and raises. Nothing in this section needs -the ones after it; those carry the measurements the refusals were built on. - -### `Proxy` +Because the gateway does not tell you when you get it wrong. A misspelt +parameter is not refused - the tunnel opens, the setting is dropped, and the +traffic you are paying to route through a medium-quality US pool goes out +wherever the gateway felt like: ```python -Proxy(*, login=None, password=None, host=None, port=None, provider=None, **params) -``` - -Keyword-only. Builds a username and opens nothing. - -| argument | falls back to | refused when | -|---|---|---| -| `login` | `NODEMAVEN_LOGIN` | missing → `CredentialsError` | -| `password` | `NODEMAVEN_PASSWORD` | missing → `CredentialsError` | -| `host` | `NODEMAVEN_HOST`, then the definition's own | - | -| `port` | `NODEMAVEN_PORT`, then the definition's own | not a whole number 1 to 65535 → `CredentialsError` | -| `provider` | the shipped `nodemaven` definition | - | -| `**params` | - | a name outside `known_params`, or a value that is empty, carries a separator, carries whitespace the definition does not fold, or is outside a list the definition declares → `ParamError` | - -The environment names come from the definition's id in upper case, so a gateway -of your own reads its own pair - see [Other gateways](#other-gateways). - -| attribute | is | -|---|---| -| `.username` | the username, in the gateway's dialect | -| `.server` | `host:port`, with no credentials in it | -| `.params` | the parameters as they will be sent, already folded. A copy | -| `.provider` | the `Provider` behind it | - -| call | returns | -|---|---| -| `.url(scheme="http")` | `http://user:pass@host:port`, both credentials percent-encoded | -| `.requests(scheme="http")` | `{"http": ..., "https": ...}` | -| `.httpx(scheme="http")` | `{"http://": ..., "https://": ...}` | -| `.playwright()` | `{"server": ..., "username": ..., "password": ...}`, credentials **not** encoded | -| `.session(session_id)` | a new `Proxy` pinned to that sticky session | -| `.sessions(count, *, length=6)` | a list of `count` new `Proxy` objects, ids distinct | -| `.replace(**changes)` | a new `Proxy`; a value of `None` removes that parameter | -| `.check(*, target="api.ipify.org:443", timeout=15.0)` | a `Check`. **The only call here that opens a socket** | - -`session()` raises `ParamError` on a definition that declares no session -parameter. `sessions()` ids are `2 * length` hexadecimal characters, so the -default holds 2\*\*48; it raises `ParamError` when `count` asks for more distinct -ids than `length` bytes can hold, rather than looping forever looking for them. - -### `Check` - -Returned by `proxy.check()`. Frozen, and a refusal arrives as one of these -rather than as an exception. - -| attribute | is | -|---|---| -| `.ok` | `True` only on 200 | -| `.status` | the CONNECT status, an `int` | -| `.reason` | the reason phrase, verbatim - it labels which back end answered | -| `.server` | the `host:port` that was asked | -| `.elapsed` | seconds, including DNS and the TCP handshake | -| `.headers` | every response header, keys lower-cased | -| `.exit_ip` | the exit address, or `None` when the gateway did not send one | -| `.meaning` | what this status means **on this gateway**, or `None` | - -`CheckError` is raised only when nothing usable came back at all: DNS failure, a -refused connection, a timeout, a response head that never ended, or a first line -that is not a status line. Refusals that happen before anything is sent are -`CheckError` too, and those messages end in *Nothing was sent.* - -### `Client` and `Page` +# by hand: a typo the gateway answers 200 to, and never mentions again +"http://user-country-us-filtr-medium:pass@gate.nodemaven.com:8080" -```python -Client(api_key=None, *, base_url=None, timeout=30.0, transport=None) +# with this library: refused before anything is sent +Proxy(login="user", password="pass", country="us", filtr="medium") ``` -`api_key` falls back to `NODEMAVEN_APIKEY` and raises `CredentialsError` when -neither is set. `transport` is `(method, url, headers, body) -> (status, bytes)` -and defaults to `urllib.request`. - -| call | returns | -|---|---| -| `.me()` | the account object, the server's own field names | -| `.countries()` `.regions()` `.cities()` | a `Page` | -| `.zip_codes()` `.zip_code_regions()` `.zip_code_cities()` | a `Page` | -| `.isps()` `.isp_regions()` `.isp_cities()` | a `Page` - `isps()` has an [envelope that is not the usual one](#the-isp-catalogue-answers-a-different-shape) | -| `.statistics_data(proxy_username, **filters)` / `.statistics_requests(...)` | a `dict` | -| `.domain_statistics(proxy_username, **filters)` | a `Page` that does not page - the whole answer is one object | -| `.sub_users()` `.whitelist_ips()` | a `Page`, numbered by page rather than by offset | -| `.whitelist_ip(id)` | a `dict` | -| `.create_sub_user(username, password, *, traffic_limit=None, is_traffic_limited=None, **extra)` | a `dict` | -| `.update_sub_user(id, **changes)` / `.delete_sub_user(id)` | a `dict` | -| `.reset_sub_user_usage(ids)` | a `dict`. Takes a list, not one id | -| `.upsert_whitelist_ip(ip, ports_count, *, name=None, protocol="HTTP", sticky=None, ttl=None, **extra)` | a `dict`. Creates **or replaces**. [`protocol` is always sent](#the-whitelist-needs-a-field-the-spec-marks-optional) | -| `.delete_whitelist_ip(id)` | a `dict` | -| `.iterate(page, *, max_pages=100)` | an iterator over the rest of the collection, page by page | -| `.validate(proxy)` | a list of problem strings, empty when the catalogue agrees | - -Every list call takes `**filters`, passed through as query parameters in the -server's own spelling - `country__code="us"`. - -**Paging differs by endpoint and the difference is not cosmetic.** The nine -location endpoints take `limit` and `offset`, both **always sent**: `limit` -defaults to `DEFAULT_PAGE_SIZE`, 1000, and `offset` to 0. `sub_users()` takes -`page` and `per_page`; `whitelist_ips()` takes `page` and `page_size`, whose -documented default is 5 and maximum 100. An unknown query parameter is ignored -rather than refused, so one hard-coded pair would silently do nothing on two -thirds of this surface. - -1000 is the largest limit the location endpoints accept: `limit=10000` is -refused outright by `isps`, and on `cities` it returns the same 1000 rows -`limit=1000` does. - -**That 1000 is a ceiling on the answer, not a count of the rows behind it.** -Asking `cities` for the next offset at the same limit returns a further 965, so -one call sees about half that collection and nothing in the reply says so - -`count`, `next` and `previous` all come back `None`. Use `iterate()` there. The -other catalogues do fit in one request. - -A `Page` iterates its own rows and has a `len()`. `.count`, `.next` and -`.previous` are the server's, and are `None` everywhere - no schema in the -vendor's own specification declares either field, so this is the API's shape and -not a gap in one endpoint. `.request_path`, `.request_params` and `.paging` -record the call the page came from, which is what lets `iterate()` ask for the -next one. - -`iterate()` raises `ApiError` rather than looping in four cases: more than -`max_pages` pages, a next-page url that has already been returned, a page -identical to the one before it - the server accepted the cursor and ignored it - -and a next-page url pointing at a different scheme, host or port than -`base_url`. The host check is there because the request that would follow the -url carries the API key in a header. - -**Both page-numbered endpoints start at page 1**, measured 2026-09-09 by -`lab/probes/probe_account_api.py --phase 9`. On `sub-users/` the evidence is -where the rows are - page 0 came back with none and page 1 with the account's -only sub-user, which a 0-based server cannot produce. On `whitelist/ips` the -collection is empty, so the reading is weaker: page 1 is the only number -answered `200` at all, while 0, 2 and 9999 are answered `404`. - -**The two endpoints do not end their collections the same way.** Past the last -page `sub-users/` answers `200` with an empty payload and `whitelist/ips` -answers `404`, so a `404` is treated as the end of the walk on the whitelist and -nowhere else - everywhere else it is still an error, because a `404` is also -what a wrong path looks like. - -### `Provider` and the module functions - -| call | returns | -|---|---| -| `load(provider_id="nodemaven")` | a shipped definition | -| `load_file(path, provider_id=None)` | a definition from a TOML file; the id is the filename unless named | -| `available()` | the ids of every shipped definition, as a list | +``` +ParamError: NodeMaven does not know the parameter 'filtr': it is answered with +200 and dropped, so the connection would succeed and your setting would NOT be +applied. Known: ['city', 'country', 'filter', 'ipv4', 'isp', 'region', 'sid', +'speed', 'ttl', 'type'] +``` -A `Provider` is a frozen description of one gateway: `id`, `label`, -`known_params`, `status`, `prefix`, `separator`, `pair_separator`, -`session_param`, `host`, `port`, `aliases`, `values`, `normalize`, -`connect_reactions`, `exit_ip_header`, `source`, `source_read`, `notes`. Only -`id`, `label` and `known_params` are required. `.is_measured` is `True` when -`status` is `measured`. +That is the whole reason the package exists. The gateway answers a wrong *value* +five different ways and names the parameter in none of them; the table of what +it does instead, and the probes behind it, are in +[docs/validation.md](https://github.com/nodemaven/nodemaven-python/blob/main/docs/validation.md). ## Parameters @@ -315,74 +224,79 @@ against the gateway rather than transcribed: | `sid` | the sticky session - see below | any string with no `-` | | `ttl` | how long that session is held | `1m`, `10m`, `10h`, `24h` | | `filter` | IP quality | `low`, `medium`, `high` | -| `speed` | claims a connection speed class - see below | `fast`, `slow` | -| `ipv4` | claims to force IPv4 - see below | `True` | +| `speed` | claims a connection speed class | `fast`, `slow` | +| `ipv4` | claims to force IPv4 | `True` | `type` picks a different pool rather than a filter over one pool. Five requests -per arm with a fresh `sid` and `country=us`: -`type=mobile` drew T-Mobile and Verizon Wireless ASNs, while `type=residential` -and leaving it unset drew Comcast, Charter, Windstream and other wireline -carriers, with no mobile ASN among them. +per arm with a fresh `sid` and `country=us`: `type=mobile` drew T-Mobile and +Verizon Wireless ASNs, while `type=residential` and leaving it unset drew +Comcast, Charter, Windstream and other wireline carriers, with no mobile ASN +among them. **`ipv4` and `speed` are confirmed names whose effects are unmeasured**, and the -table says `claims to` for that reason. For `ipv4` the name took a different -method to confirm: a junk value on it is answered `200`, so it cannot be told -apart from an unimplemented name that way. What tells them apart is the sticky -session, whose key is the parsed parameter set - over two -independent session ids with both controls holding, `ipv4=True` moves the exit -and an unknown name does not. `ipv4=False` lands on the same exit as leaving it -out, which is what a default would do and also what a dropped value would do. - -**Names are validated. Values, on this gateway, are not.** Passing a name that is -not in this table raises before anything is sent, because the gateway answers an -unknown name with 200 and drops the setting. Values are passed through, because -what is known is which ones have been observed to work - and that is not the same -as the set the gateway accepts. Refusing on a guessed list would block a setting -that would have worked, which is the worse mistake of the two. The schema does -carry a per-parameter list of legal values and refuses anything outside it; the -shipped definition leaves that list empty for every parameter, deliberately, and -a definition you write yourself gets the check as soon as you fill it in. - -### Case and spacing - -`country`, `region`, `city`, `isp` and `type` are folded before they are sent: -surrounding whitespace trimmed, ASCII `A-Z` lowered, each remaining space turned -into `_`. `country="US"` and `country="us"` are therefore the same request, and +table says `claims to` for that reason. + +**Names are validated. Values, on this gateway, are not.** A name outside the +table raises before anything is sent, because the gateway answers an unknown +name with 200 and drops the setting. Values are passed through, because what is +known is which ones have been *observed* to work, and that is not the same as +the set the gateway accepts - refusing on a guessed list would block a setting +that would have worked. + +`country`, `region`, `city`, `isp` and `type` are folded before they are sent - +trimmed, lowered, and each remaining space turned into `_` - so `country="US"` +and `region="District of Columbia"` are not your problem: ```python Proxy(login="u", password="p", region="District of Columbia").username # u-region-district_of_columbia ``` -`region-district_of_columbia` is the form this gateway generates for itself - it -appears in a username the dashboard issued - and the vendor's own client applies -the same transformation. Without the fold a space reaches the username, which -cannot carry one: the CONNECT line is a single token, so the value is either -malformed or cut short. +`sid`, `filter`, `ttl` and `speed` keep their case, and for `ttl` that matters: +`ttl-10m` opens the tunnel and `ttl-10M` is answered `407`. Every value that is +not folded is refused if it contains whitespace. + +Credentials come from `NODEMAVEN_LOGIN` and `NODEMAVEN_PASSWORD` when not passed +in, and the gateway address from `NODEMAVEN_HOST` and `NODEMAVEN_PORT`. -**`sid`, `filter`, `ttl` and `speed` are sent with their case unchanged.** `sid` -is yours, and folding an identifier a caller chose would rename their session, so -it is left alone whatever the gateway does with it. +Which names fold, why the values are not checked against a list, and what each +wrong value is answered with, are in +[docs/validation.md](https://github.com/nodemaven/nodemaven-python/blob/main/docs/validation.md). -For the other three, **pass lower case**, and for `ttl` that is not advice: -`ttl-10m` opens the tunnel and `ttl-10M` is answered `407`, -which reads as a credentials problem and is not one. `filter` and `speed` were -not refused in either case, and this library still does not fold them - what the -gateway accepts today and what it will accept next month are different claims, -and the fold list is data in the gateway definition rather than a decision in -this package. +## Sticky sessions -Which parameters fold is declared in the gateway definition, as data, so a -gateway you describe yourself folds what you say it folds and nothing else. +One `Proxy` is one identity. Pin it to a sticky session: -**Every other value is refused if it contains whitespace.** There is no spelling -of a space that works here - `username` would emit it raw, `url()` would -percent-encode it to `%20`, and `playwright()` would hand over a third thing - -so between the fold and the refusal, no value with whitespace in it can reach -the wire by any path. +```python +held = proxy.session("order4417") +``` -Credentials come from `NODEMAVEN_LOGIN` and `NODEMAVEN_PASSWORD` when not passed -in, and the gateway address from `NODEMAVEN_HOST` and `NODEMAVEN_PORT`. +**A session id cannot contain the character the gateway separates parameters +with**, which for this one is `-`, and passing one raises rather than +connecting. The gateway cuts the value at the separator, so without the refusal +every order id beginning `order` would quietly share one session and one exit. + +**The session key is the whole parameter set, not the session id.** Adding or +removing any parameter moves you to a different exit address, which is why +parameters change through a method that returns a new object rather than by +assignment - the move is a different identity, and the code should say so: + +```python +germany = proxy.replace(country="de") # a new identity, a new exit +plain = proxy.replace(filter=None) # also a new identity +``` + +For a worker pool, one identity per worker: + +```python +for identity in proxy.sessions(50): + queue.put(identity) # each one a different exit +``` + +Each id is `2 * length` hexadecimal characters from `secrets`, `length=6` by +default, and the ids are distinct **within one call**. The measurements behind +the separator rule, and why the ids are hex rather than `uuid4()`, are in +[docs/validation.md](https://github.com/nodemaven/nodemaven-python/blob/main/docs/validation.md). ## Errors @@ -416,239 +330,12 @@ anything is sent; `AuthError` from `Client` is the **dashboard API key**, and it has been to the server. A library that reported both as one would tell you to fix the key when the password is wrong. -## Sticky sessions - -One `Proxy` is one identity. Pin it to a sticky session: - -```python -held = proxy.session("order4417") -``` - -**A session id cannot contain the character the gateway separates parameters -with**, which for this one is `-`, and passing one raises rather than connecting. -That is measured and not a precaution: a probe opened tunnels with -`sid-order8e3bf9-4417` and with `sid-order8e3bf9`, four rounds each, interleaved, -and both landed on **one exit address** while a third arm spelled -`sid-order8e3bf94417` held a different one throughout. The gateway cuts the value -at the separator and reads the rest as something else, so every order id -beginning `order` would quietly share one session and one exit. - -**The same cut applies to every parameter, not just `sid`,** which is why a -separator in any value is refused. `isp-verizon` opens the -tunnel, a junk `isp` is answered `406`, and `isp-verizon-zzqqx-zzqqx` is answered -`200` - so the gateway took `verizon` as the ISP and read the tail as a parameter -name it does not know, which it drops silently. - -**The session key is the whole parameter set, not the session id.** -`country=us, sid=A` and `country=us, sid=A, filter=medium` are two different -sessions on the gateway, so adding or removing any parameter moves you to a -different exit address. That is why parameters change through a method that -returns a new object rather than by assignment - the move is a different -identity, and the code should say so: - -```python -germany = proxy.replace(country="de") # a new identity, a new exit -plain = proxy.replace(filter=None) # also a new identity -``` - -**The set, not the order.** Measured over 20 rounds a side: the canonical -parameter order and a shuffled one drew the same exit 20 times each, while a -control differing by one parameter *value* drew a different exit 20 times. So -the order this library emits parameters in cannot change which exit you get. - -### Many sessions at once - -For a worker pool, one identity per worker: - -```python -for identity in proxy.sessions(50): - queue.put(identity) # each one a different exit -``` - -Each id is `2 * length` hexadecimal characters from `secrets`, `length=6` by -default, and the ids are distinct **within one call**. Asking for the whole -space or more raises `ParamError` - `sessions(256, length=1)` wants every one of -the 256 ids an eight-bit space holds, and drawing them without repeating is a -loop that either never finishes or leaves nothing for the next caller. - -Hex, and not the alphabets people reach for first, because the gateway cuts a -value at its separator and every id sharing a prefix then collapses onto one -exit - silently, since the connection still succeeds. `secrets.token_urlsafe` -emits `-` and `_`, `uuid4()` emits `-` four times, base64 emits `+` and `/`, and -each of those is a separator on some gateway. From `secrets` and not `random` -because `random` is seeded from the clock: two workers starting in the same -millisecond would draw the same ids. - -## Why the validation is the point - - - -Every gateway behaviour below was measured against the live gateway rather than -transcribed from documentation, and each one carries its date and the probe -behind it in [CHANGELOG.md](https://github.com/nodemaven/nodemaven-python/blob/main/CHANGELOG.md). - -A gateway is bad at telling you that you got the username wrong. One class of -mistake - a value it will not take - comes back five different ways, and not one -of them names the parameter. Read by raw CONNECT, one arm per row: - -| you sent | the gateway answers | -|---|---| -| bad `country` value | `407 Proxy Authentication Required` | -| bad `region` value | `406 Not Acceptable` | -| bad `city` value | `406 Not Acceptable` | -| `city` sent without `region` | `500 Internal Server Error` | -| bad `isp` value | `406 Not Acceptable`, and `410 Gone` for `comcast` | -| bad `filter` value | `407 Proxy Authentication Required` | -| bad `ttl` value | `407 Proxy Authentication Required` | -| bad `type` or `speed` value | `407 Proxy Authentication Required` | -| empty value | nothing, the connection hangs about 20 s | -| **unknown parameter name** | **`200`, and the parameter is ignored** | - -Every `407` there sends you to check credentials that are correct, and the `406` -does not even say which of the two parameters it refused: a bad `region`, a bad -`isp` and `charter` - a real ISP - all answer it, so it separates neither the -parameter nor a misspelling from a pool you cannot have. `comcast` is the one -value measured to answer `410` instead, which reads as a name the gateway knows -and a pool this account cannot reach - one ISP, so read it that narrowly. - -**The two `city` rows are one rule: send `city` with its `region`.** Measured -over six CONNECTs holding the login, the password, the target, the -gateway host and port and the parameter order fixed. `country=us`, -`region=louisiana`, `city=abbeville` answers `200`, and so does a second city in -a second region. The same city with the region left out answers `500`, and an -invented name sent with a real region answers `406` - so the gateway does look -the name up, and the `500` is a request it could not resolve rather than a fault -on their side. `Client.validate()` refuses that combination before it goes out. - -The last row is worse than any of them: the request succeeds, your code carries -on, and the setting you asked for was never applied. Nothing that comes back -over the wire can tell you. - -`ttl` counts in minutes and hours - `1m`, `10m`, `10h` and `24h` connect, while -`10s`, `10d` and a bare `10` are answered `407`. It is also the one parameter -whose value case matters: `10M` is refused where `10m` is accepted. Parameter -*names* are case-insensitive at the gateway; this library folds values for -`country`, `region`, `city`, `isp` and `type` to the wire form anyway, so a space -or a capital in a place name is not your problem. - -So this library checks before anything is sent: - -```python ->>> Proxy(login="u", password="p", contry="us") -ParamError: NodeMaven does not know the parameter 'contry': it is answered with -200 and dropped, so the connection would succeed and your setting would NOT be -applied. Known: ['city', 'country', 'filter', 'ipv4', 'isp', 'region', 'sid', -'speed', 'ttl', 'type'] -``` - -## Asking the gateway - -Validation catches everything knowable without sending anything. For the rest - -a wrong password, a country the pool does not have, a value the gateway dislikes -- there is one call that opens a single CONNECT and reports what came back: - -```python -result = proxy.check() -``` - - - -``` -200 Connection established via gate.nodemaven.com:8080 in 0.42s, exit 203.0.113.7 -``` - -The exit address arrives **on the CONNECT reply itself**, on a header the gateway -definition names, so knowing where you came out costs one handshake and no -traffic through the tunnel. - -**Do not build anything on it being there.** More than one implementation -answers behind this hostname, which one you reach is decided by your username, -and they do not agree about the header: one measured `200` carried -`X-Exit-IP` where the shipped definition names `X-Proxy-Exit-IP`, and others -send no address at all. So `result.exit_ip` is `None` more often than the -definition suggests, and that is normal rather than an error. If you need the -address every time, read it through the tunnel from a service that echoes it. - -**A refusal is a return value, not an exception.** The status code is the thing -you came for, and raising would bury it in a traceback - which is what a general -HTTP client does. So a failed tunnel comes back as an object, carrying the -gateway's own reading of its own status code: - -``` -407 Proxy Authentication Required via gate.nodemaven.com:8080 in 0.19s -usually NOT your credentials, despite what the status says. A value the gateway -will not take on `country`, `filter`, `ttl`, `type` or `speed` answers 407, and -so does a wrong password. Check the values before the password - and check the -case of `ttl`, which is the one value that is case-sensitive: `10M` is refused -where `10m` is accepted. -``` - -That second paragraph is data in the gateway definition, not a string in this -library, because what a status code means is per-gateway. The fields a `Check` -carries are in the [reference](#check); `CheckError` and when it is raised are -there too. - -**`ok` does not mean your parameters were applied.** An unrecognised parameter -name is also answered `200` and dropped, which is the whole reason the section -above refuses unknown names before sending. No call can recover that after the -fact, and this one does not pretend to. - -`check()` names the host it tunnels to - `api.ipify.org:443` by default - and -whatever you name will see a TCP connection from your exit address. There is no -CONNECT to nowhere, so this is a parameter rather than a constant: - -```python -proxy.check(target="example.com:443", timeout=15.0) -``` - -The timeout defaults to 15 seconds rather than something brisk, because one of -this gateway's measured reactions is no reply for about 20 seconds. A 5-second -timeout would report that as a network problem. - ## Account API Quota, usage, sub-users and the location catalogue. Separately credentialled, because the API key and the proxy password are different secrets from different places. -**Every path here now comes from the vendor's own OpenAPI specification**, read -2026-09-09. Five of the calls have also been sent against a live account and -answered: `users/me`, `countries`, `regions`, `cities` and `isps`, measured -2026-09-08. **The rest are still transcribed** - from the specification now -rather than from the vendor's client at `github.com/nodemavencom/proxy`, -`python/nodemaven/client.py`, where four paths were wrong. The five write calls -have still never been called, on purpose: probing one costs a real object on a -production account. - - - -**A wrong path is not answered `404` here**, measured 2026-09-09. -`locations/zip-codes/`, `statistics/` and `whitelist-ips/` - three paths this -package sent before the specification was read - are answered `200`, -`text/html`, 6415 bytes, byte-identical to a path invented on the spot as a -negative control. The dashboard serves its front end for anything it does not -route, so **calling a path cannot tell you whether the path exists.** If you -wrap a path of your own against this API, compare its response against a path -nobody could have implemented rather than against a status code. - -The real paths are `locations/zipcodes/` (solid, no separator), -`statistics/data/`, `statistics/requests/`, `statistics/domains/` and -`whitelist/ips`. `sub-users/` was real all along but was being read under -`results`, where the rows are under `payload`. - -Calling `isps` is how its [different envelope](#the-isp-catalogue-answers-a-different-shape) -was found; the other four came out of the specification. - ```python from nodemaven import Client @@ -658,31 +345,16 @@ print(me["data"]) # traffic left ``` `me()` returns the server's own object with its own field names, unrenamed and -unmodelled. It answers six fields: - -| field | type | | -|---|---|---| -| `data` | `int` | traffic left. **Not** `traffic_left` | -| `email` | `str` | | -| `is_traffic_frozen` | `str` | **not a bool** - see below | -| `proxy_password` | `str` | the proxy password, not your API key | -| `proxy_username` | `str` | | -| `subscription_status` | `str` | | - -**`is_traffic_frozen` is a string, so `if me["is_traffic_frozen"]:` is true -whichever way it reads.** Compare it against the value rather than for truth. - -Not modelling this into a dataclass is deliberate, and the response above is the -argument for it. Written a day earlier from the vendor's client, a dataclass -would have declared `traffic_left`, which does not exist, and typed -`is_traffic_frozen` as a bool, which it is not. The raw dict was wrong about -nothing, because it claimed nothing. +unmodelled - `data` is the traffic left and there is no `traffic_left`, and +`is_traffic_frozen` is a **string**. **It also returns your proxy password in +clear text, on every call, and so does every row of `sub_users()`.** Do not +print, log or paste either. ```python client.countries() # the catalogue, paginated client.regions(country__code="us") # Django's field lookup, the server's spelling client.cities(country__code="us", region__code="dc") -client.isps(country__code="us") # a different envelope, see below +client.isps(country__code="us") # a different envelope, see the docs client.zip_codes(country__code="us") # Which regions and cities the ISP and zip-code catalogues actually cover. @@ -694,7 +366,7 @@ client.zip_code_cities(country__code="us", region__code="dc") # Statistics are per proxy username, and the username is required. # The range is `start` and `end`, not `start_date` and `end_date`, and the -# dates are `dd-mm-yyyy`. ISO is answered 400 - see below. +# dates are `dd-mm-yyyy`. ISO is answered 400. client.statistics_data("acct-1", start="01-09-2026", end="07-09-2026") client.statistics_requests("acct-1", start="01-09-2026") client.domain_statistics("acct-1", period="hours24") @@ -711,29 +383,6 @@ client.upsert_whitelist_ip("203.0.113.7", 10, name="the office") client.delete_whitelist_ip(id) ``` -The dates go as `dd-mm-yyyy`, measured 2026-09-09 by `--phase 10`: -`start=20-08-2026` is answered **200 with 21 data points** and -`start=2026-08-20` is answered **400**, with a body byte-identical to the one -`start=not-a-date` draws. The vendor's document writes the format both ways - -`dd-mm-yyyy` in the prose, `format: date` in the type - and **the prose is the -half that is right**. So ISO is not a rival spelling the server declines, it is -a string the server cannot parse, and every client generated from that -specification sends the one form that fails. - -This paragraph said the opposite until 2026-09-09, and said it for the worst -possible reason: "the type is what the server parses" was an inference about -which half of a self-contradicting document to trust, written before either -half had been sent. The measurement cost one request. - -**`sub_users()` returns each sub-user's `proxy_password` in clear text**, on -every row, by the specification's own required-field list. So does `me()`. Do -not print a row of either, and do not paste one into an issue. - -Five of the vendor's twenty-four documented paths are deliberately not wrapped: -`locations/all-doc/`, `notifications/`, `llm/submit/`, `llm/results/{id}/` and -`llm/balance/`. Listing them is the point - an omission nobody wrote down reads -the same as an oversight. - A list endpoint returns a `Page`, which iterates **one page** and not the collection. `iterate()` gets the rest: @@ -743,99 +392,13 @@ for country in client.iterate(page): # all of them ... ``` -**`page.count` is `None` everywhere, and that is the API's shape rather than a -gap here.** No schema in the vendor's specification declares a `count` or a -`next` at all - `PaginatedCountryList` is `{"results": [...]}` and nothing else - -and a page of 50 drawn from a catalogue of nearly 200 countries came back with -all three fields empty. A full page is therefore indistinguishable from a -complete collection by looking at it, so `iterate()` asks for the next cursor -and reads the answer. - -**It stops on an empty page, not on a short one.** That rule changed on -2026-09-09 and the one before it was unsafe against this server in particular: -it stopped at the first page shorter than the size it had asked for, which is -wrong wherever the server caps the size below the request. `cities(limit=10000)` -is answered with 1000 rows out of 1965, and "shorter than asked" reads those -1000 as the end. The measurement that says so is four paragraphs up in this same -file and was already there - a measurement sitting in a document is not a -measurement anyone applied. Stopping on empty costs one spare request per walk -and cannot truncate. - -`limit` and `offset` are sent on your behalf and are overridable: - -```python -client.countries(limit=50) # four requests instead of one -``` - -They are sent rather than omitted because `isps()` **refuses** a request without -them, while the others answer one with a silently partial list. `offset` is -honoured: `offset=50` returns rows disjoint from `offset=0`, and an offset past -the end answers `200` with zero rows. - -`sub_users()` and `whitelist_ips()` are numbered by page instead, and **both -start at page 1**, measured 2026-09-09 - see -[the reference](#client-and-page). They mark the end of a collection -differently, so `iterate()` cannot use one rule for both: past the last page -`sub-users/` answers `200` with an empty payload and `whitelist/ips` answers -`404`. A `404` therefore ends the walk on the whitelist and nowhere else. - -`iterate()` raises rather than looping in four cases: a next-page url it has -already returned, more than `max_pages` pages, a page identical to the -one before it - which means the server took the cursor and ignored it - and a -next-page url on another host. None is a tuning knob; each is the difference -between a bug you can see and one that surfaces as a rate limit or a short -answer. - -### The ISP catalogue answers a different shape - -`isps()` returns a `Page` like the others and most callers can stop here. The -rest of this section is why it took two runs to get there. - -Its `200` is an object keyed `city`, `country`, `isps` and `region`, with no -`results` anywhere. The rows are the list under `isps` - 358 of them for -`country__code="us"`; the other three fields are strings and are not rows. -`isps()` reads that key and the other four list methods read `results`. That is -per-endpoint knowledge in the client rather than a page builder that goes -looking for whichever value happens to be a list, which is a rule decided by key -order the day a second list appears. - -For a day this method returned a `Page` of exactly one row - the envelope itself -- whatever the account held. Nothing was released in that state; it was written -up as broken rather than fixed, because the run that found it printed the key -*names* and not the values, and a key unwrapped because its name reads right is -how a parameter the gateway ignores once got into this package and a field the -server does not send got into this README. The fix waited for one command, and -was the same edit either way. - -### The whitelist needs a field the spec marks optional - -`upsert_whitelist_ip()` always sends `protocol`, defaulting to `"HTTP"`. The -vendor's OpenAPI document marks `ip` and `ports_count` required and gives -`protocol` a `default: "HTTP"`; the server does not apply that default. Measured -2026-09-09 at 16:18, one field at a time: - -| body | answer | -|---|---| -| `ip`, `ports_count`, `name` | `400 "Please enter a valid protocol(HTTP or SOCKS5)."` | -| the same plus `protocol: "HTTP"` | `201`, `{"ip_id", "message"}`, and a read of `whitelist/ips` found the address | - -So a client written from the specification sends the one body that fails. The -default here is a compensation for that, not a convenience, which is why -`protocol` is a keyword with a value rather than another `None`. - -Two things that run did not settle: `name` was present in every rung, so whether -it is also required is untested, and no rung tried `ip`, `ports_count` and -`protocol` alone, so that trio is not known to be a complete body. - -An earlier version of this section said only that the required pair was refused -and that which field was short was the server's word. That was accurate and it -was one call away from being a measurement. - -### Checking a Proxy against the catalogue +`page.count` is `None` everywhere, paging differs by endpoint, and one page can +be half a collection with nothing in the reply saying so. That is the API's +shape rather than a gap here, and it is measured in +[docs/observed-behavior.md](https://github.com/nodemaven/nodemaven-python/blob/main/docs/observed-behavior.md). The gateway answers a country it does not have with `407`, which reads as a -credentials problem, and does not say which parameter was wrong. The catalogue -knows, so it can be asked: +credentials problem. The catalogue knows better, so it can be asked: ```python problems = client.validate(proxy) @@ -843,70 +406,6 @@ if problems: raise SystemExit("\n".join(problems)) ``` -This is a method on `Client` and not a check inside `Proxy`, for one reason: a -refusal that ships in a release can be wrong forever, and the catalogue moves. -Asking the live catalogue cannot go stale - and it costs a network call, so it -has to be your decision rather than a hidden one. - -Two things are checked. `country` is matched against the catalogue, and **a -`city` sent without a `region` is refused before the request goes out** - that -one needs no network. The gateway answers a city with no region with `500`, -which reads as a fault on their side and is not one; the same city with its own -region answers `200`. The values of `region`, `city` and -`isp` are not matched against the catalogue, because city codes repeat across -regions - `aberdeen`, `albany` and `alexandria` each appear twice in a single -page - so matching a bare code would report success and mean nothing. - -### No dependencies, and your own client if you want one - -The transport is `urllib.request` from the standard library, so this adds nothing -to your dependency tree. If you would rather it went through the client you -already have, that is one function: - -```python -def transport(method, url, headers, body): - r = requests.request(method, url, headers=headers, data=body) - return r.status_code, r.content - -client = Client(transport=transport) -``` - -Return the status rather than raising on it; mapping statuses to exceptions is -this library's job, and doing it in both places is how a `NotFoundError` becomes -somebody else's exception halfway up a stack. - -## What this library does not do - -**It does not retry.** That is deliberate, and it is the one design decision -here taken against a measurement rather than a preference. - -Retrying a refused request is the thing that most reliably makes the next one -worse: each retry confirms automation to the target and burns the exit range for -everyone else sharing the pool. Measured over 1464 attempts, the chance that the -next attempt succeeds, by how many failures came immediately before it: - -| failures before | P(next attempt succeeds) | -|---|---| -| 0 | 75% | -| 1 | 21% | -| 3 | 5.9% | -| 5 | 5.8% | -| 6 | 1.6% | -| 7-9 | 0.5% | - -294 attempts were spent past six consecutive failures and returned 3 pages - 98 -attempts per delivered page, against 1.7 in a healthy session. A library that -shipped automatic retry as a default would be spending that on your behalf -without telling you. - -Those 1464 attempts, and the cells they came from, are in -[nodemaven/proxy-benchmark](https://github.com/nodemaven/proxy-benchmark) - the -harness that measured them, open source, so the table above can be re-run rather -than believed. - -It also does not own an HTTP client, a connection pool or a browser. Those are -yours, and they are better than anything a vendor SDK would bundle. - ## Other gateways **No account here? Any proxy you already have works.** Parameters are data, not @@ -925,8 +424,7 @@ proxy = Proxy(provider=mine, login="u", password="p", host="proxy.example.com", port=8000) ``` -Describe the parameters it does take and it validates those too. Or keep the -definition in a TOML file: +Or keep the definition in a TOML file: ```toml # my-gateway.toml @@ -963,22 +461,39 @@ looked for, so this is one guess you never have to make. Every definition carries a `status`. `measured` means traffic has gone through that gateway and the dialect was read off the wire. `documented` means it was -transcribed from documentation and never exercised. Only `nodemaven` is shipped -here, and it is `measured`. +transcribed from documentation and never exercised. `available()` lists the ids +shipped here, and only `nodemaven` is one of them - it is `measured`. -## Requirements +## What this library does not do -Python 3.9 or newer. No dependencies on 3.11 and newer; `tomli` on older ones. +**It does not retry.** Retrying a refused request is the thing that most +reliably makes the next one worse: measured over 1464 attempts, the chance the +next attempt succeeds falls from 75% with no failures behind it to 1.6% after +six, and 294 attempts spent past six failures returned three pages. A library +that shipped automatic retry as a default would be spending that on your behalf +without telling you. The table is in +[docs/observed-behavior.md](https://github.com/nodemaven/nodemaven-python/blob/main/docs/observed-behavior.md). + +It also does not own an HTTP client, a connection pool or a browser. Those are +yours, and they are better than anything a vendor SDK would bundle. -## Changes +## Documentation -[CHANGELOG.md](https://github.com/nodemaven/nodemaven-python/blob/main/CHANGELOG.md). -Entries carry the probe and the date behind any change to what the gateway is -believed to accept. + -## License +- [API reference](https://github.com/nodemaven/nodemaven-python/blob/main/docs/api-reference.md) - every public name, what it takes and what it raises +- [Why the validation is the point](https://github.com/nodemaven/nodemaven-python/blob/main/docs/validation.md) - what the gateway answers to a wrong value, and why names are refused and values are not +- [Observed behaviour](https://github.com/nodemaven/nodemaven-python/blob/main/docs/observed-behavior.md) - the gateway and dashboard findings this package is built on, each with its run +- [CHANGELOG.md](https://github.com/nodemaven/nodemaven-python/blob/main/CHANGELOG.md) - entries carry the probe and the date behind any change to what the gateway is believed to accept +- [NodeMaven docs](https://docs.nodemaven.com?utm_source=github&utm_medium=sdk_python&utm_campaign=readme) - the product documentation + +## Requirements - +Python 3.9 or newer. No dependencies on 3.11 and newer; `tomli` on older ones. + +## License [MIT](https://github.com/nodemaven/nodemaven-python/blob/main/LICENSE). diff --git a/docs/api-reference.md b/docs/api-reference.md new file mode 100644 index 0000000..2c57266 --- /dev/null +++ b/docs/api-reference.md @@ -0,0 +1,206 @@ +# API reference + +What every public name takes, returns and raises. + + + +Nothing here needs [observed-behavior.md](observed-behavior.md) or +[validation.md](validation.md); those carry the measurements the refusals were +built on. + +- [`Proxy`](#proxy) +- [`Check`](#check) +- [`Client` and `Page`](#client-and-page) +- [`Provider` and the module functions](#provider-and-the-module-functions) +- [Your own transport](#your-own-transport) + +## `Proxy` + +```python +Proxy(*, login=None, password=None, host=None, port=None, provider=None, **params) +``` + +Keyword-only. Builds a username and opens nothing. + +| argument | falls back to | refused when | +|---|---|---| +| `login` | `NODEMAVEN_LOGIN` | missing → `CredentialsError` | +| `password` | `NODEMAVEN_PASSWORD` | missing → `CredentialsError` | +| `host` | `NODEMAVEN_HOST`, then the definition's own | - | +| `port` | `NODEMAVEN_PORT`, then the definition's own | not a whole number 1 to 65535 → `CredentialsError` | +| `provider` | the shipped `nodemaven` definition | - | +| `**params` | - | a name outside `known_params`, or a value that is empty, carries a separator, carries whitespace the definition does not fold, or is outside a list the definition declares → `ParamError` | + +The environment names come from the definition's id in upper case, so a gateway +of your own reads its own pair. + +| attribute | is | +|---|---| +| `.username` | the username, in the gateway's dialect | +| `.server` | `host:port`, with no credentials in it | +| `.params` | the parameters as they will be sent, already folded. A copy | +| `.provider` | the `Provider` behind it | + +| call | returns | +|---|---| +| `.url(scheme="http")` | `http://user:pass@host:port`, both credentials percent-encoded | +| `.requests(scheme="http")` | `{"http": ..., "https": ...}` | +| `.httpx(scheme="http")` | `{"http://": ..., "https://": ...}` | +| `.playwright()` | `{"server": ..., "username": ..., "password": ...}`, credentials **not** encoded | +| `.session(session_id)` | a new `Proxy` pinned to that sticky session | +| `.sessions(count, *, length=6)` | a list of `count` new `Proxy` objects, ids distinct | +| `.replace(**changes)` | a new `Proxy`; a value of `None` removes that parameter | +| `.check(*, target="api.ipify.org:443", timeout=15.0)` | a `Check`. **The only call here that opens a socket** | + +`session()` raises `ParamError` on a definition that declares no session +parameter. `sessions()` ids are `2 * length` hexadecimal characters, so the +default holds 2\*\*48; it raises `ParamError` when `count` asks for more distinct +ids than `length` bytes can hold, rather than looping forever looking for them. + +## `Check` + +Returned by `proxy.check()`. Frozen, and a refusal arrives as one of these +rather than as an exception. + +| attribute | is | +|---|---| +| `.ok` | `True` only on 200 | +| `.status` | the CONNECT status, an `int` | +| `.reason` | the reason phrase, verbatim - it labels which back end answered | +| `.server` | the `host:port` that was asked | +| `.elapsed` | seconds, including DNS and the TCP handshake | +| `.headers` | every response header, keys lower-cased | +| `.exit_ip` | the exit address, or `None` when the gateway did not send one | +| `.meaning` | what this status means **on this gateway**, or `None` | + +`CheckError` is raised only when nothing usable came back at all: DNS failure, a +refused connection, a timeout, a response head that never ended, or a first line +that is not a status line. Refusals that happen before anything is sent are +`CheckError` too, and those messages end in *Nothing was sent.* + +`.exit_ip` is `None` more often than the definition suggests - see +[the exit address is not promised](observed-behavior.md#the-exit-address-is-not-promised). + +## `Client` and `Page` + +```python +Client(api_key=None, *, base_url=None, timeout=30.0, transport=None) +``` + +`api_key` falls back to `NODEMAVEN_APIKEY` and raises `CredentialsError` when +neither is set. `transport` is `(method, url, headers, body) -> (status, bytes)` +and defaults to `urllib.request`. + +| call | returns | +|---|---| +| `.me()` | the account object, the server's own field names | +| `.countries()` `.regions()` `.cities()` | a `Page` | +| `.zip_codes()` `.zip_code_regions()` `.zip_code_cities()` | a `Page` | +| `.isps()` `.isp_regions()` `.isp_cities()` | a `Page` - `isps()` has an [envelope that is not the usual one](observed-behavior.md#the-isp-catalogue-answers-a-different-shape) | +| `.statistics_data(proxy_username, **filters)` / `.statistics_requests(...)` | a `dict` | +| `.domain_statistics(proxy_username, **filters)` | a `Page` that does not page - the whole answer is one object | +| `.sub_users()` `.whitelist_ips()` | a `Page`, numbered by page rather than by offset | +| `.whitelist_ip(id)` | a `dict` | +| `.create_sub_user(username, password, *, traffic_limit=None, is_traffic_limited=None, **extra)` | a `dict` | +| `.update_sub_user(id, **changes)` / `.delete_sub_user(id)` | a `dict` | +| `.reset_sub_user_usage(ids)` | a `dict`. Takes a list, not one id | +| `.upsert_whitelist_ip(ip, ports_count, *, name=None, protocol="HTTP", sticky=None, ttl=None, **extra)` | a `dict`. Creates **or replaces**. [`protocol` is always sent](observed-behavior.md#the-whitelist-needs-a-field-the-spec-marks-optional) | +| `.delete_whitelist_ip(id)` | a `dict` | +| `.iterate(page, *, max_pages=100)` | an iterator over the rest of the collection, page by page | +| `.validate(proxy)` | a list of problem strings, empty when the catalogue agrees | + +Every list call takes `**filters`, passed through as query parameters in the +server's own spelling - `country__code="us"`. + +**Those filters are not validated, and this server ignores a query parameter it +does not know.** A misspelt filter is therefore accepted by Python, sent, and +dropped, with nothing anywhere saying so. The filter names the statistics +endpoints have are `proxy_username`, `timezone`, `start`, `end`, `period`, +`request_source` and `limit`. + +### Paging + +**It differs by endpoint and the difference is not cosmetic.** The nine location +endpoints take `limit` and `offset`, both **always sent**: `limit` defaults to +`DEFAULT_PAGE_SIZE`, 1000, and `offset` to 0. `sub_users()` takes `page` and +`per_page`; `whitelist_ips()` takes `page` and `page_size`, whose documented +default is 5 and maximum 100. An unknown query parameter is ignored rather than +refused, so one hard-coded pair would silently do nothing on two thirds of this +surface. + +1000 is the largest limit the location endpoints accept: `limit=10000` is +refused outright by `isps`, and on `cities` it returns the same 1000 rows +`limit=1000` does. + +**That 1000 is a ceiling on the answer, not a count of the rows behind it.** +Asking `cities` for the next offset at the same limit returns a further 965, so +one call sees about half that collection and nothing in the reply says so - +`count`, `next` and `previous` all come back `None`. Use `iterate()` there. The +other catalogues do fit in one request. + +A `Page` iterates its own rows and has a `len()`. `.count`, `.next` and +`.previous` are the server's, and are `None` everywhere - no schema in the +vendor's own specification declares either field, so this is the API's shape and +not a gap in one endpoint. `.request_path`, `.request_params` and `.paging` +record the call the page came from, which is what lets `iterate()` ask for the +next one. + +`iterate()` raises `ApiError` rather than looping in four cases: more than +`max_pages` pages, a next-page url that has already been returned, a page +identical to the one before it - the server accepted the cursor and ignored it - +and a next-page url pointing at a different scheme, host or port than +`base_url`. The host check is there because the request that would follow the +url carries the API key in a header. + +**Both page-numbered endpoints start at page 1**, and they do not mark the end +of a collection the same way - see +[paging is three conventions](observed-behavior.md#paging-is-three-conventions-not-one). + +## `Provider` and the module functions + +| call | returns | +|---|---| +| `load(provider_id="nodemaven")` | a shipped definition | +| `load_file(path, provider_id=None)` | a definition from a TOML file; the id is the filename unless named | +| `available()` | the ids of every shipped definition, as a list | + +A `Provider` is a frozen description of one gateway: `id`, `label`, +`known_params`, `status`, `prefix`, `separator`, `pair_separator`, +`session_param`, `host`, `port`, `aliases`, `values`, `normalize`, +`connect_reactions`, `exit_ip_header`, `source`, `source_read`, `notes`. Only +`id`, `label` and `known_params` are required. `.is_measured` is `True` when +`status` is `measured`. + +## Your own transport + +The transport is `urllib.request` from the standard library, so the account API +adds nothing to your dependency tree. If you would rather it went through the +client you already have, that is one function. The example below is `requests`, +which this package does not install: + +``` +pip install nodemaven[requests] +``` + +```python +import requests + +def transport(method, url, headers, body): + r = requests.request(method, url, headers=headers, data=body) + return r.status_code, r.content + +client = Client(transport=transport) +``` + +Return the status rather than raising on it; mapping statuses to exceptions is +this library's job, and doing it in both places is how a `NotFoundError` becomes +somebody else's exception halfway up a stack. diff --git a/docs/observed-behavior.md b/docs/observed-behavior.md new file mode 100644 index 0000000..d6ea1d1 --- /dev/null +++ b/docs/observed-behavior.md @@ -0,0 +1,271 @@ +# Observed behaviour + + + +The gateway and the dashboard API both answer in ways no client can guess. Each +finding below is a run, not a reading of the vendor's documentation - where the +two disagree, which happens often, the run is what this package is built on. + +- [The exit address is not promised](#the-exit-address-is-not-promised) +- [A wrong path is not answered 404](#a-wrong-path-is-not-answered-404) +- [What `me()` actually returns](#what-me-actually-returns) +- [Statistics dates are `dd-mm-yyyy`](#statistics-dates-are-dd-mm-yyyy) +- [Paging is three conventions, not one](#paging-is-three-conventions-not-one) +- [The ISP catalogue answers a different shape](#the-isp-catalogue-answers-a-different-shape) +- [The whitelist needs a field the spec marks optional](#the-whitelist-needs-a-field-the-spec-marks-optional) +- [What the catalogue can and cannot check](#what-the-catalogue-can-and-cannot-check) +- [Why there is no automatic retry](#why-there-is-no-automatic-retry) + +## The exit address is not promised + +The exit address arrives **on the CONNECT reply itself**, on a header the +gateway definition names, so knowing where you came out costs one handshake and +no traffic through the tunnel. + +**Do not build anything on it being there.** More than one implementation +answers behind this hostname, which one you reach is decided by your username, +and they do not agree about the header: one measured `200` carried `X-Exit-IP` +where the shipped definition names `X-Proxy-Exit-IP`, and others send no address +at all. So `result.exit_ip` is `None` more often than the definition suggests, +and that is normal rather than an error. If you need the address every time, +read it through the tunnel from a service that echoes it. + +**`ok` does not mean your parameters were applied.** An unrecognised parameter +name is answered `200` and dropped, which is why unknown names are refused +before anything is sent - see [validation.md](validation.md). No call can +recover that after the fact, and `check()` does not pretend to. + +The timeout defaults to 15 seconds rather than something brisk, because one of +this gateway's measured reactions is no reply for about 20 seconds. A 5-second +timeout would report that as a network problem. + +## A wrong path is not answered 404 + +Measured 2026-09-09. `locations/zip-codes/`, `statistics/` and +`whitelist-ips/` - three paths this package sent before the vendor's OpenAPI +specification was read - are answered `200`, `text/html`, 6415 bytes, +byte-identical to a path invented on the spot as a negative control. The +dashboard serves its front end for anything it does not route, so **calling a +path cannot tell you whether the path exists.** If you wrap a path of your own +against this API, compare its response against a path nobody could have +implemented rather than against a status code. + +This is the same defect as the gateway answering `200` to an unknown username +parameter and dropping it, one layer up. + +The real paths are `locations/zipcodes/` (solid, no separator), +`statistics/data/`, `statistics/requests/`, `statistics/domains/` and +`whitelist/ips`. `sub-users/` was real all along but was being read under +`results`, where the rows are under `payload`. + +**Every path this package uses now comes from the vendor's own OpenAPI +specification**, read 2026-09-09. What has also been *sent* is more than the +paths: `users/me`, `countries`, `regions`, `cities` and `isps` were measured +2026-09-08, and all five wrapped write calls were measured 2026-09-09 against a +throwaway sub-user and an RFC 5737 address, both removed in the same run. The +rest are transcribed - from the specification now rather than from the vendor's +client at `github.com/nodemavencom/proxy`, `python/nodemaven/client.py`, where +four paths were wrong. + +**The specification is documentation and gets no more authority than the +vendor's client did.** Its `info.description` names a host, `api.nodemaven.com`, +that answers 404 at nginx, and four of the five writes answered something it does +not declare: `create` answers `201` where its path declares only `200`, +`reset/usage` answers an array where every other envelope here carries an +object, `delete` answers `200` with a body where the document declares `204`, +and the whitelist upsert is refused when sent exactly the fields its own schema +marks required. + +Five of the vendor's twenty-four documented paths are deliberately not wrapped: +`locations/all-doc/`, `notifications/`, `llm/submit/`, `llm/results/{id}/` and +`llm/balance/`. Listing them is the point - an omission nobody wrote down reads +the same as an oversight. + +## What `me()` actually returns + +`me()` returns the server's own object with its own field names, unrenamed and +unmodelled. It answers six fields: + +| field | type | | +|---|---|---| +| `data` | `int` | traffic left. **Not** `traffic_left` | +| `email` | `str` | | +| `is_traffic_frozen` | `str` | **not a bool** - see below | +| `proxy_password` | `str` | the proxy password, not your API key | +| `proxy_username` | `str` | | +| `subscription_status` | `str` | | + +**`is_traffic_frozen` is a string, so `if me["is_traffic_frozen"]:` is true +whichever way it reads.** Compare it against the value rather than for truth. + +Not modelling this into a dataclass is deliberate, and the response above is the +argument for it. Written a day earlier from the vendor's client, a dataclass +would have declared `traffic_left`, which does not exist, and typed +`is_traffic_frozen` as a bool, which it is not. The raw dict was wrong about +nothing, because it claimed nothing. + +**`me()` returns a live proxy password, and `sub_users()` returns one per row**, +by the specification's own required-field list. Do not print a row of either, +do not log one, and do not paste one into an issue. One reached a console here +on 2026-09-09 and had to be rotated. + +## Statistics dates are `dd-mm-yyyy` + +Measured 2026-09-09 by `--phase 10`: `start=20-08-2026` is answered **200 with +21 data points** and `start=2026-08-20` is answered **400**, with a body +byte-identical to the one `start=not-a-date` draws. The vendor's document writes +the format both ways - `dd-mm-yyyy` in the prose, `format: date` in the type - +and **the prose is the half that is right**. So ISO is not a rival spelling the +server declines, it is a string the server cannot parse, and every client +generated from that specification sends the one form that fails. + +This paragraph said the opposite until 2026-09-09, and said it for the worst +possible reason: "the type is what the server parses" was an inference about +which half of a self-contradicting document to trust, written before either half +had been sent. The measurement cost one request. + +All three statistics endpoints answer **500** - not 400 - when `start`, `end` +and `period` are all omitted, though the document marks all three optional. So +`domain_statistics("acct-1")` on its own is an example of a 500. `period` takes +`today` or `hours24`. + +## Paging is three conventions, not one + +`limit`/`offset` on the nine location endpoints, `page`/`per_page` on +`sub_users()`, `page`/`page_size` on `whitelist_ips()`. An unknown query +parameter is ignored rather than refused, so a single hard-coded pair silently +did nothing on two thirds of this surface until 2026-09-09. + +`limit` and `offset` are sent on your behalf rather than omitted because +`isps()` **refuses** a request without them, while the others answer one with a +silently partial list. `offset` is honoured: `offset=50` returns rows disjoint +from `offset=0`, and an offset past the end answers `200` with zero rows. + +**`iterate()` stops on an empty page, not on a short one.** That rule changed on +2026-09-09 and the one before it was unsafe against this server in particular: +it stopped at the first page shorter than the size it had asked for, which is +wrong wherever the server caps the size below the request. `cities(limit=10000)` +is answered with 1000 rows out of 1965, and "shorter than asked" reads those +1000 as the end. The measurement that says so was already written down in this +package's own documentation - a measurement sitting in a document is not a +measurement anyone applied. Stopping on empty costs one spare request per walk +and cannot truncate. + +**`page.count` is `None` everywhere, and that is the API's shape rather than a +gap here.** No schema in the vendor's specification declares a `count` or a +`next` at all - `PaginatedCountryList` is `{"results": [...]}` and nothing else - +and a page of 50 drawn from a catalogue of nearly 200 countries came back with +all three fields empty. A full page is therefore indistinguishable from a +complete collection by looking at it, so `iterate()` asks for the next cursor +and reads the answer. + +**Both page-numbered endpoints start at page 1**, measured 2026-09-09 by +`lab/probes/probe_account_api.py --phase 9`. On `sub-users/` the evidence is +where the rows are - page 0 came back with none and page 1 with the account's +only sub-user, which a 0-based server cannot produce. On `whitelist/ips` the +collection is empty, so the reading is weaker: page 1 is the only number +answered `200` at all, while 0, 2 and 9999 are answered `404`. + +**The two endpoints do not end their collections the same way.** Past the last +page `sub-users/` answers `200` with an empty payload and `whitelist/ips` +answers `404`, so a `404` is treated as the end of the walk on the whitelist and +nowhere else - everywhere else it is still an error, because a `404` is also +what a wrong path looks like. + +## The ISP catalogue answers a different shape + +`isps()` returns a `Page` like the others and most callers can stop here. The +rest of this section is why it took two runs to get there. + +Its `200` is an object keyed `city`, `country`, `isps` and `region`, with no +`results` anywhere. The rows are the list under `isps` - 358 of them for +`country__code="us"`; the other three fields are strings and are not rows. +`isps()` reads that key and the other four list methods read `results`. That is +per-endpoint knowledge in the client rather than a page builder that goes +looking for whichever value happens to be a list, which is a rule decided by key +order the day a second list appears. + +For a day this method returned a `Page` of exactly one row - the envelope itself +- whatever the account held. Nothing was released in that state; it was written +up as broken rather than fixed, because the run that found it printed the key +*names* and not the values, and a key unwrapped because its name reads right is +how a parameter the gateway ignores once got into this package and a field the +server does not send got into its documentation. The fix waited for one command, +and was the same edit either way. + +## The whitelist needs a field the spec marks optional + +`upsert_whitelist_ip()` always sends `protocol`, defaulting to `"HTTP"`. The +vendor's OpenAPI document marks `ip` and `ports_count` required and gives +`protocol` a `default: "HTTP"`; the server does not apply that default. Measured +2026-09-09 at 16:18, one field at a time: + +| body | answer | +|---|---| +| `ip`, `ports_count`, `name` | `400 "Please enter a valid protocol(HTTP or SOCKS5)."` | +| the same plus `protocol: "HTTP"` | `201`, `{"ip_id", "message"}`, and a read of `whitelist/ips` found the address | + +So a client written from the specification sends the one body that fails. The +default here is a compensation for that, not a convenience, which is why +`protocol` is a keyword with a value rather than another `None`. + +The success body is `{"ip_id", "message"}` and **neither key is documented** - +the path binds `200` and `201` to a schema whose only property is `message` - +and the identifier is `ip_id` where the listing calls the same value `id`. + +Two things that run did not settle: `name` was present in every rung, so whether +it is also required is untested, and no rung tried `ip`, `ports_count` and +`protocol` alone, so that trio is not known to be a complete body. + +An earlier version of this section said only that the required pair was refused +and that which field was short was the server's word. That was accurate and it +was one call away from being a measurement. + +## What the catalogue can and cannot check + +`Client.validate(proxy)` checks two things. `country` is matched against the +catalogue, and **a `city` sent without a `region` is refused before the request +goes out** - that one needs no network. + +The values of `region`, `city` and `isp` are **not** matched against the +catalogue, because city codes repeat across regions - `aberdeen`, `albany` and +`alexandria` each appear twice in a single page - so matching a bare code would +report success and mean nothing. + +This is a method on `Client` and not a check inside `Proxy`, for one reason: a +refusal that ships in a release can be wrong forever, and the catalogue moves. +Asking the live catalogue cannot go stale - and it costs a network call, so it +has to be your decision rather than a hidden one. + +## Why there is no automatic retry + +Retrying a refused request is the thing that most reliably makes the next one +worse: each retry confirms automation to the target and burns the exit range for +everyone else sharing the pool. Measured over 1464 attempts, the chance that the +next attempt succeeds, by how many failures came immediately before it: + +| failures before | P(next attempt succeeds) | +|---|---| +| 0 | 75% | +| 1 | 21% | +| 3 | 5.9% | +| 5 | 5.8% | +| 6 | 1.6% | +| 7-9 | 0.5% | + +294 attempts were spent past six consecutive failures and returned 3 pages - 98 +attempts per delivered page, against 1.7 in a healthy session. A library that +shipped automatic retry as a default would be spending that on your behalf +without telling you. + +Those 1464 attempts, and the cells they came from, are in +[nodemaven/proxy-benchmark](https://github.com/nodemaven/proxy-benchmark) - the +harness that measured them, open source, so the table above can be re-run rather +than believed. diff --git a/docs/validation.md b/docs/validation.md new file mode 100644 index 0000000..53a6e90 --- /dev/null +++ b/docs/validation.md @@ -0,0 +1,169 @@ +# Why the validation is the point + + + +Every gateway behaviour below was measured against the live gateway rather than +transcribed from documentation, and each one carries its date and the probe +behind it in [CHANGELOG.md](../CHANGELOG.md). + +- [What a wrong value looks like](#what-a-wrong-value-looks-like) +- [Case and spacing](#case-and-spacing) +- [Why a separator in a value is refused](#why-a-separator-in-a-value-is-refused) +- [Why session ids are hexadecimal](#why-session-ids-are-hexadecimal) + +## What a wrong value looks like + +A gateway is bad at telling you that you got the username wrong. One class of +mistake - a value it will not take - comes back five different ways, and not one +of them names the parameter. Read by raw CONNECT, one arm per row: + +| you sent | the gateway answers | +|---|---| +| bad `country` value | `407 Proxy Authentication Required` | +| bad `region` value | `406 Not Acceptable` | +| bad `city` value | `406 Not Acceptable` | +| `city` sent without `region` | `500 Internal Server Error` | +| bad `isp` value | `406 Not Acceptable`, and `410 Gone` for `comcast` | +| bad `filter` value | `407 Proxy Authentication Required` | +| bad `ttl` value | `407 Proxy Authentication Required` | +| bad `type` or `speed` value | `407 Proxy Authentication Required` | +| empty value | nothing, the connection hangs about 20 s | +| **unknown parameter name** | **`200`, and the parameter is ignored** | + +Every `407` there sends you to check credentials that are correct, and the `406` +does not even say which of the two parameters it refused: a bad `region`, a bad +`isp` and `charter` - a real ISP - all answer it, so it separates neither the +parameter nor a misspelling from a pool you cannot have. `comcast` is the one +value measured to answer `410` instead, which reads as a name the gateway knows +and a pool this account cannot reach - one ISP, so read it that narrowly. + +**The two `city` rows are one rule: send `city` with its `region`.** Measured +over six CONNECTs holding the login, the password, the target, the gateway host +and port and the parameter order fixed. `country=us`, `region=louisiana`, +`city=abbeville` answers `200`, and so does a second city in a second region. +The same city with the region left out answers `500`, and an invented name sent +with a real region answers `406` - so the gateway does look the name up, and the +`500` is a request it could not resolve rather than a fault on their side. +`Client.validate()` refuses that combination before it goes out. + +The last row is worse than any of them: the request succeeds, your code carries +on, and the setting you asked for was never applied. Nothing that comes back +over the wire can tell you. + +So this library checks before anything is sent: + +```python +>>> Proxy(login="u", password="p", contry="us") +ParamError: NodeMaven does not know the parameter 'contry': it is answered with +200 and dropped, so the connection would succeed and your setting would NOT be +applied. Known: ['city', 'country', 'filter', 'ipv4', 'isp', 'region', 'sid', +'speed', 'ttl', 'type'] +``` + +**Names are validated. Values, on this gateway, are not.** Passing a name that is +not in the parameter table raises before anything is sent, because the gateway +answers an unknown name with 200 and drops the setting. Values are passed +through, because what is known is which ones have been observed to work - and +that is not the same as the set the gateway accepts. Refusing on a guessed list +would block a setting that would have worked, which is the worse mistake of the +two. The schema does carry a per-parameter list of legal values and refuses +anything outside it; the shipped definition leaves that list empty for every +parameter, deliberately, and a definition you write yourself gets the check as +soon as you fill it in. + +## Case and spacing + +`country`, `region`, `city`, `isp` and `type` are folded before they are sent: +surrounding whitespace trimmed, ASCII `A-Z` lowered, each remaining space turned +into `_`. `country="US"` and `country="us"` are therefore the same request, and + +```python +Proxy(login="u", password="p", region="District of Columbia").username +# u-region-district_of_columbia +``` + +`region-district_of_columbia` is the form this gateway generates for itself - it +appears in a username the dashboard issued - and the vendor's own client applies +the same transformation. Without the fold a space reaches the username, which +cannot carry one: the CONNECT line is a single token, so the value is either +malformed or cut short. + +**`sid`, `filter`, `ttl` and `speed` are sent with their case unchanged.** `sid` +is yours, and folding an identifier a caller chose would rename their session, so +it is left alone whatever the gateway does with it. + +For the other three, **pass lower case**, and for `ttl` that is not advice: +`ttl-10m` opens the tunnel and `ttl-10M` is answered `407`, which reads as a +credentials problem and is not one. `filter` and `speed` were not refused in +either case, and this library still does not fold them - what the gateway +accepts today and what it will accept next month are different claims, and the +fold list is data in the gateway definition rather than a decision in this +package. + +`ttl` counts in minutes and hours - `1m`, `10m`, `10h` and `24h` connect, while +`10s`, `10d` and a bare `10` are answered `407`. Parameter *names* are +case-insensitive at the gateway; values are not, and `ttl` is the one where it +has been measured to matter. + +Which parameters fold is declared in the gateway definition, as data, so a +gateway you describe yourself folds what you say it folds and nothing else. + +**Every other value is refused if it contains whitespace.** There is no spelling +of a space that works here - `username` would emit it raw, `url()` would +percent-encode it to `%20`, and `playwright()` would hand over a third thing - +so between the fold and the refusal, no value with whitespace in it can reach +the wire by any path. + +## Why a separator in a value is refused + +**A session id cannot contain the character the gateway separates parameters +with**, which for this one is `-`, and passing one raises rather than +connecting. That is measured and not a precaution: a probe opened tunnels with +`sid-order8e3bf9-4417` and with `sid-order8e3bf9`, four rounds each, +interleaved, and both landed on **one exit address** while a third arm spelled +`sid-order8e3bf94417` held a different one throughout. The gateway cuts the +value at the separator and reads the rest as something else, so every order id +beginning `order` would quietly share one session and one exit. + +**The same cut applies to every parameter, not just `sid`,** which is why a +separator in any value is refused. `isp-verizon` opens the tunnel, a junk `isp` +is answered `406`, and `isp-verizon-zzqqx-zzqqx` is answered `200` - so the +gateway took `verizon` as the ISP and read the tail as a parameter name it does +not know, which it drops silently. + +**The session key is the whole parameter set, not the session id.** +`country=us, sid=A` and `country=us, sid=A, filter=medium` are two different +sessions on the gateway, so adding or removing any parameter moves you to a +different exit address. That is why parameters change through a method that +returns a new object rather than by assignment - the move is a different +identity, and the code should say so. + +**The set, not the order.** Measured over 20 rounds a side: the canonical +parameter order and a shuffled one drew the same exit 20 times each, while a +control differing by one parameter *value* drew a different exit 20 times. So +the order this library emits parameters in cannot change which exit you get. + +## Why session ids are hexadecimal + +`sessions()` draws from `secrets` in hex, and not from the alphabets people +reach for first, because the gateway cuts a value at its separator and every id +sharing a prefix then collapses onto one exit - silently, since the connection +still succeeds. `secrets.token_urlsafe` emits `-` and `_`, `uuid4()` emits `-` +four times, base64 emits `+` and `/`, and each of those is a separator on some +gateway. + +From `secrets` and not `random` because `random` is seeded from the clock: two +workers starting in the same millisecond would draw the same ids. + +Asking for the whole space or more raises `ParamError` - `sessions(256, length=1)` +wants every one of the 256 ids an eight-bit space holds, and drawing them +without repeating is a loop that either never finishes or leaves nothing for the +next caller. diff --git a/pyproject.toml b/pyproject.toml index 011ff50..dbce2fb 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -27,6 +27,21 @@ classifiers = [ ] dependencies = ["tomli>=1.1.0; python_version < '3.11'"] +[project.optional-dependencies] +# `requests` is not used anywhere in this package and this extra does not change +# that: `grep -rn "import requests" src/` finds nothing, `Proxy.requests()` is a +# three-line dict of strings, and `check.connect()` writes its own CONNECT by +# hand precisely so that no HTTP library is needed to read a status line. +# +# It exists so that the documented example is installable in one line. An +# outside developer reported on 2026-09-10, against 0.1.3 from PyPI, that the +# quickstart imported `requests` while `pip install nodemaven` does not install +# it - reproduced the same day in a clean venv holding only `nodemaven==0.1.3`. +# The alternative was making `requests` a hard dependency, which would have +# broken the one property that separates this package from the vendor's own +# client, for the benefit of one code block. +requests = ["requests>=2.25"] + [project.urls] # These render as the sidebar on the PyPI page, so every one of them has to answer # 200 to a logged-out visitor. @@ -37,12 +52,13 @@ dependencies = ["tomli>=1.1.0; python_version < '3.11'"] # `gh api repos/nodemaven/nodemaven-python --jq .visibility` answered `public` # on 2026-09-08, so both links resolve for a logged-out reader and can stay. # -# **They still will not appear on the package page that is up now.** PyPI bakes -# this table into each uploaded distribution and never re-reads the repository, -# so 0.1.2's sidebar is fixed whatever this file says - re-read on 2026-09-08, -# it carries Homepage and Documentation and no Source. The links arrive with the -# next release; the visibility flip on its own did nothing to them, which is the -# whole reason this note is worth keeping. +# PyPI bakes this table into each uploaded distribution and never re-reads the +# repository, so a visibility flip cannot put a link on a page that is already +# up: 0.1.2's sidebar carried Homepage and Documentation and no Source, re-read +# on 2026-09-08, with both entries already sitting in this file since 0.1.1. +# 0.1.3 shipped on 2026-09-09 and `Source` and `Issues` are on the project page, +# so the rule is measured in both directions now - metadata travels with a +# release and never with anything else. Homepage = "https://nodemaven.com" Documentation = "https://docs.nodemaven.com" Source = "https://github.com/nodemaven/nodemaven-python" diff --git a/src/nodemaven/api.py b/src/nodemaven/api.py index f6e1c80..84c7db1 100644 --- a/src/nodemaven/api.py +++ b/src/nodemaven/api.py @@ -104,14 +104,29 @@ What is still not measured -------------------------- -* **The six write endpoints have never been called** - creating, updating, - deleting and resetting a sub-user, upserting a whitelist address, deleting - one - because each costs a real object on a production account. Their paths, - methods and body fields are the spec's, cross-checked against nothing. -* **The page-number base.** ``sub-users/`` and ``whitelist/ips`` page by page - number rather than by row offset, and neither the spec nor any run says - whether the first page is 0 or 1. This module refuses to guess - see - :class:`Paging` and :meth:`Client.iterate`. +**This section said two things until 2026-09-10 and both had been settled the +day before, by probes whose findings are written into the method docstrings of +this same file.** They are corrected here rather than deleted, because the +failure is worth more than the correction: a module docstring is a summary of +the module, it reads as the authoritative statement of what is known, and it +was believed over the code it summarises. It shipped that way in 0.1.3. + +* It said **the six write endpoints have never been called**. ``--phase 11`` + sent all five wrapped writes on 2026-09-09 against a ``probe_delete_me_`` + sub-user and an RFC 5737 address, each removed in the same run - see + :meth:`Client.create_sub_user`, :meth:`Client.update_sub_user`, + :meth:`Client.reset_sub_user_usage`, :meth:`Client.delete_sub_user` and + :meth:`Client.upsert_whitelist_ip`, each of which carries what its own call + answered. Four of the five answer something the spec does not declare. +* It said **the page-number base** was unknown and that this module refuses to + guess. ``--phase 9`` measured both on 2026-09-09, the refusal was taken out + the same day, and :attr:`Paging.first_cursor` has carried the answer ever + since - ``1`` for both page-numbered endpoints. + +What is genuinely unmeasured is narrower and is recorded where it belongs, on +the call it applies to: whether ``name`` is required by +``whitelist/ip/upsert``, and whether ``ip``, ``ports_count`` and ``protocol`` +alone are a complete body. Neither was varied in the run that measured the rest. Twenty-four paths are in the spec and this module wraps nineteen of them. Not wrapped, deliberately: ``locations/all-doc/``, ``notifications/``, diff --git a/tests/test_readme.py b/tests/test_readme.py index 5f701be..cd3f1f9 100644 --- a/tests/test_readme.py +++ b/tests/test_readme.py @@ -1,19 +1,28 @@ -"""Every output the README quotes, compared against the real thing. +"""Every output the documentation quotes, compared against the real thing. This file exists because the one quotation in the README that had no test drifted without anyone noticing: it listed nine parameter names where the code produced ten, and the Rust port's equivalent test is what caught it. The rule that came -out of that is the reason this file is here - **a README that quotes real output -needs a test per quotation, or the quotation is a comment.** +out of that is the reason this file is here - **a document that quotes real +output needs a test per quotation, or the quotation is a comment.** Comparisons collapse whitespace. That is deliberate and it is the only slack allowed: a paragraph in a fenced block has to be re-wrappable to stay readable at 80 columns, and nothing else about it may change. Every word still has to match. + +**It covers `docs/` as well as `README.md`, from 2026-09-10.** The README was 984 +lines and half of it was reference and measurement, so it was split. The split is +the moment those tests were most likely to become green no-ops - every one of +them scanned `README.md` by name, and content moving out from under a scan is +exactly the shape a test takes when it stops testing anything. So the fixtures +below are the corpus, the per-file checks say which file they mean, and every +scan asserts it found something before it checks it. """ from __future__ import annotations import re +import sys from pathlib import Path import pytest @@ -21,7 +30,19 @@ from nodemaven import Proxy, load from nodemaven.check import Check -README = Path(__file__).resolve().parent.parent / "README.md" +ROOT = Path(__file__).resolve().parent.parent +README = ROOT / "README.md" +DOCS = ROOT / "docs" + +#: The documents this file is responsible for. Named rather than globbed: a new +#: file under `docs/` should fail this suite until somebody decides what checks +#: it is owed, and a glob would adopt it silently with none. +DOCUMENTS = { + "README.md": README, + "docs/api-reference.md": DOCS / "api-reference.md", + "docs/validation.md": DOCS / "validation.md", + "docs/observed-behavior.md": DOCS / "observed-behavior.md", +} def flat(text: str) -> str: @@ -33,12 +54,12 @@ def _headings(readme: str) -> set: **Lines inside fenced code blocks are skipped, and a heading needs a space after its hashes**, both from 2026-09-09. Before that this took any line - beginning with `#`, which in this README means every Python and TOML + beginning with `#`, which in these files means every Python and TOML comment inside every example - `# Statistics are per proxy username` became the anchor `statistics-are-per-proxy-username`, and GitHub creates no such anchor. - That made the two anchor tests below weaker than they read. They check that + That made the anchor tests below weaker than they read. They check that every in-page link points at a heading that exists; with comments in the set, a link could point at a comment and pass. The tests were not wrong about their subject, they were quietly checking a superset of it, and a @@ -61,9 +82,55 @@ def _headings(readme: str) -> set: return headings +def _blocks(text: str): + """Every fenced block in one document, with its language tag.""" + return re.findall(r"```([a-z]*)\n(.*?)```", text, re.DOTALL) + + +def _imports(blocks) -> set: + """Every module name imported by a Python block, top level only.""" + return { + module + for language, body in blocks + if language == "python" + for module in re.findall(r"^\s*(?:import|from)\s+([\w]+)", body, re.M) + } + + +@pytest.fixture(scope="module") +def documents() -> dict: + missing = [name for name, path in DOCUMENTS.items() if not path.exists()] + assert missing == [], f"a document this suite checks is gone: {missing}" + return { + name: path.read_text(encoding="utf-8") for name, path in DOCUMENTS.items() + } + + +@pytest.fixture(scope="module") +def readme(documents: dict) -> str: + return documents["README.md"] + + @pytest.fixture(scope="module") -def readme() -> str: - return README.read_text(encoding="utf-8") +def reference(documents: dict) -> str: + return documents["docs/api-reference.md"] + + +@pytest.fixture(scope="module") +def behavior(documents: dict) -> str: + return documents["docs/observed-behavior.md"] + + +@pytest.fixture(scope="module") +def corpus(documents: dict) -> str: + """Every document at once. + + For assertions of the form "the documentation says X" - which file says it + is an editing decision and should not fail a test. Assertions about layout, + about what a reader arriving from PyPI sees, or about one section's contract + take the single-file fixture instead. + """ + return "\n\n".join(documents.values()) @pytest.fixture(scope="module") @@ -80,9 +147,9 @@ def prose(readme: str) -> str: @pytest.fixture(scope="module") -def blocks(readme: str): - """Every fenced block in the file, with its language tag.""" - return re.findall(r"```([a-z]*)\n(.*?)```", readme, re.DOTALL) +def blocks(corpus: str): + """Every fenced block in the corpus, with its language tag.""" + return _blocks(corpus) class TestTheReadmeIsSelfConsistent: @@ -103,15 +170,15 @@ def test_it_no_longer_claims_nothing_is_raised_from_a_response( assert "nothing here sends one" not in prose def test_the_account_api_separates_what_was_called_from_what_was_not( - self, readme: str + self, behavior: str ): # Documenting a call with an example is a claim that it works, so the - # section has to say which calls that claim rests on. Until 2026-09-08 - # the answer was "none of them" and this test pinned the words - # "transcribed, not measured". Calls were then sent to the live API and - # the paragraph had to change - which is exactly what it was pinned for, - # and it is pinned again for the same reason: the next call that gets - # measured moves the boundary again. + # documentation has to say which calls that claim rests on. Until + # 2026-09-08 the answer was "none of them" and this test pinned the + # words "transcribed, not measured". Calls were then sent to the live + # API and the paragraph had to change - which is exactly what it was + # pinned for, and it is pinned again for the same reason: the next call + # that gets measured moves the boundary again. # # It moved a second time on 2026-09-09, in the other direction. The # section used to argue that transcribed paths were safe to ship because @@ -119,11 +186,20 @@ def test_the_account_api_separates_what_was_called_from_what_was_not( # and the dashboard's own HTML, so that argument was never true here. # The retraction is pinned too, because the tempting edit is to delete a # wrong sentence rather than to say what it cost. - section = readme.split("## Account API", 1)[1].split("\n## ", 1)[0] + # + # It moved a third time on 2026-09-10, and that one is why this test now + # names the writes explicitly. The wording it used to pin - "still never + # been called" - had been false since the day before, when `--phase 11` + # sent all five, and it was false in `api.py`'s module docstring at the + # same time while that file's own method docstrings recorded the runs. + # A test that pins the cautious half of a claim keeps the caution alive + # after the measurement has replaced it. + section = behavior.split("## A wrong path is not answered 404", 1)[1] + section = section.split("\n## ", 1)[0] assert "measured" in section and "transcribed" in section - assert "still never been called" in section assert "negative control" in section - # The five are named. A count with no names cannot be checked by a + assert "all five wrapped write calls were measured" in section + # The read five are named. A count with no names cannot be checked by a # reader, and cannot be checked here either. for path in ("users/me", "countries", "regions", "cities", "isps"): assert f"`{path}`" in section, path @@ -138,7 +214,7 @@ def test_every_error_class_the_package_exports_is_in_the_table( for name in nodemaven.__all__ if name.endswith("Error") and name != "NodeMavenError" ] - table = readme.split("## Errors", 1)[1].split("##", 1)[0] + table = readme.split("## Errors", 1)[1].split("\n## ", 1)[0] missing = [name for name in exported if f"`{name}`" not in table] assert missing == [], f"exported and undocumented: {missing}" @@ -148,18 +224,46 @@ def test_the_nav_line_points_at_headings_that_exist(self, readme: str): assert anchors, "the nav line was not found, so nothing was checked" assert set(anchors) <= _headings(readme), set(anchors) - _headings(readme) - def test_every_anchor_in_the_body_points_at_a_heading_too(self, readme: str): + @pytest.mark.parametrize("name", sorted(DOCUMENTS)) + def test_every_in_page_anchor_points_at_a_heading_in_that_page( + self, name, documents: dict + ): # Widened from the nav line after a section was renamed and two links - # elsewhere in the file went on pointing at the old anchor. GitHub - # renders a dead in-page link as ordinary text that does nothing when - # clicked - no 404, no warning - so nothing but this would have said so. - anchors = set(re.findall(r"\]\(#([a-z0-9-]+)\)", readme)) - assert len(anchors) > 5, "the anchor scan found almost nothing" - assert anchors <= _headings(readme), anchors - _headings(readme) + # elsewhere went on pointing at the old anchor. GitHub renders a dead + # in-page link as ordinary text that does nothing when clicked - no 404, + # no warning - so nothing but this would have said so. + # + # Parametrised over the documents from 2026-09-10. Before the split this + # was one scan of README.md; had it stayed that way, the three files the + # content moved into would have had no anchor checking at all, and they + # are where most of the anchors now are. + text = documents[name] + anchors = set(re.findall(r"\]\(#([a-z0-9-]+)\)", text)) + assert anchors, f"{name} has no in-page links, so nothing was checked" + assert anchors <= _headings(text), (name, anchors - _headings(text)) + + @pytest.mark.parametrize("name", sorted(DOCUMENTS)) + def test_every_cross_document_anchor_points_at_a_heading_that_exists( + self, name, documents: dict + ): + # `docs/api-reference.md` links into `docs/observed-behavior.md` by + # anchor, and a rename on either side breaks it silently the same way an + # in-page link breaks. The split created these links; nothing checked + # them until this test. + text = documents[name] + links = re.findall(r"\]\((?:\.\./)?(?:docs/)?([a-z-]+\.md)#([a-z0-9-]+)\)", text) + if not links: + pytest.skip(f"{name} links into no other document by anchor") + for target, anchor in links: + key = next( + (k for k in documents if k.endswith(target)), None + ) + assert key, f"{name} links at {target}, which this suite does not know" + assert anchor in _headings(documents[key]), (name, target, anchor) def test_a_comment_in_a_code_block_is_not_read_as_a_heading(self): - # The two tests above check that every in-page link points at a heading - # that exists. That is only worth something if the heading set is the + # The tests above check that every in-page link points at a heading that + # exists. That is only worth something if the heading set is the # headings - until 2026-09-09 it also held every `#` comment in every # example, so a link could point at a comment and pass. Both tests were # green the whole time, on a superset of their own subject. @@ -219,11 +323,12 @@ def test_no_statistics_example_carries_an_iso_date(self, blocks): # The server parses `dd-mm-yyyy` and answers an ISO date 400, measured # 2026-09-09 by `--phase 10`. The examples carried ISO until that day. # - # Scoped to the fenced blocks on purpose: the prose two paragraphs down - # quotes `start=2026-08-20` as the form that fails, and a scan over the - # whole file would have to be written to permit the very string it is - # looking for. A test that has to make an exception for the correct case - # is one edit away from making it for the wrong one. + # Scoped to the fenced blocks on purpose: the prose in + # `docs/observed-behavior.md` quotes `start=2026-08-20` as the form that + # fails, and a scan over the whole text would have to be written to + # permit the very string it is looking for. A test that has to make an + # exception for the correct case is one edit away from making it for the + # wrong one. # # This matched `start_date=`/`end_date=` when it was written, because it # was written from the examples rather than from the API. Those names @@ -261,6 +366,152 @@ def test_no_statistics_example_omits_the_range_and_the_period(self, blocks): ) +class TestTheExamplesAreInstallable: + """An example that imports something needs an install line that installs it. + + Written 2026-09-10, after the first outside developer to install the package + reported the first block of the README failing with `ModuleNotFoundError: No + module named 'requests'`. Reproduced the same day against 0.1.3 from PyPI in + a venv holding only `nodemaven`. + + Nothing in the suite could have caught it. `test_api.py` has a test whose + subject is "zero required dependencies" and it reads `api.py`'s source, so + it says nothing about a document. This file had 62 checks on the README and + every one of them checked what it *said*, not whether what it *showed* runs. + The gap was between two files that each looked covered. + + The scan found a second one nobody had reported: the Playwright example has + the same defect and always had. + """ + + #: Modules an example may import with no install line. `nodemaven` is the + #: package itself; the rest is the standard library, taken from the + #: interpreter on 3.10 and newer and listed by hand below that, because CI + #: runs 3.9 and `sys.stdlib_module_names` does not exist there. + FREE = frozenset({"nodemaven"}) | frozenset( + getattr(sys, "stdlib_module_names", None) + or { + "base64", "dataclasses", "inspect", "json", "os", "pathlib", "re", + "secrets", "socket", "sys", "time", "typing", "urllib", + } + ) + + @pytest.mark.parametrize("name", sorted(DOCUMENTS)) + def test_every_third_party_import_is_named_in_an_install_line( + self, name, documents: dict + ): + text = documents[name] + blocks = _blocks(text) + installs = " ".join( + body for language, body in blocks if "pip install" in body + ) + imported = _imports(blocks) + for module in sorted(imported - self.FREE): + assert re.search(rf"\b{re.escape(module)}\b", installs), ( + f"{name} shows `import {module}` and no `pip install` line in " + f"that file installs it; a reader copying the example gets " + f"ModuleNotFoundError" + ) + + def test_the_import_scan_still_finds_the_examples_it_was_written_for( + self, corpus: str + ): + """The found-something guard for the test above, in the one place it works. + + It was written per-document, as `assert imported or not blocks`, and + `docs/validation.md` failed it correctly: that file's Python blocks use + `Proxy` from the surrounding text and import nothing at all, which is + not a defect. A guard that fires on a legitimate document is worse than + no guard, because the fix under time pressure is to delete it. + + So the guard belongs over the corpus, where it can name what it expects. + `requests` and `playwright` are the two third-party modules the + documentation shows, and `requests` is the whole reason the extra + exists - if the scan stops seeing them, it is the scan that broke and + not the documentation. + """ + imported = _imports(_blocks(corpus)) + for module in ("requests", "playwright"): + assert module in imported, ( + f"the scan no longer sees `import {module}` anywhere; the fenced " + f"blocks or the import regex changed and the install-line test " + f"above is now checking nothing" + ) + + def test_the_package_declares_the_extra_the_readme_tells_people_to_install( + self, readme: str + ): + # `pip install nodemaven[requests]` is only a real instruction if the + # extra exists. pip answers an undeclared extra with a warning and + # installs the package without it, so the example would fail exactly as + # before while the README looked fixed. `pyproject.toml` is read as text + # rather than through a TOML parser, because this suite runs on 3.9 where + # `tomllib` does not exist and the package's own `tomli` fallback is an + # optional install. + extras = re.findall(r"pip install nodemaven\[([a-z0-9_,-]+)\]", readme) + assert extras, "the README stopped naming an extra, so nothing was checked" + + text = (ROOT / "pyproject.toml").read_text(encoding="utf-8") + declared = text.split("[project.optional-dependencies]", 1)[1] + declared = declared.split("\n[", 1)[0] + for group in extras: + for extra in group.split(","): + assert re.search(rf"^{re.escape(extra)}\s*=", declared, re.M), ( + f"the README says `pip install nodemaven[{extra}]` and " + f"pyproject.toml declares no such extra" + ) + + +class TestTheReadmeLinksResolveOnPyPI: + """PyPI renders this file and resolves nothing relative. + + A relative `](docs/validation.md)` resolves against + `pypi.org/project/nodemaven/` on the package page and answers 404 there, + which is CEO rule 1 - a production link that goes nowhere gets removed or + fixed. The rule was written in three HTML comments in the README and + enforced by nobody until the split created five links that could break it. + + **Both tests run against `prose`, not against the raw file, and the first + run is why.** They were written against the raw text and failed on + `](LICENSE)` and `](docs/validation.md)` - two strings that are not links at + all, but the HTML comments explaining to an editor which spelling is + forbidden. So the test caught the notes documenting the rule instead of a + breach of it. The second test is worse in the same way and would not have + announced itself: a comment naming `docs/validation.md` satisfies + `name in text`, so with raw text it would have gone on passing after the + real link was deleted. + """ + + def test_no_link_in_the_readme_is_relative(self, prose: str): + links = re.findall(r"\]\(([^)]+)\)", prose) + assert links, "no links were found, so nothing was checked" + relative = [ + target + for target in links + if not target.startswith(("#", "http://", "https://", "mailto:")) + ] + assert relative == [], ( + f"relative links in the PyPI long description, which 404 there: " + f"{relative}" + ) + + def test_the_readme_links_at_every_document_this_suite_checks( + self, prose: str + ): + # The split is only an improvement if the pieces are reachable. A doc + # nobody links to is a doc nobody reads, and it goes stale first. + # + # Matched inside a `](...)` rather than anywhere in the text, so that + # naming a file in a sentence does not stand in for linking at it. + linked = re.findall(r"\]\(([^)]+)\)", prose) + for name in DOCUMENTS: + if name == "README.md": + continue + assert any(target.endswith(name) for target in linked), ( + f"the README links at nothing named {name}" + ) + + class TestTheParameterTable: def test_every_known_parameter_has_a_row(self, readme: str): # The table is what a developer reads instead of the shipped TOML. A @@ -307,10 +558,10 @@ def test_the_ttl_values_it_names_are_the_ones_the_definition_names( class TestTheUnknownParameterMessage: - def test_the_readme_quotes_it_word_for_word(self, readme: str): + def test_the_documentation_quotes_it_word_for_word(self, corpus: str): # This is the quotation that drifted, and the test that would have caught # it. It is built from the shipped definition, so adding a parameter to - # the TOML fails here until the README is updated too. + # the TOML fails here until the documentation is updated too. # # Every block is checked, not the first one. This test used to pin # `quoting[0]` against a message raised from a typo hardcoded here, so it @@ -320,11 +571,18 @@ def test_the_readme_quotes_it_word_for_word(self, readme: str): # message in it - "Did you mean 'filter'?", which this library does not # say and never has. That is the failure this test exists to catch and it # caught it, but only because the new block landed first in the file. + # + # It runs over the corpus from 2026-09-10: the two quotations now live in + # two different files, and a scan of README.md alone would have stopped + # checking one of them without failing. from nodemaven import ParamError - blocks = re.findall(r"```[a-z]*\n(.*?)```", readme, re.DOTALL) + blocks = re.findall(r"```[a-z]*\n(.*?)```", corpus, re.DOTALL) quoting = [body for body in blocks if "ParamError:" in body] - assert quoting, "the README stopped quoting the message, so nothing was checked" + assert len(quoting) >= 2, ( + "fewer ParamError quotations than there are documents that carry " + "one, so at least one stopped being checked" + ) for body in quoting: typo = re.search(r"does not know the parameter '([^']+)'", flat(body)) @@ -335,24 +593,29 @@ def test_the_readme_quotes_it_word_for_word(self, readme: str): class TestTheFoldedRegionExample: - def test_the_username_in_the_readme_is_what_the_code_builds(self, readme: str): + def test_the_username_in_the_readme_is_what_the_code_builds(self, corpus: str): # `region-district_of_columbia` is not an invention: it is the form the # dashboard itself emitted in a username for a real account, read - # 2026-09-07. The example is the one place the README shows the fold, so - # it is the one place a change to the fold has to be reflected. + # 2026-09-07. The example is the one place the fold is shown, so it is + # the one place a change to the fold has to be reflected. built = Proxy( login="u", password="p", region="District of Columbia" ).username assert built == "u-region-district_of_columbia" - assert built in readme + assert built in corpus class TestTheCheckOutput: - """The two blocks under "Asking the gateway". + """The two blocks in the quickstart. Built here rather than captured from the gateway, and that is what makes them checkable: a `Check` is a plain frozen object, so the README is quoting a real `__str__` of a real instance and not a hand-typed approximation of one. + + They stay in the README rather than moving to `docs/` with the rest of the + gateway material, because `check()` is now the first call the quickstart + makes: it needs nothing but the package, so it is the one first run that + cannot fail on a missing dependency. """ def test_the_200_line(self, readme: str): @@ -417,40 +680,41 @@ def test_the_documented_timeout_is_the_default(self, readme: str): class TestTheReferenceMatchesTheCode: - """The Reference section is a contract, so it gets the same treatment as a + """`docs/api-reference.md` is a contract, so it gets the same treatment as a quoted output: it is compared against the real thing rather than read. - Added with the section itself, 2026-09-08. The section exists because an - external review said the documentation was overloaded with justifications; - measuring that turned up the sharper version of the complaint - the package - exports 18 names and the README gave a signature for none of them, so for - several calls the rationale was the only coverage there was. A reference - written once and never checked would have been a worse answer than no - reference, because it reads as authoritative. + Added with the section itself, 2026-09-08, when it was the second half of + the README; it became its own file on 2026-09-10. The section exists because + an external review said the documentation was overloaded with + justifications; measuring that turned up the sharper version of the + complaint - the package exports 18 names and the README gave a signature for + none of them, so for several calls the rationale was the only coverage there + was. A reference written once and never checked would have been a worse + answer than no reference, because it reads as authoritative. """ - def test_the_proxy_signature_names_every_real_argument(self, readme: str): + def test_the_proxy_signature_names_every_real_argument(self, reference: str): import inspect - block = readme.split("### `Proxy`", 1)[1].split("```", 2)[1] + block = reference.split("## `Proxy`", 1)[1].split("```", 2)[1] for name in inspect.signature(Proxy.__init__).parameters: if name == "self": continue assert name in block, name - def test_the_client_signature_names_every_real_argument(self, readme: str): + def test_the_client_signature_names_every_real_argument(self, reference: str): import inspect from nodemaven import Client - block = readme.split("### `Client` and `Page`", 1)[1].split("```", 2)[1] + block = reference.split("## `Client` and `Page`", 1)[1].split("```", 2)[1] for name in inspect.signature(Client.__init__).parameters: if name == "self": continue assert name in block, name - def test_every_public_proxy_call_is_in_the_reference(self, readme: str): - section = readme.split("### `Proxy`", 1)[1].split("### `Check`", 1)[0] + def test_every_public_proxy_call_is_in_the_reference(self, reference: str): + section = reference.split("## `Proxy`", 1)[1].split("## `Check`", 1)[0] missing = [ name for name in dir(Proxy) @@ -458,11 +722,11 @@ def test_every_public_proxy_call_is_in_the_reference(self, readme: str): ] assert missing == [], f"public on Proxy and undocumented: {missing}" - def test_every_public_client_call_is_in_the_reference(self, readme: str): + def test_every_public_client_call_is_in_the_reference(self, reference: str): from nodemaven import Client - section = readme.split("### `Client` and `Page`", 1)[1] - section = section.split("### `Provider`", 1)[0] + section = reference.split("## `Client` and `Page`", 1)[1] + section = section.split("## `Provider`", 1)[0] missing = [ name for name in dir(Client) @@ -470,7 +734,7 @@ def test_every_public_client_call_is_in_the_reference(self, readme: str): ] assert missing == [], f"public on Client and undocumented: {missing}" - def test_every_exported_name_appears_in_the_readme(self, readme: str): + def test_every_exported_name_appears_in_the_documentation(self, corpus: str): # The gap that produced this test: `available()` was exported and # appeared nowhere in 625 lines, so the only way to find it was to read # `__init__.py`. @@ -479,24 +743,24 @@ def test_every_exported_name_appears_in_the_readme(self, readme: str): missing = [ name for name in nodemaven.__all__ - if not name.startswith("_") and f"`{name}" not in readme + if not name.startswith("_") and f"`{name}" not in corpus ] assert missing == [], f"exported and unmentioned: {missing}" class TestTheAttributesTheReadmePromises: def test_every_field_a_check_carries_is_in_the_reference_table( - self, readme: str + self, reference: str ): """Derived from the dataclass, not from a list written here. This test used to carry the seven names by hand and look for - ``result.`` anywhere in the file, which matched a code block under - "Asking the gateway" that said the same thing as the reference table. - Two copies of one contract with the test pinning one of them is how the - other drifts - it is the failure this whole file was written after. The - block is gone, the table is the one copy, and the names now come from - the type so that adding a field fails here until it is documented. + ``result.`` anywhere in the file, which matched a code block that + said the same thing as the reference table. Two copies of one contract + with the test pinning one of them is how the other drifts - it is the + failure this whole file was written after. The block is gone, the table + is the one copy, and the names now come from the type so that adding a + field fails here until it is documented. """ import dataclasses @@ -504,7 +768,7 @@ def test_every_field_a_check_carries_is_in_the_reference_table( # `ok` is a property rather than a field, and is the one a caller reads # first, so it is named explicitly rather than left to the derivation. names.append("ok") - table = readme.split("### `Check`", 1)[1].split("###", 1)[0] + table = reference.split("## `Check`", 1)[1].split("\n## ", 1)[0] missing = [name for name in names if f"`.{name}`" not in table] assert missing == [], f"a Check field the reference does not list: {missing}" @@ -512,7 +776,7 @@ def test_every_field_a_check_carries_is_in_the_reference_table( "attribute", ["ok", "status", "reason", "exit_ip", "elapsed", "headers", "meaning"], ) - def test_each_one_exists_on_check(self, attribute, readme: str): + def test_each_one_exists_on_check(self, attribute, reference: str): result = Check( status=200, reason="OK", server="h:1", elapsed=0.0, headers={} ) @@ -547,11 +811,11 @@ def test_each_one_exists_on_check(self, attribute, readme: str): "validate", ], ) - def test_every_client_method_the_readme_shows_exists(self, method, readme: str): + def test_every_client_method_the_readme_shows_exists(self, method, corpus: str): from nodemaven import Client assert hasattr(Client, method) - assert f"client.{method}(" in readme or f"`{method}()`" in readme + assert f"client.{method}(" in corpus or f"`{method}()`" in corpus class TestTheEnvironmentVariableNames: