Conversation
fix image checking
deleted fixed doabs
simplify sitemap, remove expensive cover_image query
5% improvement
Maintenance 2024
fix downloads, add 3 cmp providers
Maintenance 2025
update MetadataReader for DOAB OAI feed
fix bugs in handle doab oai
oapen, downloads
implement turnstile on the search form, shorten session life
…sal log Round-2's fix dropped the recipient list from the log/exception text but added message.subject as a "safe" identifier instead -- Codex round 3 correctly pointed out a subject can itself carry PII (e.g. "Password reset for real@example.com"), recreating the exact exposure the subject/body split (two commits back) exists to avoid. Dropped the subject too; the log line is now just a bare recipient count. Added test_refusal_log_line_carries_no_pii, which uses assertLogs to capture the actual emitted log record (not just the exception text) and asserts neither the recipient address nor a PII-bearing subject appear in it. Not fixed, deliberately: round 3 also suggested _is_allowed() should do full RFC 5322 mailbox validation (parseaddr() accepts some oddly-formed-but-still-matching values, e.g. a quoted local-part or a trailing comma, that a stricter parser would reject). No concrete example was found where this lets a genuinely non-allowlisted address through -- every example given was an allowed address in an unusual form. Given three rounds have now closed every demonstrated bypass, treating this as a documented stopping point rather than open-ended hardening of a backstop feature. Verified: `manage.py test regluit.utils` -- 24 tests, all pass. `manage.py test libraryauth frontend utils` -- 74 tests, still the same 3 pre-existing failures + 2 pre-existing errors as origin/master, zero net-new. 🤖 Generated with Claude Code Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015uv8CgcRqX4qjx9C74bVHj
Add opt-in email allowlist/redirect backend for non-prod (backstop for #1238)
Password managers can auto-submit the login form right after autofill. When the user then clicks "Sign in with Password" themselves, the second POST carries the pre-login CSRF token (Django rotates the CSRF cookie on successful login), so the user lands on a bare 403 even though the first POST already logged them in. Confirmed live on prod 2026-08-31: an isTrusted:false synthetic click submitted the form 1.2s before the user's real click; every 302 login that day was paired with a 403 one second later. The guard is a document-level delegated submit listener in sitewide1.js, scoped to forms posting to /accounts/superlogin/: the first submission marks the form, later ones are prevented while it is in flight. It must live in sitewide JS rather than the form template because the sign-in lightbox is injected via jQuery .load(url + " #lightbox_content"), which strips inline <script> tags. Smoke tests pin the wiring: the login page loads sitewide1.js and the form still posts to the guarded action. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EqzfARuryr6m6jzGscgAY6
…behavioral tests - Blocker: the 8s timed unlock could itself recreate the 403 (unlocking while a slow login navigation is still pending lets a resubmission carry the stale token). Removed; the lock now holds until navigation or bfcache restore. - Lock is module-global instead of per form node: standalone page + lightbox copies (or a reopened lightbox) are all locked by one login in flight. - Split into capture-phase blocker + bubble-phase acquirer, so a submission some other handler cancels never acquires the lock, and a locked submission is stopped before other handlers run. - Action matched by resolved pathname (URL()), not substring. - pageshow handler: a bfcache-restored page still holds the lock and a pre-login CSRF token, so reload it for fresh state. - Real behavioral coverage: dependency-free Node test with DOM fakes (static/js/tests/login_guard_test.js, 9 cases) extracting the guard between source markers; run from the Django suite when node exists. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EqzfARuryr6m6jzGscgAY6
… rollback - Acquire the lock in the document CAPTURE listener (runs before any target/ancestor handler can stopPropagation the event away), with a deferred post-dispatch check that releases it if a later handler cancelled the submission -- fixes both the stuck-lock-on-late-cancel and the stopPropagation acquisition bypass. - Blocked submissions get stopImmediatePropagation() so downstream submit handlers cannot run side effects for them, and the blocked form's own button is disabled immediately (visible stuck-state even for a freshly reopened lightbox form). - isLoginForm now also requires same-origin, not just pathname. - Test harness models real propagation (doc capture -> target -> doc bubble, honoring preventDefault/stopPropagation/stopImmediate- Propagation, default action decided post-dispatch); 14 cases including one per round-2 finding. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EqzfARuryr6m6jzGscgAY6
…tter-aware - Finding 1 (deferred defaultPrevented cannot prove "no POST" -- a handler can cancel-and-replace via form.submit(), and the canceled flag can mutate after the default-action decision): adopted Codex's recommended fix for this codebase -- the lock is now FAIL-CLOSED, held until navigation or bfcache restore, with no timer rollback. A synchronous defaultPrevented check still skips acquisition for events cancelled before the guard runs. - Finding 2 (disconnected form aborts submission, sticking the lock): subsumed by fail-closed; documented as an accepted limitation (no such handler exists; a stuck form is visible and reload fixes it). - Finding 3 (window-capture listener bypass): guard moved from document capture to WINDOW capture, the earliest propagation point. - Finding 4 (stopPropagation fallback weaker than claimed): fallback dropped; stopImmediatePropagation is universal in supported browsers. - Finding 5 (submitter/formaction): event.submitter preferred for the button to disable, querySelector fallback kept for implicit Enter-key submission. - Harness models window capture -> doc capture -> target -> doc bubble -> window bubble, plus pre-guard window-capture listeners; 16 cases. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EqzfARuryr6m6jzGscgAY6
Guard login form against password-manager double-submit (403 CSRF on login) (#1240)
) Pay.__init__ used token-presence alone to decide "user is anonymous" and dropped transaction.user when calling make_account, even though a logged-in donor entering fresh card details always carries a token (no Account exists yet). Every such donation created a Stripe Customer mislabeled "anonymous user" with no Account linked back to the donor's profile (so saved-card reuse never worked either). Now passes user=transaction.user through (None for genuinely anonymous transactions, so that path is unchanged) -- make_account's existing if-user branch already does the right thing once given one. Reproduced and verified with the repo's mocked-StripeClient test pattern (no live Stripe calls): confirmed the new test_logged_in_donation_creates_identified_customer test fails against the pre-fix code and passes after; the anonymous-donation case is covered by a paired test and stays unchanged. payment+frontend suites hold at the pre-existing 3-failure/1-error/2-skipped baseline (re-verified against unmodified master), zero net-new failures. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MHjXjrfha4X6TTSzY5obUZ
Codex round-1 confirmed the fix is correct and the tests genuinely catch #1125, and flagged one out-of-scope finding: Pay.__init__ hands Execute the just-created account_id via transaction.preapproval_key, but Execute prefers a fresh transaction.user.profile.account DB lookup over it -- a narrow race (two concurrent submissions for the same user) could charge a different, concurrently-created Customer than the one this call just created/labeled. Filed as #1249 rather than folded into this narrowly-scoped fix: it's a pre-existing Execute account-resolution characteristic shared with delayed/queued campaign executions, not something #1125's Pay.__init__ change introduces. Took Codex's cheap, in-scope coverage suggestions instead: test_logged_in_donation_creates_identified_customer now uses a token-shaped (not raw-dict) value, asserts create_customer got card=<token>, asserts create_charge was called with customer=<the newly-created Customer's id> (so a post-Customer-creation failure wouldn't go unnoticed), and asserts the transaction actually reaches TRANSACTION_STATUS_COMPLETE with the right pay_key. payment+frontend suites still hold at the pre-existing 3F/1E/2S baseline. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MHjXjrfha4X6TTSzY5obUZ
Codex round-2 verdict: ship, no blocking issues -- confirmed the concurrency race deferral to #1249 is the right scope call, confirmed the fix is correct, confirmed the hardened test genuinely detects the regression (reran against pre-fix stripelib.py, failed as expected). Only optional polish suggested: assert_called_once() on the mocked create_customer/create_charge calls. Applied. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MHjXjrfha4X6TTSzY5obUZ
The AI-crawler rules added in the first commit had a precedence bug. A crawler obeys only the single most specific User-agent group that matches it and ignores "User-agent: *" entirely (RFC 9309). The ClaudeBot group carried only Crawl-delay, so declaring it would have *removed* ClaudeBot's existing Disallow rules for /accounts/, /feedback/, /socialauth/, /search/ and /googlebooks/ -- widening access to the faceted-search space whose URL explosion #1116 was cut to fix, rather than narrowing it. Every named group that is not a blanket "Disallow: /" now restates the baseline rules. ClaudeBot additionally excludes the listing and feed endpoints identified in #1189 as the expensive bot-hit paths (/free/, /bypub/, /pid/, /unglued/, /campaigns/, /api/) while leaving /work/ pages crawlable, so the throttle sheds load without costing discoverability. Verified Anthropic's published crawler documentation rather than relying on the earlier claim: ClaudeBot does support the non-standard Crawl-delay extension, so throttling it is meaningful. GPTBot and the other training crawlers document no such support and stay fully disallowed. Added Google-Extended and Applebot-Extended, which are AI-training opt-out tokens and carry no search-ranking effect. Googlebot and user-triggered agents (Claude-User, ChatGPT-User, OAI-SearchBot, Claude-SearchBot) remain unrestricted. Removed the inline "DECISION FOR REVIEW" comments now that the decisions are made, and dropped the traffic statistics from the served file -- this template is public output, so it carries policy, not operational detail. Tests: adds RobotsTxtTests (SimpleTestCase, no database) covering the production and non-production host branches, and a regression guard asserting that no named group omits the baseline rules. The guard was confirmed to fail against the previous version of this template and pass against this one. Full frontend suite: 40 tests, 3 failures, all three pre-existing on the branch without this change (RhPageTests, unrelated). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ANtyDhUBYCsvEytDfMXUJe
Codex round 1 caught a real error I carried over from the original draft without checking it: PerplexityBot was grouped with the training crawlers and given Disallow: /, which contradicts this PR's own stated policy of keeping AI *search* discovery enabled. Verified against Perplexity's published crawler documentation rather than assuming either way: PerplexityBot is their search-indexing crawler and is explicitly "not used to crawl content for AI foundation models"; the on-demand fetcher is Perplexity-User. Blocking it would have removed the collection from Perplexity search results while shedding no training load at all -- pure cost, no benefit. Removed the group so it falls through to User-agent: *. Checked Amazonbot the same way while here, since it was inherited from the same list: Amazon documents it as crawling that "may be used to train Amazon AI models," so it stays disallowed, correctly. Their separate Amzn-SearchBot does not crawl for generative training and already falls through untouched. Tests: extended the absence assertions from four agents to eight, so a future edit cannot quietly give a search or user-triggered agent its own group -- PerplexityBot, Perplexity-User, Amzn-SearchBot and Claude-SearchBot are now covered alongside the originals. This is the assertion that would have caught the PerplexityBot mistake. RobotsTxtTests: 3 tests, all passing. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ANtyDhUBYCsvEytDfMXUJe
…C wording Three findings, all fair. Taken in order of consequence: 1. Removed Google-Extended and Applebot-Extended. Codex flagged that Disallow: / on Google-Extended also costs Gemini grounding, not only training. Following that through: neither token is a crawler at all -- Googlebot and Applebot do the fetching, and these are content-*usage* opt-outs. So they shed exactly zero crawl load while carrying a real AI-answer discoverability cost. That is the same "pure cost, no benefit" shape as the PerplexityBot entry removed in the previous commit, and it is a content-policy decision rather than a load question, so it does not belong in a load-shedding change. Dropped from this PR; worth deciding deliberately on its own terms. 2. Tests now assert the rules this PR actually exists for. The previous version checked Crawl-delay but never ClaudeBot's six expensive-path exclusions, and covered only three of the blocked agents -- deleting /api/ or the Bytespider stanza would have passed. Added CLAUDEBOT_EXTRA_DISALLOWS and BLOCKED_AGENTS constants, asserted every entry, and added an exhaustive equality check on the set of groups so a stanza cannot be dropped or slipped in unnoticed. Verified by deleting /api/ and Bytespider and confirming the suite fails. 3. Corrected the RFC 9309 explanation in the template comment and the test docstring. Multiple matching groups are combined rather than one winning; the accurate and load-bearing point is that "User-agent: *" applies only to crawlers matching no named group. Behavior was already correct -- the explanation could have misled a future edit. RobotsTxtTests: 3 tests, all passing. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ANtyDhUBYCsvEytDfMXUJe
Four findings, all verified against primary sources before acting rather
than taken on the reviewer's word.
1. The tests were not actually database-free, and the docstring said they
were. Issuing the client request loads the root URLconf, which imports a
module that runs a query at import time. Under SimpleTestCase Django
creates no test database, so that query hit whatever database the
settings pointed at -- meaning the "isolated" test could read live
application data. Switched to TestCase so Django builds an isolated test
database, and removed the no-database claim. Kept the real URLconf rather
than substituting a minimal one, so the test still proves /robots.txt is
actually routed.
2. CCBot was blocked under a stated rationale that is false. Common Crawl
documents that CCBot honors Crawl-delay ("By increasing that number, you
will indicate to CCBot to slow down"), so the comment claiming these
crawlers offer nothing to throttle was wrong for it. Behavior is
unchanged -- CCBot is a real crawler and blocking it does shed load --
but the comment now says plainly that this is a deliberate opt-out from
the Common Crawl dataset rather than a technical necessity, and notes it
could be softened to a throttle. That is a content-policy call worth
making deliberately.
3. Removed cohere-ai. Cohere's own crawler documentation states they do not
operate crawlers for training generative models "at this time" and names
no active user agent; Coherebot appears only as a hypothetical example.
The token sheds no load and, pinned into an exhaustive test, implied a
protection that does not exist.
4. Fixed the test parser's group boundaries. It treated any non-User-agent
field as ending a run of user-agent lines, so an extension record such as
Crawl-delay between two consecutive User-agent lines would have split one
group into two -- and the precedence guard would then silently stop
covering the second agent. Now only Allow/Disallow close the run, per RFC
9309 2.2.4. Added a fixture covering exactly that shape.
RobotsTxtTests: 4 tests, all passing. Full frontend suite: 41 tests, 3
failures, the same three pre-existing RhPageTests failures unrelated to
this change.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ANtyDhUBYCsvEytDfMXUJe
Diffbot documents Crawl-delay support too, so the comment naming CCBot as the sole exception was wrong again -- the fourth round in a row to catch a factual error in a per-crawler capability claim. Rather than patch in "and Diffbot" and wait for the next round to find another, removed the class of claim entirely. The comment now states the policy (this is a content-usage decision to opt out of training-data collection), notes plainly that several of these crawlers do honor Crawl-delay and could be throttled instead, and tells the next editor to check the operator's current documentation before adding or moving an entry. That is accurate regardless of which agents support what, and it cannot rot the way the enumerated version kept doing. No behavior change: the same agents are disallowed. Same correction applied to the BLOCKED_AGENTS comment in the tests. RobotsTxtTests: 4 tests, all passing. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ANtyDhUBYCsvEytDfMXUJe
Round 5 returned "no blocking defects found" with three P3s. Took two, declined one. Taken: 1. The precedence guard checked only that named groups restate the baseline Disallow rules, and discarded Allow rules entirely. RFC 9309 gives the longest match precedence, so a future "Allow: /accounts/public/" would re-open a path under the shorter "Disallow: /accounts/" while the guard still passed. The parser now records Allow rules and the guard asserts no named group allows anything beneath a baseline prefix. Verified by injecting exactly that rule and confirming the suite fails. 2. Corrected two comments that claimed more than is true. "Every training crawler is blocked" was wrong -- ClaudeBot trains and is deliberately throttled instead; it now reads "every crawler selected for blocking" and says why. The template's allowed-agents note implied those publishers collect no training data at all, which glosses over the Google-Extended grounding tradeoff; it now scopes the claim to crawling and states that training opt-outs are a separate content-usage decision not made here. Declined, and flagged for a separate change instead: 3. Host matching. Because the template compares raw HTTP_HOST, "unglue.it:443" and "Unglue.it" fall to the non-production branch and would serve "Disallow: /". That is real, but it is pre-existing behavior of the prod/non-prod gate rather than anything this PR introduces, it is orthogonal to AI crawlers, and getting it wrong in either direction is costly -- too strict and prod tells crawlers to go away, too loose and staging gets indexed. It deserves its own change with its own test, not a quiet edit inside a crawler-policy PR. The same applies to "www.unglue.it". RobotsTxtTests: 4 tests, all passing. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ANtyDhUBYCsvEytDfMXUJe
robots.txt: throttle ClaudeBot, disallow training crawlers, keep search agents (refs #1172)
Pass transaction.user through to make_account for donation tokens (#1125)
…s (refs #1253) meta-webindexer was the largest single source of /free/ facet-browse requests on 2026-09-09 (48,318 of 86,287), and it fell through to the permissive "User-agent: *" group. Meta documents it as the crawler behind Meta AI search results, so it is treated like the other search-indexing crawlers here: not blocked. It gets a ClaudeBot-style group that restates the baseline rules and excludes the same expensive listing and feed endpoints, so work pages stay indexable. No Crawl-delay, since Meta does not document support for it. Test: RobotsTxtTests asserts the new group's rules and adds it to the exhaustive group set. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DjPGJQCBFvZ1D2fJgJQpe4
Every /free/ facet query starts from Work.objects.filter(is_free=True), optionally narrows by language (?pub_lang=), and by default sorts newest-first by (-featured, -created). Work has separate single-column indexes on is_free, language, featured and created, but none covering filter + sort together, so MySQL intersects the is_free and language indexes, then builds and sorts a temporary table on every request. EXPLAIN on test and production showed exactly that plan; test's slow-query log caught three 1-5 second instances examining 600-1,700x more rows than returned. Adds two indexes (migration core 0033, AddIndex only, no data change): - (is_free, language, featured DESC, created DESC) for ?pub_lang= views - (is_free, featured DESC, created DESC) for views without a language Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DjPGJQCBFvZ1D2fJgJQpe4
- Test asserts meta-webindexer's exact Disallow set, an empty Allow list and no extension records, so an added Allow, an over-broad Disallow, a Crawl-delay or a duplicate stanza now fails (each verified by mutation). - robots.txt comment: "not blocked outright", "work pages remain crawlable". - Fix an older test comment that claimed blocking a search crawler sheds no crawl load; #1253 shows one can. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DjPGJQCBFvZ1D2fJgJQpe4
- Assert there is no crawl-delay record for meta-webindexer, rather than that it has no extension records at all: a legitimate Sitemap line after the stanza must not fail the test (verified: passes; Crawl-delay still fails). - Comment: exact equality catches a duplicated copy of the rules, which is what it actually guarantees. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DjPGJQCBFvZ1D2fJgJQpe4
- Replace the single AddIndex migration with 0033 (language index) and 0034 (no-language index), so a failure cannot leave one migration half applied. - Each uses SeparateDatabaseAndState: RunSQL with ALGORITHM=INPLACE LOCK=NONE (fail instead of falling back to a blocking table copy) and a 10-second session lock_wait_timeout (fail fast instead of queueing an exclusive metadata lock behind a long query), with matching state_operations so makemigrations sees no drift. Reverse SQL drops each index the same way. - Model comment now describes the intended benefit, not an established one: whether DISTINCT over the joined facet tables still forces a temporary table must be measured on real data. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DjPGJQCBFvZ1D2fJgJQpe4
- 0033 header: one index per migration prevents one index stranding the other, but MySQL DDL is not transactional, so a crash between DDL and migration recording can still desync schema and history; say to compare SHOW INDEX with showmigrations before retrying. - Model comment attributes the index-intersection plan to the captured slow-query plans rather than stating it universally. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DjPGJQCBFvZ1D2fJgJQpe4
robots.txt: keep meta-webindexer out of /free/ and other listing pages (refs #1253)
Adds Crawl-delay: 10 to the meta-webindexer group as a labeled experiment. Meta documents no Crawl-delay support, but the crawler has run at a flat ~7,200 requests/hour since 2026-09-09, so production logs will show whether it honors the line. Also records the observation that Meta's crawlers ignore the "*" group while obeying a group that names them. RobotsTxtTests now requires exactly one Crawl-delay of 10 in that group (mutation-checked: missing, 5, and duplicated all fail). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DjPGJQCBFvZ1D2fJgJQpe4
- The named-group observation is stated for meta-externalagent only, and the /feedback/ observation without "for weeks". - The experiment note says a rate change would be assessed, not proven, and records the baseline start as 2026-09-09 12:00 UTC. Comment-only change. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DjPGJQCBFvZ1D2fJgJQpe4
…l-delay robots.txt: Crawl-delay experiment for meta-webindexer (refs #1253)
core.Work: composite indexes for /free/ facet browsing (refs #1253)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.