Skip to content

Remove 'secure' attribute from cookies in ProxyGenerator to ensure co…#581

Open
Luen wants to merge 2 commits intoscholarly-python-package:developfrom
Luen:unexpected-keyword-arg
Open

Remove 'secure' attribute from cookies in ProxyGenerator to ensure co…#581
Luen wants to merge 2 commits intoscholarly-python-package:developfrom
Luen:unexpected-keyword-arg

Conversation

@Luen
Copy link

@Luen Luen commented Feb 25, 2026

Fix cookie transfer after CAPTCHA: strip secure before httpx Cookies.set()

When Google Scholar returns a CAPTCHA, the flow uses a browser to solve it, then copies cookies from the browser (Selenium) back into the httpx session so later requests use the same session. That copy step was raising:
TypeError: Cookies.set() got an unexpected keyword argument 'secure'
After that, the code retried with a new session, so the solved CAPTCHA session was effectively lost.

In _proxy_generator.py, _handle_captcha2() does:
Wait for the user to solve the CAPTCHA in the browser.
Call self._get_webdriver().get_cookies() (Selenium), which returns dicts that include browser-specific keys such as httpOnly, expiry, sameSite, and secure.
Strip only httpOnly, expiry, and sameSite, then call self._session.cookies.set(**cookie) (httpx).
httpx’s Cookies.set() does not accept a secure keyword. Passing the Selenium cookie dict as **cookie therefore included secure and triggered the TypeError.

Before calling self._session.cookies.set(**cookie), remove the secure key from each cookie dict, in the same way as the other non-httpx fields:
In _handle_captcha2(), add: cookie.pop("secure", None) after the existing pop calls for httpOnly, expiry, and sameSite.
With this, only supported arguments are passed to Cookies.set(), the cookie transfer completes, and the session continues to work after CAPTCHA solving without retrying due to this error.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant