feat: Add validate_response option

[vgavro]

Adding Retry.validate_response callback to Retry configuration.
Inspired by pydantic_ai.retries approach https://github.com/pydantic/pydantic-ai/blob/main/pydantic_ai_slim/pydantic_ai/retries.py

This should effectively solve two cases:

1. Timeouts on reading response body, see Retries on response.read and response.aread can't be handled inside RetryTransport #29 - this shouldn't be default, but we're allowing user to do:

async validate_response_read(response: httpx.Response):
    await response.aread()
    
retry = Retry(validate_response=validate_response_read)

While ReadTimeout is already in default Retry.retry_on_exceptions, it will work as expected. httpx will not re-read body on second .read() invocation, body is cached once readed anyway.

2. For cases with blocks in http body (or with redirect to authorization/captcha page) and access with rotating proxies, when content access is blocked but with 200 status code.

class ContentBlocked(ValueError):
  ...
  
async def validate_response(response: httpx.Response):
    response.raise_on_status()
    await response.aread()
    if "content blocked" in response.text:
        raise ContentBlocked()

retry = Retry(validate_response=validate_response, retry_on_exceptions=[httpx.HTTPError, ContentBlocked])

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 100.00%. Comparing base (999e2aa) to head (61a9f9c).

Additional details and impacted files

@@            Coverage Diff             @@
##              main       #74    +/-   ##
==========================================
  Coverage   100.00%   100.00%            
==========================================
  Files            6         6            
  Lines         1001      1132   +131     
==========================================
+ Hits          1001      1132   +131     

Files with missing lines	Coverage Δ
httpx_retries/retry.py	100.00% <100.00%> (ø)
httpx_retries/transport.py	100.00% <100.00%> (ø)
tests/test_transport.py	100.00% <100.00%> (ø)

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[will-ockmore]

Thanks for this @vgavro . It's a well thought out addition.

I'd like to take validate_response, but as a response validation hook, not the fix for #29. Your second example is a real problem that we should aim to solve, and validate_response gives users the tool to solve it. The implementation is

Why I don't think this is the right fix in general for #29 (and I'll give more detail there; apologies for my slow response on that thread, I've been moving countries) is twofold:

1. At the general level: it breaks the layer boundary. Setting the request attribute is a symptom of this, but the transport should not be handling these aspects outside its remit. The first example would break streaming; the whole body would be read into memory, and this would not be clear to the user. This is another symptom of the same issue.
2. It requires users to understand the library internals to fix something that should be solved by default. They should not need to know the incantation to solve body-phase retries, it should be transparent to them.

So to get this PR ready to merge:

• Drop the faq.md edit that presents this as the solution to body-phase retries.
• Document it for content validation; mention that calling .read() inside it will buffer stream=True responses, so it shouldn't be used that way.
• Consider moving the async/sync-transport-mismatch check to RetryTransport.init so it fails fast

[will-ockmore]

this is the only slightly iffy operation. We should not be changing the behaviour of httpx in general. This is probably fine here.

[vgavro]

Thanks for response!

1,2. I totally get your concerns about streaming, still i don't actually see better compromise on this, but your concerns are totally make sense. Anyway - having this feature for this purpose even in validate_response is good enough personally for me. I'll do documentation changes on this.

--
3. About moving check to __init__

as we're inheriting both SyncTransport/AsyncTransport (which is great by the way) - we don't know in __init__ if it's used in sync or async. Same way we don't know where exactly Retry object will be used to validate it there. Implementing AsyncRetry/Retry is totally overhead, of course.

I can move this check to __enter__ then (and AsyncTransport is using __aenter__ for opening instead), but __enter__/__aenter__ is triggered only if client is used in context, not on resp = Client().get("hello") for example, so this validation still be not reliable enough.

In [2]: class MyTransport(httpx.BaseTransport):
   ...:     def __enter__(self):
   ...:         print("__enter__ triggered")
   ...:     def handle_request(self, request):
   ...:         print("request triggered")
   ...: 

In [3]: httpx.Client(transport=MyTransport()).get("hello")
request triggered

In [5]: with httpx.Client(transport=MyTransport()) as client:
   ...:     client.get("hello")
   ...: 
__enter__ triggered
request triggered

Also - if Retry can be passed with request.extensions (in #75) we still should validate it per request.

So, i think it's nothing we can do here, what do you think?

[vgavro]

plz review, thank you!

• Drop the faq.md edit that presents this as the solution to body-phase retries.
• Document it for content validation; mention that calling .read() inside it will buffer stream=True responses, so it shouldn't be used that way.
• Added explanaiton above Consider moving the async/sync-transport-mismatch check to RetryTransport.init so it fails fast

[will-ockmore]

This is looking great @vgavro . Agree on your take with the init limitations. I'll merge this, thanks!