Streamed uploads fail against local httpbin

[jkbrzt]

Against the local httpbin It fails every time and the failuire has two flavours (see bellow).

Relevant: https://github.com/kennethreitz/requests/issues/2422

test.py

import sys
import requests


def stream():
    yield b'ham'
    yield b'spam'


def test_stream():
    r = requests.post('http://httpbin.org/post', data=stream())
    assert r.json()['data'] == 'hamspam'


def test_stream_local_httpbin(httpbin):
    r = requests.post(httpbin.url + '/post', data=stream())
    assert r.json()['data'] == 'hamspam'

---

body missing:

$ py.test test.py
===================================================== test session starts ======================================================
platform darwin -- Python 3.5.1, pytest-2.9.0, py-1.4.31, pluggy-0.3.1
rootdir: /private/tmp, inifile:
plugins: cov-2.2.1, httpbin-0.2.3
collected 2 items

test.py .F

=========================================================== FAILURES ===========================================================
__________________________________________________ test_stream_local_httpbin ___________________________________________________

httpbin = <Server(<class 'pytest_httpbin.serve.Server'>, started 123145309245440)>

    def test_stream_local_httpbin(httpbin):
        r = requests.post(httpbin.url + '/post', data=stream())
>       assert r.json()['data'] == 'hamspam'
E       assert '' == 'hamspam'
E         + hamspam

test.py:17: AssertionError
----------------------------------------------------- Captured stderr call -----------------------------------------------------
127.0.0.1 - - [07/Mar/2016 13:39:07] "POST /post HTTP/1.1" 200 389
============================================== 1 failed, 1 passed in 1.94 seconds ==============================================

---

requests.exceptions.ConnectionError: [Errno 32] Broken pipe:

$ py.test test.py
============================= test session starts =============================
platform darwin -- Python 3.5.1, pytest-2.9.0, py-1.4.31, pluggy-0.3.1
rootdir: /private/tmp, inifile:
plugins: cov-2.2.1, httpbin-0.2.3
collected 2 items

test.py .F

================================== FAILURES ===================================
__________________________ test_stream_local_httpbin __________________________

httpbin = <Server(<class 'pytest_httpbin.serve.Server'>, started 123145309245440)>

    def test_stream_local_httpbin(httpbin):
>       r = requests.post(httpbin.url + '/post', data=stream())

test.py:16:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/Users/jakub/.virtualenvs/httpie/lib/python3.5/site-packages/requests/api.py:107: in post
    return request('post', url, data=data, json=json, **kwargs)
/Users/jakub/.virtualenvs/httpie/lib/python3.5/site-packages/requests/api.py:53: in request
    return session.request(method=method, url=url, **kwargs)
/Users/jakub/.virtualenvs/httpie/lib/python3.5/site-packages/requests/sessions.py:468: in request
    resp = self.send(prep, **send_kwargs)
/Users/jakub/.virtualenvs/httpie/lib/python3.5/site-packages/requests/sessions.py:576: in send
    r = adapter.send(request, **kwargs)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <requests.adapters.HTTPAdapter object at 0x103501eb8>
request = <PreparedRequest [POST]>, stream = False
timeout = <requests.packages.urllib3.util.timeout.Timeout object at 0x103521860>
verify = True, cert = None, proxies = OrderedDict()

    def send(self, request, stream=False, timeout=None, verify=True, cert=None, proxies=None):
        """Sends PreparedRequest object. Returns Response object.

            :param request: The :class:`PreparedRequest <PreparedRequest>` being sent.
            :param stream: (optional) Whether to stream the request content.
            :param timeout: (optional) How long to wait for the server to send
                data before giving up, as a float, or a :ref:`(connect timeout,
                read timeout) <timeouts>` tuple.
            :type timeout: float or tuple
            :param verify: (optional) Whether to verify SSL certificates.
            :param cert: (optional) Any user-provided SSL certificate to be trusted.
            :param proxies: (optional) The proxies dictionary to apply to the request.
            """

        conn = self.get_connection(request.url, proxies)

        self.cert_verify(conn, request.url, verify, cert)
        url = self.request_url(request, proxies)
        self.add_headers(request)

        chunked = not (request.body is None or 'Content-Length' in request.headers)

        if isinstance(timeout, tuple):
            try:
                connect, read = timeout
                timeout = TimeoutSauce(connect=connect, read=read)
            except ValueError as e:
                # this may raise a string formatting error.
                err = ("Invalid timeout {0}. Pass a (connect, read) "
                       "timeout tuple, or a single float to set "
                       "both timeouts to the same value".format(timeout))
                raise ValueError(err)
        else:
            timeout = TimeoutSauce(connect=timeout, read=timeout)

        try:
            if not chunked:
                resp = conn.urlopen(
                    method=request.method,
                    url=url,
                    body=request.body,
                    headers=request.headers,
                    redirect=False,
                    assert_same_host=False,
                    preload_content=False,
                    decode_content=False,
                    retries=self.max_retries,
                    timeout=timeout
                )

            # Send the request.
            else:
                if hasattr(conn, 'proxy_pool'):
                    conn = conn.proxy_pool

                low_conn = conn._get_conn(timeout=DEFAULT_POOL_TIMEOUT)

                try:
                    low_conn.putrequest(request.method,
                                        url,
                                        skip_accept_encoding=True)

                    for header, value in request.headers.items():
                        low_conn.putheader(header, value)

                    low_conn.endheaders()

                    for i in request.body:
                        low_conn.send(hex(len(i))[2:].encode('utf-8'))
                        low_conn.send(b'\r\n')
                        low_conn.send(i)
                        low_conn.send(b'\r\n')
                    low_conn.send(b'0\r\n\r\n')

                    # Receive the response from the server
                    try:
                        # For Python 2.7+ versions, use buffering of HTTP
                        # responses
                        r = low_conn.getresponse(buffering=True)
                    except TypeError:
                        # For compatibility with Python 2.6 versions and back
                        r = low_conn.getresponse()

                    resp = HTTPResponse.from_httplib(
                        r,
                        pool=conn,
                        connection=low_conn,
                        preload_content=False,
                        decode_content=False
                    )
                except Exception as e:
                    # If we hit any problems here, clean up the connection.
                    # Then, reraise so that we can handle the actual exception.
                    low_conn.close()
                    raise

        except (ProtocolError, socket.error) as err:
>           raise ConnectionError(err, request=request)
E           requests.exceptions.ConnectionError: [Errno 32] Broken pipe

/Users/jakub/.virtualenvs/httpie/lib/python3.5/site-packages/requests/adapters.py:426: ConnectionError
---------------------------- Captured stderr call -----------------------------
127.0.0.1 - - [07/Mar/2016 13:36:28] "POST /post HTTP/1.1" 200 389
===================== 1 failed, 1 passed in 2.10 seconds ======================

[sigmavirus24]

This probably belongs as a report against httpbin.

[kevin1024]

@jkbrzt does this work against httpbin.org?

[jkbrzt]

@kevin1024 yes it does work against httpbin.org. The test.py has two test cases:

1. test_stream runs against httpbin.org and always passes
2. test_stream_local_httpbin runs agans the local httpbin and always fails

[jkbrzt]

I'm wondering if switching to Gunicorn (#28) would solve it. Will try to play with it a bit.

[kennethreitz]

Gunicorn solves everything :)

[jkbrzt]

@kennethreitz I wish! 🦄

After some more debugging it doesn't seem to have anything to do with neither gunicorn nor pytest.

test.py:

from __future__ import print_function
import requests

def stream():
    yield b'spam'

def test_httpbin_org():
    r = requests.post('http://httpbin.org/post', data=stream())
    assert r.json()['data'] == 'spam'

def test_local_httpbin():
    r = requests.post('http://127.0.0.1:5000/post', data=stream())
    assert r.json()['data'] == 'spam'

def test_local_httpbin_gunicorn():
    r = requests.post('http://127.0.0.1:8000/post', data=stream())
    assert r.json()['data'] == 'spam'

def test_pytest_httpbin(httpbin):
    r = requests.post(httpbin.url + '/post', data=stream())
    assert r.json()['data'] == 'spam'

if __name__ == '__main__':
    for test in [test_httpbin_org, test_local_httpbin, test_local_httpbin_gunicorn]:
        print(test.__name__, end=': ')
        try:
            test()
        except AssertionError as e:
            print('fail')
        else:
            print('ok')

python -m httpbin.core &  # naked httpbin
gunicorn httpbin:app &    # httpbin behind gunicorn

$ python test.py   # outside pytest
test_httpbin_org: ok
test_local_httpbin: fail
test_local_httpbin_gunicorn: fail

$ py.test test.py  # through pytest
=============================== test session starts ===============================
platform darwin -- Python 3.5.1, pytest-2.9.0, py-1.4.31, pluggy-0.3.1
rootdir: /Users/jakub/Code/OS/pytest-httpbin, inifile:
plugins: httpbin-0.2.2, cov-2.2.1
collected 4 items

test.py .FFF

==================================== FAILURES =====================================
_______________________________ test_local_httpbin ________________________________

    def test_local_httpbin():
        r = requests.post('http://127.0.0.1:5000/post', data=stream())
>       assert r.json()['data'] == 'spam'
E       assert '' == 'spam'
E         + spam

test.py:13: AssertionError
___________________________ test_local_httpbin_gunicorn ___________________________

    def test_local_httpbin_gunicorn():
        r = requests.post('http://127.0.0.1:8000/post', data=stream())
>       assert r.json()['data'] == 'spam'
E       assert '' == 'spam'
E         + spam

test.py:17: AssertionError
_______________________________ test_pytest_httpbin _______________________________

httpbin = <Server(<class 'pytest_httpbin.serve.Server'>, started 123145309245440)>

    def test_pytest_httpbin(httpbin):
        r = requests.post(httpbin.url + '/post', data=stream())
>       assert r.json()['data'] == 'spam'
E       assert '' == 'spam'
E         + spam

test.py:21: AssertionError
------------------------------ Captured stderr call -------------------------------
127.0.0.1 - - [08/Mar/2016 18:19:27] "POST /post HTTP/1.1" 200 389
======================= 3 failed, 1 passed in 1.73 seconds ========================

[guyskk]

The test does not work against httpbin.org now, it will response 411 Length Required:

>>> import requests
>>> requests.__version__
'2.18.4'
>>> 
>>> def stream():
...     yield b'ham'
...     yield b'spam'
... 
>>> 
>>> r = requests.post('http://httpbin.org/post', data=stream())
>>> r
<Response [411]>
>>> r.reason
'Length Required'
>>> 

If test against a local gunicorn server, r.json()['data'] will be an empty string, the reason is werkzeug ignore chunked request body(werkzeug.wsgi.py:get_input_stream):

    # If the request doesn't specify a content length and there is no max
    # length set, returning the stream is potentially dangerous because it
    # could be infinite, maliciously or not. If safe_fallback is true, return
    # an empty stream for safety instead.
    if content_length is None and max_content_length is None:
        return safe_fallback and _empty_stream or stream

Sometimes it will cause requests.exceptions.ConnectionError: [Errno 32] Broken pipe, the reason is requests still write date after the server close connection, the net traffic is:

time>>>---------------------------------------------------------------------------------------------------------------->>>
requests:  send request headers                                                           send chunked body ...
httpbin:                                          send response    close connection

If add time.sleep(1) before the yield of stream function, it will always cause Broken pipe.