2023년 10월 14일 토요일

python requests timeout 의 고찰

requests 모듈을 사용하다가 timeout이 발생하여 관련해서 찾아보고 정리해봤습니다.

Timeout 의 기초

문서에서는 아래와 같이 나와있습니다.

https://requests.readthedocs.io/en/stable/user/advanced/#timeouts

보통은 timeout 을 설정하지 않고 사용하는 경우 timeout이 얼마나 발생하는지 궁금해서 문서를 찾아봤는데 정보가 애매하게 되어있습니다.

시간 초과가 없으면 코드가 몇 분 이상 중단될 수 있습니다. 

시간 정보가 구체적이지 않았습니다.

그래서 몇가지 테스트를 해봤습니다. 아래와 같이 코드를 구현해서 timeout이 발생할 수 있는 조건으로 시험해봤습니다.

import requests
import datetime
import traceback
import time


def url_connect(url, timeout=None):
    nowtime = datetime.datetime.now()
    try:
        print(f"********** trying connect {url} *************")
        if timeout != None:
            r = requests.get(url, timeout=timeout)
        else:
            r = requests.get(url)
        print(r)
    except Exception as e:
        print("end time:", datetime.datetime.now() - nowtime)
        time.sleep(1)
        print("EXCEPTION", e)
        traceback.print_exc()


if __name__ == "__main__":
    url_connect("http://1.1.1.1:9999")
    url_connect("http://192.168.0.254")
    url_connect("http://127.0.0.1")
    url_connect("http://192.168.0.254", timeout=10)


********** trying connect http://1.1.1.1:9999 *************
end time: 0:00:21.042910
EXCEPTION HTTPConnectionPool(host='1.1.1.1', port=9999): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x000001EBCD096670>: Failed to establish a new connection: [WinError 10060] 연결된 구성원으로부터 응답이 없어 연결하지 못했거나, 호스트로부터 응답이 없어 연결이 끊어졌습니다'))
Traceback (most recent call last):
  File "C:\Users\jun\Documents\GitHub\sourcecode\venv\lib\site-packages\urllib3\connection.py", line 174, in _new_conn
    conn = connection.create_connection(
  File "C:\Users\jun\Documents\GitHub\sourcecode\venv\lib\site-packages\urllib3\util\connection.py", line 95, in create_connection
    raise err
  File "C:\Users\jun\Documents\GitHub\sourcecode\venv\lib\site-packages\urllib3\util\connection.py", line 85, in create_connection
    sock.connect(sa)
TimeoutError: [WinError 10060] 연결된 구성원으로부터 응답이 없어 연결하지 못했거나, 호스트로부터 응답이 없어 연결이 끊어졌습니다

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\jun\Documents\GitHub\sourcecode\venv\lib\site-packages\urllib3\connectionpool.py", line 703, in urlopen
    httplib_response = self._make_request(
  File "C:\Users\jun\Documents\GitHub\sourcecode\venv\lib\site-packages\urllib3\connectionpool.py", line 398, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "C:\Users\jun\Documents\GitHub\sourcecode\venv\lib\site-packages\urllib3\connection.py", line 239, in request
    super(HTTPConnection, self).request(method, url, body=body, headers=headers)
  File "C:\Users\jun\AppData\Local\Programs\Python\Python38\lib\http\client.py", line 1252, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "C:\Users\jun\AppData\Local\Programs\Python\Python38\lib\http\client.py", line 1298, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "C:\Users\jun\AppData\Local\Programs\Python\Python38\lib\http\client.py", line 1247, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "C:\Users\jun\AppData\Local\Programs\Python\Python38\lib\http\client.py", line 1007, in _send_output
    self.send(msg)
  File "C:\Users\jun\AppData\Local\Programs\Python\Python38\lib\http\client.py", line 947, in send
    self.connect()
  File "C:\Users\jun\Documents\GitHub\sourcecode\venv\lib\site-packages\urllib3\connection.py", line 205, in connect
    conn = self._new_conn()
  File "C:\Users\jun\Documents\GitHub\sourcecode\venv\lib\site-packages\urllib3\connection.py", line 186, in _new_conn
    raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x000001EBCD096670>: Failed to establish a new connection: [WinError 10060] 연결된 구성원으로부터 응답이 없어 연결하지 못했거나, 호스트로부터 응답이 없어 연결이 끊어졌습니다

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\jun\Documents\GitHub\sourcecode\venv\lib\site-packages\requests\adapters.py", line 489, in send
    resp = conn.urlopen(
  File "C:\Users\jun\Documents\GitHub\sourcecode\venv\lib\site-packages\urllib3\connectionpool.py", line 787, in urlopen
    retries = retries.increment(
  File "C:\Users\jun\Documents\GitHub\sourcecode\venv\lib\site-packages\urllib3\util\retry.py", line 592, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='1.1.1.1', port=9999): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x000001EBCD096670>: Failed to establish a new connection: [WinError 10060] 연결된 구성원으로부터 응답이 없어 연결하지 못했거나, 호스트로부터 응답이 없어 연결이 끊어졌습니다'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:/Users/jun/Documents/GitHub/sourcecode/python/example/_52_requests/timeout.py", line 14, in url_connect
    r = requests.get(url)
  File "C:\Users\jun\Documents\GitHub\sourcecode\venv\lib\site-packages\requests\api.py", line 73, in get
    return request("get", url, params=params, **kwargs)
  File "C:\Users\jun\Documents\GitHub\sourcecode\venv\lib\site-packages\requests\api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
  File "C:\Users\jun\Documents\GitHub\sourcecode\venv\lib\site-packages\requests\sessions.py", line 587, in request
    resp = self.send(prep, **send_kwargs)
  File "C:\Users\jun\Documents\GitHub\sourcecode\venv\lib\site-packages\requests\sessions.py", line 701, in send
    r = adapter.send(request, **kwargs)
  File "C:\Users\jun\Documents\GitHub\sourcecode\venv\lib\site-packages\requests\adapters.py", line 565, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='1.1.1.1', port=9999): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x000001EBCD096670>: Failed to establish a new connection: [WinError 10060] 연결된 구성원으로부터 응답이 없어 연결하지 못했거나, 호스트로부터 응답이 없어 연결이 끊어졌습니다'))
********** trying connect http://192.168.0.254 *************
end time: 0:00:21.032674
EXCEPTION HTTPConnectionPool(host='192.168.0.254', port=80): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x000001EBCCC4A8B0>: Failed to establish a new connection: [WinError 10060] 연결된 구성원으로부터 응답이 없어 연결하지 못했거나, 호스트로부터 응답이 없어 연결이 끊어졌습니다'))
Traceback (most recent call last):
  File "C:\Users\jun\Documents\GitHub\sourcecode\venv\lib\site-packages\urllib3\connection.py", line 174, in _new_conn
    conn = connection.create_connection(
  File "C:\Users\jun\Documents\GitHub\sourcecode\venv\lib\site-packages\urllib3\util\connection.py", line 95, in create_connection
    raise err
  File "C:\Users\jun\Documents\GitHub\sourcecode\venv\lib\site-packages\urllib3\util\connection.py", line 85, in create_connection
    sock.connect(sa)
TimeoutError: [WinError 10060] 연결된 구성원으로부터 응답이 없어 연결하지 못했거나, 호스트로부터 응답이 없어 연결이 끊어졌습니다

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\jun\Documents\GitHub\sourcecode\venv\lib\site-packages\urllib3\connectionpool.py", line 703, in urlopen
    httplib_response = self._make_request(
  File "C:\Users\jun\Documents\GitHub\sourcecode\venv\lib\site-packages\urllib3\connectionpool.py", line 398, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "C:\Users\jun\Documents\GitHub\sourcecode\venv\lib\site-packages\urllib3\connection.py", line 239, in request
    super(HTTPConnection, self).request(method, url, body=body, headers=headers)
  File "C:\Users\jun\AppData\Local\Programs\Python\Python38\lib\http\client.py", line 1252, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "C:\Users\jun\AppData\Local\Programs\Python\Python38\lib\http\client.py", line 1298, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "C:\Users\jun\AppData\Local\Programs\Python\Python38\lib\http\client.py", line 1247, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "C:\Users\jun\AppData\Local\Programs\Python\Python38\lib\http\client.py", line 1007, in _send_output
    self.send(msg)
  File "C:\Users\jun\AppData\Local\Programs\Python\Python38\lib\http\client.py", line 947, in send
    self.connect()
  File "C:\Users\jun\Documents\GitHub\sourcecode\venv\lib\site-packages\urllib3\connection.py", line 205, in connect
    conn = self._new_conn()
  File "C:\Users\jun\Documents\GitHub\sourcecode\venv\lib\site-packages\urllib3\connection.py", line 186, in _new_conn
    raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x000001EBCCC4A8B0>: Failed to establish a new connection: [WinError 10060] 연결된 구성원으로부터 응답이 없어 연결하지 못했거나, 호스트로부터 응답이 없어 연결이 끊어졌습니다

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\jun\Documents\GitHub\sourcecode\venv\lib\site-packages\requests\adapters.py", line 489, in send
    resp = conn.urlopen(
  File "C:\Users\jun\Documents\GitHub\sourcecode\venv\lib\site-packages\urllib3\connectionpool.py", line 787, in urlopen
    retries = retries.increment(
  File "C:\Users\jun\Documents\GitHub\sourcecode\venv\lib\site-packages\urllib3\util\retry.py", line 592, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='192.168.0.254', port=80): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x000001EBCCC4A8B0>: Failed to establish a new connection: [WinError 10060] 연결된 구성원으로부터 응답이 없어 연결하지 못했거나, 호스트로부터 응답이 없어 연결이 끊어졌습니다'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:/Users/jun/Documents/GitHub/sourcecode/python/example/_52_requests/timeout.py", line 14, in url_connect
    r = requests.get(url)
  File "C:\Users\jun\Documents\GitHub\sourcecode\venv\lib\site-packages\requests\api.py", line 73, in get
    return request("get", url, params=params, **kwargs)
  File "C:\Users\jun\Documents\GitHub\sourcecode\venv\lib\site-packages\requests\api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
  File "C:\Users\jun\Documents\GitHub\sourcecode\venv\lib\site-packages\requests\sessions.py", line 587, in request
    resp = self.send(prep, **send_kwargs)
  File "C:\Users\jun\Documents\GitHub\sourcecode\venv\lib\site-packages\requests\sessions.py", line 701, in send
    r = adapter.send(request, **kwargs)
  File "C:\Users\jun\Documents\GitHub\sourcecode\venv\lib\site-packages\requests\adapters.py", line 565, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='192.168.0.254', port=80): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x000001EBCCC4A8B0>: Failed to establish a new connection: [WinError 10060] 연결된 구성원으로부터 응답이 없어 연결하지 못했거나, 호스트로부터 응답이 없어 연결이 끊어졌습니다'))
********** trying connect http://127.0.0.1 *************
end time: 0:00:02.053697
EXCEPTION HTTPConnectionPool(host='127.0.0.1', port=80): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x000001EBCD1310D0>: Failed to establish a new connection: [WinError 10061] 대상 컴퓨터에서 연결을 거부했으므로 연결하지 못했습니다'))
Traceback (most recent call last):
  File "C:\Users\jun\Documents\GitHub\sourcecode\venv\lib\site-packages\urllib3\connection.py", line 174, in _new_conn
    conn = connection.create_connection(
  File "C:\Users\jun\Documents\GitHub\sourcecode\venv\lib\site-packages\urllib3\util\connection.py", line 95, in create_connection
    raise err
  File "C:\Users\jun\Documents\GitHub\sourcecode\venv\lib\site-packages\urllib3\util\connection.py", line 85, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [WinError 10061] 대상 컴퓨터에서 연결을 거부했으므로 연결하지 못했습니다

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\jun\Documents\GitHub\sourcecode\venv\lib\site-packages\urllib3\connectionpool.py", line 703, in urlopen
    httplib_response = self._make_request(
  File "C:\Users\jun\Documents\GitHub\sourcecode\venv\lib\site-packages\urllib3\connectionpool.py", line 398, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "C:\Users\jun\Documents\GitHub\sourcecode\venv\lib\site-packages\urllib3\connection.py", line 239, in request
    super(HTTPConnection, self).request(method, url, body=body, headers=headers)
  File "C:\Users\jun\AppData\Local\Programs\Python\Python38\lib\http\client.py", line 1252, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "C:\Users\jun\AppData\Local\Programs\Python\Python38\lib\http\client.py", line 1298, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "C:\Users\jun\AppData\Local\Programs\Python\Python38\lib\http\client.py", line 1247, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "C:\Users\jun\AppData\Local\Programs\Python\Python38\lib\http\client.py", line 1007, in _send_output
    self.send(msg)
  File "C:\Users\jun\AppData\Local\Programs\Python\Python38\lib\http\client.py", line 947, in send
    self.connect()
  File "C:\Users\jun\Documents\GitHub\sourcecode\venv\lib\site-packages\urllib3\connection.py", line 205, in connect
    conn = self._new_conn()
  File "C:\Users\jun\Documents\GitHub\sourcecode\venv\lib\site-packages\urllib3\connection.py", line 186, in _new_conn
    raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x000001EBCD1310D0>: Failed to establish a new connection: [WinError 10061] 대상 컴퓨터에서 연결을 거부했으므로 연결하지 못했습니다

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\jun\Documents\GitHub\sourcecode\venv\lib\site-packages\requests\adapters.py", line 489, in send
    resp = conn.urlopen(
  File "C:\Users\jun\Documents\GitHub\sourcecode\venv\lib\site-packages\urllib3\connectionpool.py", line 787, in urlopen
    retries = retries.increment(
  File "C:\Users\jun\Documents\GitHub\sourcecode\venv\lib\site-packages\urllib3\util\retry.py", line 592, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='127.0.0.1', port=80): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x000001EBCD1310D0>: Failed to establish a new connection: [WinError 10061] 대상 컴퓨터에서 연결을 거부했으므로 연결하지 못했습니다'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:/Users/jun/Documents/GitHub/sourcecode/python/example/_52_requests/timeout.py", line 14, in url_connect
    r = requests.get(url)
  File "C:\Users\jun\Documents\GitHub\sourcecode\venv\lib\site-packages\requests\api.py", line 73, in get
    return request("get", url, params=params, **kwargs)
  File "C:\Users\jun\Documents\GitHub\sourcecode\venv\lib\site-packages\requests\api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
  File "C:\Users\jun\Documents\GitHub\sourcecode\venv\lib\site-packages\requests\sessions.py", line 587, in request
    resp = self.send(prep, **send_kwargs)
  File "C:\Users\jun\Documents\GitHub\sourcecode\venv\lib\site-packages\requests\sessions.py", line 701, in send
    r = adapter.send(request, **kwargs)
  File "C:\Users\jun\Documents\GitHub\sourcecode\venv\lib\site-packages\requests\adapters.py", line 565, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='127.0.0.1', port=80): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x000001EBCD1310D0>: Failed to establish a new connection: [WinError 10061] 대상 컴퓨터에서 연결을 거부했으므로 연결하지 못했습니다'))
********** trying connect http://192.168.0.254 *************
end time: 0:00:10.010868
EXCEPTION HTTPConnectionPool(host='192.168.0.254', port=80): Max retries exceeded with url: / (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x000001EBCD131910>, 'Connection to 192.168.0.254 timed out. (connect timeout=10)'))
Traceback (most recent call last):
  File "C:\Users\jun\Documents\GitHub\sourcecode\venv\lib\site-packages\urllib3\connection.py", line 174, in _new_conn
    conn = connection.create_connection(
  File "C:\Users\jun\Documents\GitHub\sourcecode\venv\lib\site-packages\urllib3\util\connection.py", line 95, in create_connection
    raise err
  File "C:\Users\jun\Documents\GitHub\sourcecode\venv\lib\site-packages\urllib3\util\connection.py", line 85, in create_connection
    sock.connect(sa)
socket.timeout: timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\jun\Documents\GitHub\sourcecode\venv\lib\site-packages\urllib3\connectionpool.py", line 703, in urlopen
    httplib_response = self._make_request(
  File "C:\Users\jun\Documents\GitHub\sourcecode\venv\lib\site-packages\urllib3\connectionpool.py", line 398, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "C:\Users\jun\Documents\GitHub\sourcecode\venv\lib\site-packages\urllib3\connection.py", line 239, in request
    super(HTTPConnection, self).request(method, url, body=body, headers=headers)
  File "C:\Users\jun\AppData\Local\Programs\Python\Python38\lib\http\client.py", line 1252, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "C:\Users\jun\AppData\Local\Programs\Python\Python38\lib\http\client.py", line 1298, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "C:\Users\jun\AppData\Local\Programs\Python\Python38\lib\http\client.py", line 1247, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "C:\Users\jun\AppData\Local\Programs\Python\Python38\lib\http\client.py", line 1007, in _send_output
    self.send(msg)
  File "C:\Users\jun\AppData\Local\Programs\Python\Python38\lib\http\client.py", line 947, in send
    self.connect()
  File "C:\Users\jun\Documents\GitHub\sourcecode\venv\lib\site-packages\urllib3\connection.py", line 205, in connect
    conn = self._new_conn()
  File "C:\Users\jun\Documents\GitHub\sourcecode\venv\lib\site-packages\urllib3\connection.py", line 179, in _new_conn
    raise ConnectTimeoutError(
urllib3.exceptions.ConnectTimeoutError: (<urllib3.connection.HTTPConnection object at 0x000001EBCD131910>, 'Connection to 192.168.0.254 timed out. (connect timeout=10)')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\jun\Documents\GitHub\sourcecode\venv\lib\site-packages\requests\adapters.py", line 489, in send
    resp = conn.urlopen(
  File "C:\Users\jun\Documents\GitHub\sourcecode\venv\lib\site-packages\urllib3\connectionpool.py", line 787, in urlopen
    retries = retries.increment(
  File "C:\Users\jun\Documents\GitHub\sourcecode\venv\lib\site-packages\urllib3\util\retry.py", line 592, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='192.168.0.254', port=80): Max retries exceeded with url: / (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x000001EBCD131910>, 'Connection to 192.168.0.254 timed out. (connect timeout=10)'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:/Users/jun/Documents/GitHub/sourcecode/python/example/_52_requests/timeout.py", line 12, in url_connect
    r = requests.get(url, timeout=timeout)
  File "C:\Users\jun\Documents\GitHub\sourcecode\venv\lib\site-packages\requests\api.py", line 73, in get
    return request("get", url, params=params, **kwargs)
  File "C:\Users\jun\Documents\GitHub\sourcecode\venv\lib\site-packages\requests\api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
  File "C:\Users\jun\Documents\GitHub\sourcecode\venv\lib\site-packages\requests\sessions.py", line 587, in request
    resp = self.send(prep, **send_kwargs)
  File "C:\Users\jun\Documents\GitHub\sourcecode\venv\lib\site-packages\requests\sessions.py", line 701, in send
    r = adapter.send(request, **kwargs)
  File "C:\Users\jun\Documents\GitHub\sourcecode\venv\lib\site-packages\requests\adapters.py", line 553, in send
    raise ConnectTimeout(e, request=request)
requests.exceptions.ConnectTimeout: HTTPConnectionPool(host='192.168.0.254', port=80): Max retries exceeded with url: / (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x000001EBCD131910>, 'Connection to 192.168.0.254 timed out. (connect timeout=10)'))

call stack 부분을 빼고 다시 정리해봤습니다.

********** trying connect http://1.1.1.1:9999 *************
end time: 0:00:21.042910
EXCEPTION HTTPConnectionPool(host='1.1.1.1', port=9999): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x000001EBCD096670>: Failed to establish a new connection: [WinError 10060] 연결된 구성원으로부터 응답이 없어 연결하지 못했거나, 호스트로부터 응답이 없어 연결이 끊어졌습니다'))
********** trying connect http://192.168.0.254 *************
end time: 0:00:21.032674
EXCEPTION HTTPConnectionPool(host='192.168.0.254', port=80): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x000001EBCCC4A8B0>: Failed to establish a new connection: [WinError 10060] 연결된 구성원으로부터 응답이 없어 연결하지 못했거나, 호스트로부터 응답이 없어 연결이 끊어졌습니다'))
********** trying connect http://127.0.0.1 *************
end time: 0:00:02.053697
EXCEPTION HTTPConnectionPool(host='127.0.0.1', port=80): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x000001EBCD1310D0>: Failed to establish a new connection: [WinError 10061] 대상 컴퓨터에서 연결을 거부했으므로 연결하지 못했습니다'))
********** trying connect http://192.168.0.254 *************
end time: 0:00:10.010868
EXCEPTION HTTPConnectionPool(host='192.168.0.254', port=80): Max retries exceeded with url: / (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x000001EBCD131910>, 'Connection to 192.168.0.254 timed out. (connect timeout=10)'))

test code가 크게 4가지 인데요.

1.1.1.1:9999 의 주소는 응답을 안할꺼라는 가정하에 테스트 해봤습니다.

21초 걸렸고 Max retries exceeded 트라이가 초과되었다? 라는 애매한 표현을 하고 있습니다.

두번째는 내부 공유기 쪽으로 안쓰는 포트에 시도해보았습니다.

마찬가지로 21초가 걸렸습니다.

세번째 127.0.0.1 은 내부 포트로 2초만에 응답이 없습니다. timeout이 아니고 거부 형태입니다.

마지막은 timeout을 10초로 넣고 두번째 테스트 했던 내부 공유기 주소로 사용해봤습니다.

10초에 Max retries exceeded 부분은 동일하나 뒤쪽에 보면 timed out 이 발생함을 알 수 있습니다.

위 테스트에서 21초가 걸리긴 했지만 그것을 내부 기본 timeout이라고 할 수는 없습니다. 내부적으로 소켓에 대한 타임 아웃이 있고 그걸 retry 하는 형태로 구현이 되어 있음을 알 수 있으며, 우리가 설정하는 timeout 과는 Exception 이 다름을 알 수 있었습니다.


사용법

타임 아웃이 필요한경우는

r = requests.get('https://github.com', timeout=5)

위와 같이 사용해주면 되는데, 2가지 종류의 시간을 설정이 가능합니다. connect, read 시간입니다. connect timeout은 일반적으로 listen 하고 있는 서버에 접속하면 accept (연결) 를 하게되는데 그때까지의 시간 이라고 보면 됩니다.

The connect timeout is the number of seconds Requests will wait for your client to establish a connection to a remote machine (corresponding to the connect()) call on the socket.

read 시간은 데이터를 주고 받을때의 시간입니다. 클라이언트가 서버에서 전송된 바이트 사이에 대기하는 시간(초)입니다.

(Specifically, it’s the number of seconds that the client will wait between bytes sent from the server. In 99.9% of cases, this is the time before the server sends the first byte).

r = requests.get('https://github.com', timeout=(3.05, 27))

timeout=(connect timeout, read time) 형태로 설정합니다.

무한정 기다리는 경우 아래와 같이 합니다. 그런데 앞에서 테스트 해봤듯이 꼭 무한정이 되는것은 아닙니다.

r = requests.get('https://github.com', timeout=None)


Timeout 을 넘어서

여기에서 한가지 궁금한 점이 생겼습니다. requests에서 파일을 받거나 web page를 받을때의 timeout은 전체 크기를 받는 부분의 timeout 인가라는 의문입니다.

예를 들어 100MB의 파일을 받는 상황이고 requests.get의 timeout이 1초 일때 100MB를 1초만에 받지 못하면 timeout이 발생하는가입니다.

그래서 예제를 만들어 봤습니다.

웹 서버를 구성해야하는 예제이므로 복잡하지만 간단하게 예제를 위해서 socket 으로 설계했습니다.


import socket


IP = '0.0.0.0'
PORT = 8080
ADDR = (IP, PORT)


def make_dummy_message(len):
    ret = ""
    for i in range(0, len):
        ret = ret + 'A'
    return ret

head_data = f"""HTTP/1.1 200 OK

Server: Werkzeug/2.2.2 Python/3.8.10
Date: Sat, 14 Oct 2023 01:43:25 GMT
Content-Type: text/html; charset=utf-8
Content-Length: 100000000
Connection: close

<!doctype html>
<html lang=en>
"""

with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as server_socket:
    server_socket.bind(ADDR)
    server_socket.listen()
    print(f"listen [{ADDR}]")

    while True:
        client_socket, client_addr = server_socket.accept()
        print(f"accept [{client_addr}]")
        # msg = client_socket.recv(SIZE)
        print(head_data)
        client_socket.sendall(head_data.encode())
        for i in range(1, 100):
            print(f"send {i}")
            client_socket.sendall(make_dummy_message(1000000).encode())

        client_socket.close()

web 소켓을 열어서 http 응답을 주는 형태입니다. make_dummy_message 에 의해서 약 1MB의 메세지를 client_socket.sendall 함수로 전달해 줍니다.

        for i in range(1, 100):
            print(f"send {i}")
            client_socket.sendall(make_dummy_message(1000000).encode())

1MB씩 100번 100MB가 전달되게 됩니다.


이제 client 코드 입니다.

import requests
import datetime
import traceback
import time
import socket

def url_connect(url, timeout=None):
    nowtime = datetime.datetime.now()
    try:
        print(f"********** trying connect {url} *************")
        if timeout != None:
            r = requests.get(url, timeout=timeout)
        else:
            r = requests.get(url)
        print(r)
    except Exception as e:
        print("end time:", datetime.datetime.now() - nowtime)
        time.sleep(1)
        print("EXCEPTION", e)
        traceback.print_exc()


if __name__ == "__main__":
   s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
   s.connect(("8.8.8.8", 80))
   myip = s.getsockname()[0]
   print(myip)
   s.close()
   url_connect(f"http://{myip}:8080", timeout=1)

나의 ip를 구한다음 8080 포트로 접속해 보는것입니다.

서버를 실행하지 않은 상태는 1초만에 아래 형태로 나타납니다.

192.168.0.35
********** trying connect http://192.168.0.35:8080 *************
end time: 0:00:01.013626
EXCEPTION HTTPConnectionPool(host='192.168.0.35', port=8080): Max retries exceeded with url: / (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x000001A957F37670>, 'Connection to 192.168.0.35 timed out. (connect timeout=1)'))
Traceback (most recent call last):
  File "C:\Users\jun\Documents\GitHub\sourcecode\venv\lib\site-packages\urllib3\connection.py", line 174, in _new_conn
    conn = connection.create_connection(
  File "C:\Users\jun\Documents\GitHub\sourcecode\venv\lib\site-packages\urllib3\util\connection.py", line 95, in create_connection
    raise err
  File "C:\Users\jun\Documents\GitHub\sourcecode\venv\lib\site-packages\urllib3\util\connection.py", line 85, in create_connection
    sock.connect(sa)
socket.timeout: timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\jun\Documents\GitHub\sourcecode\venv\lib\site-packages\urllib3\connectionpool.py", line 703, in urlopen
    httplib_response = self._make_request(
  File "C:\Users\jun\Documents\GitHub\sourcecode\venv\lib\site-packages\urllib3\connectionpool.py", line 398, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "C:\Users\jun\Documents\GitHub\sourcecode\venv\lib\site-packages\urllib3\connection.py", line 239, in request
    super(HTTPConnection, self).request(method, url, body=body, headers=headers)
  File "C:\Users\jun\AppData\Local\Programs\Python\Python38\lib\http\client.py", line 1252, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "C:\Users\jun\AppData\Local\Programs\Python\Python38\lib\http\client.py", line 1298, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "C:\Users\jun\AppData\Local\Programs\Python\Python38\lib\http\client.py", line 1247, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "C:\Users\jun\AppData\Local\Programs\Python\Python38\lib\http\client.py", line 1007, in _send_output
    self.send(msg)
  File "C:\Users\jun\AppData\Local\Programs\Python\Python38\lib\http\client.py", line 947, in send
    self.connect()
  File "C:\Users\jun\Documents\GitHub\sourcecode\venv\lib\site-packages\urllib3\connection.py", line 205, in connect
    conn = self._new_conn()
  File "C:\Users\jun\Documents\GitHub\sourcecode\venv\lib\site-packages\urllib3\connection.py", line 179, in _new_conn
    raise ConnectTimeoutError(
urllib3.exceptions.ConnectTimeoutError: (<urllib3.connection.HTTPConnection object at 0x000001A957F37670>, 'Connection to 192.168.0.35 timed out. (connect timeout=1)')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\jun\Documents\GitHub\sourcecode\venv\lib\site-packages\requests\adapters.py", line 489, in send
    resp = conn.urlopen(
  File "C:\Users\jun\Documents\GitHub\sourcecode\venv\lib\site-packages\urllib3\connectionpool.py", line 787, in urlopen
    retries = retries.increment(
  File "C:\Users\jun\Documents\GitHub\sourcecode\venv\lib\site-packages\urllib3\util\retry.py", line 592, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='192.168.0.35', port=8080): Max retries exceeded with url: / (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x000001A957F37670>, 'Connection to 192.168.0.35 timed out. (connect timeout=1)'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:/Users/jun/Documents/GitHub/sourcecode/python/example/_52_requests/timeout3.py", line 12, in url_connect
    r = requests.get(url, timeout=timeout)
  File "C:\Users\jun\Documents\GitHub\sourcecode\venv\lib\site-packages\requests\api.py", line 73, in get
    return request("get", url, params=params, **kwargs)
  File "C:\Users\jun\Documents\GitHub\sourcecode\venv\lib\site-packages\requests\api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
  File "C:\Users\jun\Documents\GitHub\sourcecode\venv\lib\site-packages\requests\sessions.py", line 587, in request
    resp = self.send(prep, **send_kwargs)
  File "C:\Users\jun\Documents\GitHub\sourcecode\venv\lib\site-packages\requests\sessions.py", line 701, in send
    r = adapter.send(request, **kwargs)
  File "C:\Users\jun\Documents\GitHub\sourcecode\venv\lib\site-packages\requests\adapters.py", line 553, in send
    raise ConnectTimeout(e, request=request)
requests.exceptions.ConnectTimeout: HTTPConnectionPool(host='192.168.0.35', port=8080): Max retries exceeded with url: / (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x000001A957F37670>, 'Connection to 192.168.0.35 timed out. (connect timeout=1)'))

서버를 실행한 다음에는 다른 에러가 발생하는데

192.168.0.35
********** trying connect http://192.168.0.35:8080 *************
end time: 0:00:11.814797
EXCEPTION ("Connection broken: ConnectionResetError(10054, '현재 연결은 원격 호스트에 의해 강제로 끊겼습니다', None, 10054, None)", ConnectionResetError(10054, '현재 연결은 원격 호스트에 의해 강제로 끊겼습니다', None, 10054, None))

대략 위와 같이 11초 후에 에러같은 것이 발생합니다.

몇가지 이유가 있는데 서버쪽을 대충 만들어서 그렇습니다. contents크기를 받고 소켓을 닫도록 되어있는데 해당 부분이 적절치 않게 만들었기 때문입니다.

timeout시간만 보려고 테스트 코드를 만든것이라 에러에 대해서는 무시하면됩니다.

11초가 걸렸는데도 1초 timeout이 발생하지 않았다는 의미는 구간 구간 socket read 시 1초 timeout이 발생할때만 멈춤다는 것을 알 수 있습니다.




작성자: 시간:
라벨: ,

댓글 없음:

댓글 쓰기

피드 구독하기: 댓글 (Atom)