This view contacts the LCO observe portal to find out what targets are being observed.
When this query times out, the page returns a 500 error.
These queries need to be async calls.
mop-546f67dd66-2fbfk mop PRIORITYTARGETS started context get 2026-03-27 04:43:39.555299
mop-546f67dd66-2fbfk mop CHECKPOINT: N DB connections: 0, memory: 692.13MiB
mop-546f67dd66-2fbfk mop /mop/.venv/lib/python3.10/site-packages/django/views/generic/list.py:91: UnorderedObjectListWarning:
mop-546f67dd66-2fbfk mop
mop-546f67dd66-2fbfk mop Pagination may yield inconsistent results with an unordered object_list: <class 'tom_targets.models.TargetExtra'> QuerySet.
mop-546f67dd66-2fbfk mop
mop-546f67dd66-2fbfk mop QueryTools: identified 0 targets
mop-546f67dd66-2fbfk mop QueryTools: identified 24 targets
mop-546f67dd66-2fbfk mop Internal Server Error: /prioritytargets/
mop-546f67dd66-2fbfk mop Traceback (most recent call last):
mop-546f67dd66-2fbfk mop File "/mop/.venv/lib/python3.10/site-packages/urllib3/connectionpool.py", line 534, in _make_request
mop-546f67dd66-2fbfk mop response = conn.getresponse()
mop-546f67dd66-2fbfk mop File "/mop/.venv/lib/python3.10/site-packages/urllib3/connection.py", line 571, in getresponse
mop-546f67dd66-2fbfk mop httplib_response = super().getresponse()
mop-546f67dd66-2fbfk mop File "/usr/local/lib/python3.10/http/client.py", line 1375, in getresponse
mop-546f67dd66-2fbfk mop response.begin()
mop-546f67dd66-2fbfk mop File "/usr/local/lib/python3.10/http/client.py", line 318, in begin
mop-546f67dd66-2fbfk mop version, status, reason = self._read_status()
mop-546f67dd66-2fbfk mop File "/usr/local/lib/python3.10/http/client.py", line 279, in _read_status
mop-546f67dd66-2fbfk mop line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
mop-546f67dd66-2fbfk mop File "/usr/local/lib/python3.10/socket.py", line 705, in readinto
mop-546f67dd66-2fbfk mop return self._sock.recv_into(b)
mop-546f67dd66-2fbfk mop File "/mop/.venv/lib/python3.10/site-packages/gevent/ssl.py", line 635, in recv_into
mop-546f67dd66-2fbfk mop return self.read(nbytes, buffer)
mop-546f67dd66-2fbfk mop File "/mop/.venv/lib/python3.10/site-packages/gevent/ssl.py", line 443, in read
mop-546f67dd66-2fbfk mop self._wait(self._read_event, timeout_exc=_SSLErrorReadTimeout)
mop-546f67dd66-2fbfk mop File "src/gevent/_hub_primitives.py", line 317, in gevent._gevent_c_hub_primitives.wait_on_socket
mop-546f67dd66-2fbfk mop File "src/gevent/_hub_primitives.py", line 322, in gevent._gevent_c_hub_primitives.wait_on_socket
mop-546f67dd66-2fbfk mop File "src/gevent/_hub_primitives.py", line 313, in gevent._gevent_c_hub_primitives._primitive_wait
mop-546f67dd66-2fbfk mop File "src/gevent/_hub_primitives.py", line 314, in gevent._gevent_c_hub_primitives._primitive_wait
mop-546f67dd66-2fbfk mop File "src/gevent/_hub_primitives.py", line 55, in gevent._gevent_c_hub_primitives.WaitOperationsGreenlet.wait
mop-546f67dd66-2fbfk mop File "src/gevent/_waiter.py", line 154, in gevent._gevent_c_waiter.Waiter.get
mop-546f67dd66-2fbfk mop File "src/gevent/_greenlet_primitives.py", line 65, in gevent._gevent_c_greenlet_primitives.SwitchOutGreenletWithLoop.switch
mop-546f67dd66-2fbfk mop File "src/gevent/_gevent_c_greenlet_primitives.pxd", line 35, in gevent._gevent_c_greenlet_primitives._greenlet_switch
mop-546f67dd66-2fbfk mop TimeoutError: The read operation timed out
mop-546f67dd66-2fbfk mop
mop-546f67dd66-2fbfk mop The above exception was the direct cause of the following exception:
mop-546f67dd66-2fbfk mop
mop-546f67dd66-2fbfk mop Traceback (most recent call last):
mop-546f67dd66-2fbfk mop File "/mop/.venv/lib/python3.10/site-packages/requests/adapters.py", line 644, in send
mop-546f67dd66-2fbfk mop resp = conn.urlopen(
mop-546f67dd66-2fbfk mop File "/mop/.venv/lib/python3.10/site-packages/urllib3/connectionpool.py", line 841, in urlopen
mop-546f67dd66-2fbfk mop retries = retries.increment(
mop-546f67dd66-2fbfk mop File "/mop/.venv/lib/python3.10/site-packages/urllib3/util/retry.py", line 490, in increment
mop-546f67dd66-2fbfk mop raise reraise(type(error), error, _stacktrace)
mop-546f67dd66-2fbfk mop File "/mop/.venv/lib/python3.10/site-packages/urllib3/util/util.py", line 39, in reraise
mop-546f67dd66-2fbfk mop raise value
mop-546f67dd66-2fbfk mop File "/mop/.venv/lib/python3.10/site-packages/urllib3/connectionpool.py", line 787, in urlopen
mop-546f67dd66-2fbfk mop response = self._make_request(
mop-546f67dd66-2fbfk mop File "/mop/.venv/lib/python3.10/site-packages/urllib3/connectionpool.py", line 536, in _make_request
mop-546f67dd66-2fbfk mop self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
mop-546f67dd66-2fbfk mop File "/mop/.venv/lib/python3.10/site-packages/urllib3/connectionpool.py", line 367, in _raise_timeout
mop-546f67dd66-2fbfk mop raise ReadTimeoutError(
mop-546f67dd66-2fbfk mop urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='observe.lco.global', port=443): Read timed out. (read timeout=20)
mop-546f67dd66-2fbfk mop
mop-546f67dd66-2fbfk mop During handling of the above exception, another exception occurred:
mop-546f67dd66-2fbfk mop
mop-546f67dd66-2fbfk mop Traceback (most recent call last):
mop-546f67dd66-2fbfk mop File "/mop/.venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 55, in inner
mop-546f67dd66-2fbfk mop response = get_response(request)
mop-546f67dd66-2fbfk mop File "/mop/.venv/lib/python3.10/site-packages/django/core/handlers/base.py", line 199, in _get_response
mop-546f67dd66-2fbfk mop response = self.process_exception_by_middleware(e, request)
mop-546f67dd66-2fbfk mop File "/mop/.venv/lib/python3.10/site-packages/django/core/handlers/base.py", line 362, in process_exception_by_middleware
mop-546f67dd66-2fbfk mop response = middleware_method(request, exception)
mop-546f67dd66-2fbfk mop File "/mop/.venv/lib/python3.10/site-packages/tom_common/middleware.py", line 30, in process_exception
mop-546f67dd66-2fbfk mop raise exception
mop-546f67dd66-2fbfk mop File "/mop/.venv/lib/python3.10/site-packages/django/core/handlers/base.py", line 197, in _get_response
mop-546f67dd66-2fbfk mop response = wrapped_callback(request, *callback_args, **callback_kwargs)
mop-546f67dd66-2fbfk mop File "/mop/.venv/lib/python3.10/site-packages/django/views/generic/base.py", line 105, in view
mop-546f67dd66-2fbfk mop return self.dispatch(request, *args, **kwargs)
mop-546f67dd66-2fbfk mop File "/mop/.venv/lib/python3.10/site-packages/django/views/generic/base.py", line 144, in dispatch
mop-546f67dd66-2fbfk mop return handler(request, *args, **kwargs)
mop-546f67dd66-2fbfk mop File "/mop/.venv/lib/python3.10/site-packages/django/views/generic/list.py", line 174, in get
mop-546f67dd66-2fbfk mop context = self.get_context_data()
mop-546f67dd66-2fbfk mop File "/mop/mop/views.py", line 261, in get_context_data
mop-546f67dd66-2fbfk mop obs_targets = self.get_observed_targets_data()
mop-546f67dd66-2fbfk mop File "/mop/mop/views.py", line 370, in get_observed_targets_data
mop-546f67dd66-2fbfk mop response = fetch_all_lco_requestgroups()
mop-546f67dd66-2fbfk mop File "/mop/mop/toolbox/obs_control.py", line 47, in fetch_all_lco_requestgroups
mop-546f67dd66-2fbfk mop response = requests.get(url, headers=headers, timeout=20).json()
mop-546f67dd66-2fbfk mop File "/mop/.venv/lib/python3.10/site-packages/requests/api.py", line 73, in get
mop-546f67dd66-2fbfk mop return request("get", url, params=params, **kwargs)
mop-546f67dd66-2fbfk mop File "/mop/.venv/lib/python3.10/site-packages/requests/api.py", line 59, in request
mop-546f67dd66-2fbfk mop return session.request(method=method, url=url, **kwargs)
mop-546f67dd66-2fbfk mop File "/mop/.venv/lib/python3.10/site-packages/requests/sessions.py", line 589, in request
mop-546f67dd66-2fbfk mop resp = self.send(prep, **send_kwargs)
mop-546f67dd66-2fbfk mop File "/mop/.venv/lib/python3.10/site-packages/requests/sessions.py", line 703, in send
mop-546f67dd66-2fbfk mop r = adapter.send(request, **kwargs)
mop-546f67dd66-2fbfk mop File "/mop/.venv/lib/python3.10/site-packages/requests/adapters.py", line 690, in send
mop-546f67dd66-2fbfk mop raise ReadTimeout(e, request=request)
mop-546f67dd66-2fbfk mop requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='observe.lco.global', port=443): Read timed out. (read timeout=20)
mop-546f67dd66-2fbfk nginx 10.100.15.142 - - [27/Mar/2026:04:43:59 +0000] "GET /prioritytargets/ HTTP/1.1" 500 145 "https://mop.lco.global/targets/56073/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/146.0.0.0 Safari/537.36" "67.150.30.180"
mop-546f67dd66-2fbfk mop 127.0.0.1 - - [27/Mar/2026:04:43:59 +0000] "GET /prioritytargets/ HTTP/1.1" 500 145 "https://mop.lco.global/targets/56073/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/146.0.0.0 Safari/537.36"
This view contacts the LCO observe portal to find out what targets are being observed.
When this query times out, the page returns a 500 error.
These queries need to be async calls.