GlyGen: cfde-c2m2 prepare error

[jeet-vora]

When cfde-c2m2 prepare is run I am getting HTTPError: 413 Client Error. It seems the script is sending an HTTP request to the OLS (Ontology Lookup Service) API endpoint that is too large for the server to accept.

[rewrite protein]: loading existing IRIs...: 0it [00:00, ?it/s]
[rewrite protein]: preserving existing IRIs...: 307681it [00:01, 159613.06it/s]
[rewrite protein]: resolving new IRIs...: 0it [00:00, ?it/s]Traceback (most recent call last):
  File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/jeetvora/.local/share/pipx/venvs/cfde-c2m2/lib64/python3.9/site-packages/cfde_c2m2/ols/bulk_by_id.py", line 18, in <module>
    req.raise_for_status()
  File "/home/jeetvora/.local/share/pipx/venvs/cfde-c2m2/lib64/python3.9/site-packages/requests/models.py", line 1026, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 413 Client Error: Request Entity Too Large for url: https://ols.cfde.cloud/api/v1/bulk-by-id/UniProtKB
[rewrite protein]: resolving new IRIs...: 0it [00:00, ?it/s]
[rewrite protein]: finding ontology IRIs...: 0it [00:00, ?it/s]

[u8sand]

Hi @jeet-vora ; we've disabled this max body size on the ontology lookup service route. The code is already designed to operate in a streaming fashion so the size of the request shouldn't be an issue.

The other potential thing you could run across is the 10 minute max timeout. If you do run into this let us know and we can update the client to reconnect after 10 minutes and continue where it left off.

If you're still running into the issue, do let us know.

[jeet-vora]

@u8sand Yes still getting the error after 10 min timeoout.

[rewrite protein]: resolving new IRIs...: 20821it [00:01, 20303.16it/s]Traceback (most recent call last):
File "/home/jeetvora/.local/share/pipx/venvs/cfde-c2m2/lib64/python3.9/site-packages/requests/models.py", line 820, in generate
yield from self.raw.stream(chunk_size, decode_content=True)
File "/home/jeetvora/.local/share/pipx/venvs/cfde-c2m2/lib64/python3.9/site-packages/urllib3/response.py", line 1250, in stream
yield from self.read_chunked(amt, decode_content=decode_content)
File "/home/jeetvora/.local/share/pipx/venvs/cfde-c2m2/lib64/python3.9/site-packages/urllib3/response.py", line 1418, in read_chunked
self._update_chunk_length()
File "/home/jeetvora/.local/share/pipx/venvs/cfde-c2m2/lib64/python3.9/site-packages/urllib3/response.py", line 1344, in _update_chunk_length
raise ProtocolError("Response ended prematurely") from None
urllib3.exceptions.ProtocolError: Response ended prematurely

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/home/jeetvora/.local/share/pipx/venvs/cfde-c2m2/lib64/python3.9/site-packages/cfde_c2m2/ols/bulk_by_id.py", line 19, in
for chunk in req.iter_content(512):
File "/home/jeetvora/.local/share/pipx/venvs/cfde-c2m2/lib64/python3.9/site-packages/requests/models.py", line 822, in generate
raise ChunkedEncodingError(e)
requests.exceptions.ChunkedEncodingError: Response ended prematurely

[u8sand]

Hi Jeet. Okay, I've gone ahead and chunked requests into smaller blocks client-side so I believe this should no longer be an issue, in the chunk branch. But I've not gotten to testing it. If you can test it and let me know, otherwise I'll test it myself tomorrow.

It could be run like:

uvx --from 'git+https://github.com/maayanlab/cfde-c2m2@chunk' cfde-c2m2 prepare