Ability to resume pulling image details

[glensc]

The way this tool works is slow, and if there's one single error, the whole fetching will be aborted. And the longer the program runs the higher chance is there will be network error, meaning can never get the result from the command.

[glensc]

Could some caching be added, so could take off where left off with an error? perhaps:

1. fetch list of images. cache list
2. fetch list of tags. cache list
3. retrieve attributes of each tag.

so each cache list is resumed until everything is worked over. so basically it continues to walk snapshot of the images, i.e at the time first list was obtained.

[glensc]

here's exception trace, to get a feeling how painful this is:

11:35:42 root[load: 0.07]@ipa gitlab-registry-tools/registry-usage# time pipenv run ./registry_analyze.py cred.prod.py
Traceback (most recent call last):
  File "/home/glen/.local/share/virtualenvs/registry-usage-1e3voAqq/lib/python3.7/site-packages/urllib3/connection.py", line 160, in _new_conn
    (self._dns_host, self.port), self.timeout, **extra_kw
  File "/home/glen/.local/share/virtualenvs/registry-usage-1e3voAqq/lib/python3.7/site-packages/urllib3/util/connection.py", line 84, in create_connection
    raise err
  File "/home/glen/.local/share/virtualenvs/registry-usage-1e3voAqq/lib/python3.7/site-packages/urllib3/util/connection.py", line 74, in create_connection
    sock.connect(sa)
TimeoutError: [Errno 110] Connection timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/glen/.local/share/virtualenvs/registry-usage-1e3voAqq/lib/python3.7/site-packages/urllib3/connectionpool.py", line 677, in urlopen
    chunked=chunked,
  File "/home/glen/.local/share/virtualenvs/registry-usage-1e3voAqq/lib/python3.7/site-packages/urllib3/connectionpool.py", line 381, in _make_request
    self._validate_conn(conn)
  File "/home/glen/.local/share/virtualenvs/registry-usage-1e3voAqq/lib/python3.7/site-packages/urllib3/connectionpool.py", line 976, in _validate_conn
    conn.connect()
  File "/home/glen/.local/share/virtualenvs/registry-usage-1e3voAqq/lib/python3.7/site-packages/urllib3/connection.py", line 308, in connect
    conn = self._new_conn()
  File "/home/glen/.local/share/virtualenvs/registry-usage-1e3voAqq/lib/python3.7/site-packages/urllib3/connection.py", line 172, in _new_conn
    self, "Failed to establish a new connection: %s" % e
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPSConnection object at 0x7f17ef4ced30>: Failed to establish a new connection: [Errno 110] Connection timed out



During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/glen/.local/share/virtualenvs/registry-usage-1e3voAqq/lib/python3.7/site-packages/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/home/glen/.local/share/virtualenvs/registry-usage-1e3voAqq/lib/python3.7/site-packages/urllib3/connectionpool.py", line 725, in urlopen
    method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
  File "/home/glen/.local/share/virtualenvs/registry-usage-1e3voAqq/lib/python3.7/site-packages/urllib3/util/retry.py", line 439, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='gitlab.example.net', port=4567): Max retries exceeded with url: /v2/ee/portal/vue-digi/admin/production/blobs/sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f17ef4ced30>: Failed to establish a new connection: [Errno 110] Connection timed out'))


During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "./registry_analyze.py", line 56, in <module>
    gitlab_registry = getRegistry(cache_file='{hostname}.json'.format(hostname=urlparse(c['registry_base_url']).netloc), config=c)
  File "./registry_analyze.py", line 44, in getRegistry
    data['repository_tags'] = gitlab_registry.repository_tags
  File "/home/glen/.local/share/virtualenvs/registry-usage-1e3voAqq/lib/python3.7/site-packages/gitlab_registry_usage/registry/high_level_api.py", line 132, in repository_tags
    for repository, tag_layers in self.repository_layers.items()
  File "/home/glen/.local/share/virtualenvs/registry-usage-1e3voAqq/lib/python3.7/site-packages/gitlab_registry_usage/registry/high_level_api.py", line 139, in repository_layers
    self._repository_layers, self._layer_sizes = self._get_repository_layers_and_layer_sizes()
  File "/home/glen/.local/share/virtualenvs/registry-usage-1e3voAqq/lib/python3.7/site-packages/gitlab_registry_usage/registry/high_level_api.py", line 65, in _get_repository_layers_and_layer_sizes
    layer_size = get_layer_size(self._registry_url, repository_auth_token, repository, layer)
  File "/home/glen/.local/share/virtualenvs/registry-usage-1e3voAqq/lib/python3.7/site-packages/gitlab_registry_usage/registry/low_level_api.py", line 120, in get_layer_size
    response = requests.head(blob_url, headers={"Authorization": "Bearer " + auth_token})
  File "/home/glen/.local/share/virtualenvs/registry-usage-1e3voAqq/lib/python3.7/site-packages/requests/api.py", line 104, in head
    return request('head', url, **kwargs)
  File "/home/glen/.local/share/virtualenvs/registry-usage-1e3voAqq/lib/python3.7/site-packages/requests/api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "/home/glen/.local/share/virtualenvs/registry-usage-1e3voAqq/lib/python3.7/site-packages/requests/sessions.py", line 530, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/glen/.local/share/virtualenvs/registry-usage-1e3voAqq/lib/python3.7/site-packages/requests/sessions.py", line 643, in send
    r = adapter.send(request, **kwargs)
  File "/home/glen/.local/share/virtualenvs/registry-usage-1e3voAqq/lib/python3.7/site-packages/requests/adapters.py", line 516, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='gitlab.example.net', port=4567): Max retries exceeded with url: /v2/ee/portal/vue-digi/admin/production/blobs/sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f17ef4ced30>: Failed to establish a new connection: [Errno 110] Connection timed out'))

real    49m16.475s
user    23m16.942s
sys     1m9.929s

[glensc]

I haven't checked the code, but how smart is the program fetching JWT token? does it obtain new one for any request it makes, or is able to re-use the token? maybe some time could be won from there making it smart, reusing the token until it expires?

[IngoMeyer441]

The tool fetches an auth token for every repository (I don't know another way for GitLab registries). The main problem is, that there is no other way than fetching every image layer size with a separate http request which makes this tool terribly slow. Maybe, it would be better to completely rewrite this tool to access the Docker registry on a filesystem level (with a daemon which runs on the GitLab host itself).