handle gracefully registry errors

[glensc]

gitlab, registry, deleting from gitlab ui, or any combination of these tends to create some state in registry that not all data is consistent.

here's another one. please make to fail gracefully on these:

Traceback (most recent call last):
  File "/home/vagrant/scm/docker/gitlab-registry-tools/registry-usage/registry_analyze.py", line 56, in <module>
    gitlab_registry = getRegistry(cache_file='{hostname}.json'.format(hostname=urlparse(c['registry_base_url']).netloc), config=c)
  File "/home/vagrant/scm/docker/gitlab-registry-tools/registry-usage/registry_analyze.py", line 44, in getRegistry
    data['repository_tags'] = gitlab_registry.repository_tags
  File "/home/vagrant/.local/share/virtualenvs/registry-usage--61pTqM7/lib/python3.7/site-packages/gitlab_registry_usage/registry/high_level_api.py", line 109, in repository_tags
    for repository, tag_layers in self.repository_layers.items()
  File "/home/vagrant/.local/share/virtualenvs/registry-usage--61pTqM7/lib/python3.7/site-packages/gitlab_registry_usage/registry/high_level_api.py", line 116, in repository_layers
    self._repository_layers, self._layer_sizes = self._get_repository_layers_and_layer_sizes()
  File "/home/vagrant/.local/share/virtualenvs/registry-usage--61pTqM7/lib/python3.7/site-packages/gitlab_registry_usage/registry/high_level_api.py", line 45, in _get_repository_layers_and_layer_sizes
    tag_layers = get_tag_layers(self._registry_url, repository_auth_token, repository, tag)
  File "/home/vagrant/.local/share/virtualenvs/registry-usage--61pTqM7/lib/python3.7/site-packages/gitlab_registry_usage/registry/low_level_api.py", line 99, in get_tag_layers
    raise LayersReadError
gitlab_registry_usage.registry.low_level_api.LayersReadError

I might be able to provide more information after retry. it runs ~45minutes to get to the error, so you might understand why handling these more gracefully may be important ;)

I think i've reported this at least once somewhere, but can't find right now which project :(

[glensc]

looks like my diff, was never published, and is something like:

root@guinea-pig docker/gitlab-registry-usage# g diff
diff --git a/gitlab_registry_usage/registry/low_level_api.py b/gitlab_registry_usage/registry/low_level_api.py
index 538d949..61e50b1 100644
--- a/gitlab_registry_usage/registry/low_level_api.py
+++ b/gitlab_registry_usage/registry/low_level_api.py
@@ -96,7 +96,10 @@ def get_tag_layers(registry_url: str, auth_token: str, repository: str, tag: str
     tag_layers_url = '{base}v2/{repository}/manifests/{tag}'.format(base=registry_url, repository=repository, tag=tag)
     response = requests.get(tag_layers_url, headers={'Authorization': 'Bearer ' + auth_token})
     if response.status_code != 200:
-        raise LayersReadError
+        from pprint import pprint
+        pprint("layers read error: %s, %r" % (tag_layers_url, response))
+#        raise LayersReadError
+        return []
     try:
         json_response = response.json()
     except json.decoder.JSONDecodeError:
@@ -116,15 +119,20 @@ def get_tag_layers(registry_url: str, auth_token: str, repository: str, tag: str
 def get_layer_size(registry_url: str, auth_token: str, repository: str, layer: str) -> int:
     blob_url = '{base}v2/{repository}/blobs/{layer}'.format(base=registry_url, repository=repository, layer=layer)
     response = requests.head(blob_url, headers={'Authorization': 'Bearer ' + auth_token})
+    redirect_location = None

     # Handle redirect (S3 Backend for example)
     if response.status_code == 307:
         if 'Location' not in response.headers:
             raise LayerSizeReadError
+        redirect_location = response.headers['Location']
         response = requests.head(response.headers['Location'])

     if response.status_code != 200:
-        raise LayerSizeReadError
+        from pprint import pprint
+        pprint("layers read error: %s, %s: %r" % (blob_url, redirect_location, response))
+        return 0
+#        raise LayerSizeReadError
     response_header = response.headers
     if 'Content-Length' not in response_header:
         raise LayerSizeReadError

[IngoMeyer441]

Hey, thanks for your problem report. I had a similar issue during the initial development of this package for registry images which belong to projects which have been deleted after image creation (in this case GitLab denied access to the still existing images in the catalog). However, I am not sure if only a warning should be printed because it is a real error case (the existing layer or size cannot be read) and IMHO the user of this package should be able to handle this error case. If it is better to ignore this error in your program, couldn't you use a try/except block that simply passes on error?

[glensc]

But what to do then, thrown exception is not suitable. it breaks the whole application rather than skipping some specific image.

I still have this problem as this hasn't been solved (gitlab or docker registry or whoever problem this is)

08:16:52 root[load: 0.37]@ipa gitlab-registry-tools/registry-usage# time pipenv run ./registry_analyze.py cred.prod.py
Traceback (most recent call last):
  File "./registry_analyze.py", line 56, in <module>
    gitlab_registry = getRegistry(cache_file='{hostname}.json'.format(hostname=urlparse(c['registry_base_url']).netloc), config=c)
  File "./registry_analyze.py", line 44, in getRegistry
    data['repository_tags'] = gitlab_registry.repository_tags
  File "/home/glen/.local/share/virtualenvs/registry-usage-1e3voAqq/lib/python3.7/site-packages/gitlab_registry_usage/registry/high_level_api.py", line 128, in repository_tags
    for repository, tag_layers in self.repository_layers.items()
  File "/home/glen/.local/share/virtualenvs/registry-usage-1e3voAqq/lib/python3.7/site-packages/gitlab_registry_usage/registry/high_level_api.py", line 135, in repository_layers
    self._repository_layers, self._layer_sizes = self._get_repository_layers_and_layer_sizes()
  File "/home/glen/.local/share/virtualenvs/registry-usage-1e3voAqq/lib/python3.7/site-packages/gitlab_registry_usage/registry/high_level_api.py", line 64, in _get_repository_layers_and_layer_sizes
    layer_size = get_layer_size(self._registry_url, repository_auth_token, repository, layer)
  File "/home/glen/.local/share/virtualenvs/registry-usage-1e3voAqq/lib/python3.7/site-packages/gitlab_registry_usage/registry/low_level_api.py", line 129, in get_layer_size
    raise LayerSizeReadError
gitlab_registry_usage.registry.low_level_api.LayerSizeReadError

real    48m0.519s
user    21m19.819s
sys     1m3.501s

09:04:53 root[load: 0.72]@ipa gitlab-registry-tools/registry-usage#

[IngoMeyer441]

I could add the LayerSizeReadError to the errors which are ignored in high level api (similar to commit e186896). However, I am not quite sure how to handle this case correctly. TagsReadError and LayersReadError lead to a repository tag with unknown size. But if only one layer cannot be read, we get a wrong size.

[glensc]

Well, yeah, the result will be incorrect either way. I would just ignore the image, because the image is not accessible anyway, so the size of pullable images would be correct.