Random 404 responses when fetching certain images from dotnet-buildtools/prereqs

[akoeplinger]

See dotnet/runtime#117164 (comment)

This sometimes happens for mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-22.04-helix-webassembly or mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-22.04-helix-sqlserver-amd64

Our build analysis shows that we've seen this 139 times in the last month for the webassembly image.

We think it might be a service issue with MCR.

[stevenhvtran]

Hi @akoeplinger, I can't see any corresponding 404 logs on our end. If possible, could you provide the value of the X-Msedge-Ref header returned by MCR when you observe a 404 response?

[cincuranet]

Any progress on this? Some days we're hitting so often with mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-22.04-helix-sqlserver-amd64 it is causing headache when reviewing PRs.

[stevenhvtran]

@akoeplinger , @cincuranet I've taken another look at this and think I've found the issue. It's likely related to your pipeline's usage of docker-py and throttling limits on MAR.

When using docker-py's high-level client.images.pull(), any failure during the pull stream is silently discarded. The caller instead receives a 404 Client Error: Not Found ("no such image: ...") which falsely suggests the image does not exist on the registry.

The high-level ImageCollection.pull() method calls the low-level streaming pull API and then discards every message in the stream — including error messages:

pull_log = self.client.api.pull(
    repository, tag=tag, stream=True, all_tags=all_tags, **kwargs
)
for _ in pull_log:
    pass  # ← all stream messages (including errors) silently discarded

(source)

After consuming the stream, it attempts to inspect the image locally:

return self.get(f'{repository}{sep}{tag}')

(source)

Since the pull silently failed, the image doesn't exist locally, and the Docker daemon returns a 404 "no such image". This is then raised as an ImageNotFound exception — completely masking the real error.

In your case, the actual failure was likely throttling by MAR. Large image pulls generate many 4MB range-request chunks for layer downloads, which can exceed MCR's rate limit. When throttled:

1. MAR blocks incoming requests
2. The Docker daemon reports the throttling error in the pull stream as {"error": "..."}
3. docker-py discards it and raises a misleading ImageNotFound
4. Retrying the full pull doesn't help because the rate-limit window hasn't expired

The problem is exacerbated with many clients performing pulls and larger images. It may be why you experience docker pull failures on some images but not others.

Possible steps

• Send me the timestamps in which you experience this issue and the particular images you're facing it with. I can confirm whether throttling is occurring for your client or not
• Check the Docker Daemon logs with journalctl -u docker.service or an equivalent for your OS to find more accurate error messages than the 404 when the error occurs
• Explore options to decrease the rate at which images are pulled in your pipelines, can you cache images?
• Push an upstream fix to docker-py to not suppress error logs in the pull stream
• Implement exponential back-offs for your pulls