DockerHub 429 in hosted runner

[steves-hes]

Select Topic Area

Question

Body

Today we started seeing 429 errors in our actions that are run in hosted runners. e.g.

#12 [internal] load metadata for docker.io/library/node:18-alpine
#12 ERROR: failed to copy: httpReadSeeker: failed open: unexpected status code https://registry-1.docker.io/v2/library/node/manifests/sha256:e37da457874383fa9217***67867ec85fe8fe59f***bfa351ec9752a9543868***56e: 429 Too Many Requests - Server message: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit

I was under the impression that docker hub rate limit did not apply to hosted runners? Is that incorrect?

I added a test step to check the DockerHub Rate limit - and it does appear to apply - only 200 per 6 hours?

0s
Run TOKEN=$(curl --user "***:***" "https://auth.docker.io/token?service=registry.docker.io&scope=repository:ratelimitpreview/test:pull" | jq -r .token)
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  ***     ***    ***     ***    ***     ***      ***      *** --:--:-- --:--:-- --:--:--     ***
  ***     ***    ***     ***    ***     ***      ***      *** --:--:-- --:--:-- --:--:--     ***
[1](https://github.com/Human-eSources/hes/actions/runs/9764474194/job/26953507765#step:9:1)***  5837    ***  5837    ***     ***  47796      *** --:--:-- --:--:-- --:--:-- 47844
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  ***     ***    ***     ***    ***     ***      ***      *** --:--:-- --:--:-- --:--:--     ***
  ***  2782    ***     ***    ***     ***      ***      *** --:--:-- --:--:-- --:--:--     ***
HTTP/1.1 2*** OK
content-length: 2782
content-type: application/vnd.docker.distribution.manifest.v1+prettyjws
docker-content-digest: sha256:767a3815c34823b355bed3[17](https://github.com/Human-eSources/hes/actions/runs/9764474194/job/26953507765#step:9:18)6***d5fa3daca***aec2ce15b217c9cd83229e***e2***2***
docker-distribution-api-version: registry/2.***
etag: "sha256:767a3815c34823b355bed3176***d5fa3daca***aec2ce15b[21](https://github.com/Human-eSources/hes/actions/runs/9764474194/job/26953507765#step:9:22)7c9cd83229e***e2***2***"
date: Tue, ***2 Jul 2***24 16:28:48 GMT
strict-transport-security: max-age=31536***
ratelimit-limit: 2***;w=216***
ratelimit-remaining: 5;w=216***
docker-ratelimit-source: b6bef***97-4d4a-4847-b578-4446***2***4f66b

[alexisseurin]

Of course, Docker Hub rate limits are applied to hosted runners.

"On November 20, 2020, rate limits anonymous and free authenticated use of Docker Hub went into effect. Anonymous and Free Docker Hub users are limited to 100 and 200 container image pull requests per six hours. Docker Pro and Docker Team accounts enable 5,000 pulls in a 24 hour period from Docker Hub."
Source: https://www.docker.com/increase-rate-limits/

Rate Limit Cheatsheet for Self-Hosting Github Runners

https://www.warpbuild.com/blog/rate-limits-self-hosted-runners

[manulifeam-rickhui]

Having the similar issue on our custom runners in these 2 days.

Seems the rate limits have been applied since 2020, but why it only start affecting us in these few days ?

[alexisseurin]

"If your CI runs on Azure and you are pulling base images from Docker Hub, you might be hit by the Docker rate limits beginning June 30, 2024.

• Use authentication to pull image : You can use a free Docker Hub user login to double the rate limit (200 pulls every 6 hours).

• Mirror or cache base images in Private Registry : if you use container based deployment, you probably use a private registry already. To avoid rate limits, either mirror or cache the public base images into your private registry, and change Dockerfile to point to private registry.

• Purchase Docker Hub Subscription : If mirror or cache base images in private registry doesn't work for some reason, you can always purchase a subscription of Docker and use that credential to authenticate, which will increase the limit to 5000 pulls per day (large enough for most organizations)."

Source: https://www.clofio.com/blog-posts/how-to-avoid-docker-hub-rate-limits

[steves-hes]

@manulifeam-rickhui
I was wondering the same thing. Why did this start affecting us in the past couple of days?

@alexisseurin - thanks for the reply. I did purchase a docker hub pro subscription to get the 5000 pulls per day and that has seemed to fix our issue.