p14nd4's blog is the best blog ever.

I’ve recently been using self-hosted Gitlab at work, particularly for its CI/CD features. We have a few projects using it already, and I’m trying to expand that utilization. It all went pretty well until I tried building with a new Docker image, and got an error:

ERROR: Job failed: failed to pull image "[..]:latest" with specified policies [always]: Error response from daemon: Head "https://[..]:5050/v2/[..]/manifests/latest": denied: access forbidden (manager.go:237:0s)

The project I most recently set my sights on requires some external tools/resources for the build. While they’re publicly available, the prospect of adding a build script step to download and extract them was unpalatable (waste of bandwidth and time), and while I toyed with the idea of trying to hack them into build cache restoration, I finally relented and just built a new Dockerfile / docker image, and pushed the image to the project’s container repository. (Incidentally, I would’ve loved if .gitlab-ci.yml understood Docker Compose, and I could just mount a volume containing these resources on top of an upstream-maintained/updated Docker image, so please let me know if I missed that or it gets added.)

The Problem

All of that got me to the point of trying to run the build for the first time, which failed quickly with a log like this:

Running with gitlab-runner 16.2.0 (782e15da)
  on gitlab-runner02 Ua6jigiQ, system ID: r_tRd8aLibbXym
Preparing the "docker" executor 00:03
Using Docker executor with image [gitlabHostName]:5050/[project/image path]:latest ...
Pulling docker image [gitlabHostName]:5050/[project/image path]:latest ...
WARNING: Failed to pull image with policy "always": Error response from daemon: Head "https://[gitlabHostName]:5050/v2/[project/image path]/manifests/latest": denied: access forbidden (manager.go:237:0s)
ERROR: Job failed: failed to pull image "[gitlabHostName]:5050/[project/image path]:latest" with specified policies [always]: Error response from daemon: Head "https://[gitlabHostName]:5050/v2/[project/image path]/manifests/latest": denied: access forbidden (manager.go:237:0s)

After trying several things and banging my head on the wall a bit, I eventually spotted a difference between this and a working build: the presence of this line before ‘pulling docker image’:

Authenticating with credentials from job payload (GitLab Registry)

The Solution

Additional searching and scrutinizing the config revealed that in a working build, the image repository path hostname was lowercase, and in the new/broken build, it had capitalization. Sure enough, when I replaced the “gitlabHostName” with “gitlabhostname” (equivalent), the new build started “Authenticating with credentials from job payload (GitLab Registry)” as well, and was able to complete the build.

So, if you’re butting your head against Failed to pull image … denied: access forbidden errors in Gitlab while trying to pull an image from your container registry, check to make sure the hostname specified in the .gitlab-ci.yml image path matches your actual Gitlab container repository and is lower-case. (After seeing this only work with the lower-cased hostname, I was surprised to find that the value in my gitlab.rb registry_external_url does have the capitalization found in the broken build, so it would seem that it’s not even a matter of matching that value, but necessarily being lower-case.)