p14nd4's blog is the best blog ever.

I’ve recently been using self-hosted Gitlab at work, particularly for its CI/CD features. We have a few projects using it already, and I’m trying to expand that utilization. It all went pretty well until I tried building with a new Docker image, and got an error:

ERROR: Job failed: failed to pull image "[..]:latest" with specified policies [always]: Error response from daemon: Head "https://[..]:5050/v2/[..]/manifests/latest": denied: access forbidden (manager.go:237:0s)

The project I most recently set my sights on requires some external tools/resources for the build. While they’re publicly available, the prospect of adding a build script step to download and extract them was unpalatable (waste of bandwidth and time), and while I toyed with the idea of trying to hack them into build cache restoration, I finally relented and just built a new Dockerfile / docker image, and pushed the image to the project’s container repository. (Incidentally, I would’ve loved if .gitlab-ci.yml understood Docker Compose, and I could just mount a volume containing these resources on top of an upstream-maintained/updated Docker image, so please let me know if I missed that or it gets added.)

The Problem

All of that got me to the point of trying to run the build for the first time, which failed quickly with a log like this:

Running with gitlab-runner 16.2.0 (782e15da)
  on gitlab-runner02 Ua6jigiQ, system ID: r_tRd8aLibbXym
Preparing the "docker" executor 00:03
Using Docker executor with image [gitlabHostName]:5050/[project/image path]:latest ...
Pulling docker image [gitlabHostName]:5050/[project/image path]:latest ...
WARNING: Failed to pull image with policy "always": Error response from daemon: Head "https://[gitlabHostName]:5050/v2/[project/image path]/manifests/latest": denied: access forbidden (manager.go:237:0s)
ERROR: Job failed: failed to pull image "[gitlabHostName]:5050/[project/image path]:latest" with specified policies [always]: Error response from daemon: Head "https://[gitlabHostName]:5050/v2/[project/image path]/manifests/latest": denied: access forbidden (manager.go:237:0s)

After trying several things and banging my head on the wall a bit, I eventually spotted a difference between this and a working build: the presence of this line before ‘pulling docker image’:

Authenticating with credentials from job payload (GitLab Registry)

The Solution

Additional searching and scrutinizing the config revealed that in a working build, the image repository path hostname was lowercase, and in the new/broken build, it had capitalization. Sure enough, when I replaced the “gitlabHostName” with “gitlabhostname” (equivalent), the new build started “Authenticating with credentials from job payload (GitLab Registry)” as well, and was able to complete the build.

So, if you’re butting your head against Failed to pull image … denied: access forbidden errors in Gitlab while trying to pull an image from your container registry, check to make sure the hostname specified in the .gitlab-ci.yml image path matches your actual Gitlab container repository and is lower-case. (After seeing this only work with the lower-cased hostname, I was surprised to find that the value in my gitlab.rb registry_external_url does have the capitalization found in the broken build, so it would seem that it’s not even a matter of matching that value, but necessarily being lower-case.)

I’ve recently been debugging an old project I inherited that features a [mostly] Java server and C++ client. I think it was mostly written 10–15 years ago (Java 6 at the latest), and probably not touched a whole lot since then, so while it’s all too easy to complain about any code, let alone old code, and code written by anyone else (and I will), I do also have respect for the fact that it’s moderately complex and has generally worked for a long time.

Although there’s plenty of low-hanging fruit for improvement in the project, the killer has been an issue seen when the system isn’t started up or shut down in its preferred sequence and speed. I never identified a perfect recipe for reproducing the issue, but basically: sometimes the system would get into a state where some clients would connect, but then the server seemed to essentially stop letting anyone new in. The workaround was to stop all clients (luckily the clients in this case were actually other servers within our [customers’] control), stop the server, start the server, and then start up the clients one by one. This is annoying with four servers; it’s a major pain with 40. (For better or worse, nobody currently has more than that.)
Read more »

I just spent a while fighting an exception that was being thrown from within a .NET runtime-generated assembly—specifically from Microsoft.Xml.Serialization.GeneratedAssembly.XmlSerializationWriter{MyClassName}.Write7_{MyClassName}. There were plenty of search results that didn’t apply to my problem, but I didn’t find any that directly addressed the issue I was encountering, so I wanted to write up a quick post to possibly save someone some time in the future.

Read more »

One of the sites I run, Techware Labs, uses WordPress for its CMS. (As an aside, we previously used our own home-grown CMS, which had a variety of benefits, but didn’t provide quite as much flexibility as we wanted. Rather than putting significant effort into rewriting or massively upgrading it, we decided to leave that heavy lifting to the pros.) We began using WP around May, 2009 (v2.7.1), since which point 17 major releases have occurred (and significantly more minor releases). We selected the Multi-Page Toolkit plugin to paginate articles (since WordPress just sticks the entire post on a single page by default), which seemed reasonable at the time—it was relatively popular, and had been updated a few months prior. Unfortunately, that update a few months prior [to early 2009] is the last update the plugin has received at the time of this writing.

A lot can happen in seven-plus years, especially in a popular software project like WordPress, but miraculously this plugin mostly kept working through it all. Mostly, however, isn’t quite the same as entirely, and some people get cranky when one thing goes wrong. (Take the Hindenburg, for example; why doesn’t anyone talk about its 62 successful flights?)

Read more »

I recently purchased a Super Micro X7SPA-HF-D525 motherboard, and was excited to try out the IP-KVM feature. While I was able to log into the web interface fine, launching the console redirection java webapplet partially proceeded, but eventually failed for me with an error no iKVM64 in java.library.path. I was able to experiment a bit and developed a patch (below) for the launch.jnlp file served for the iKVM applet that enabled this to run on my system (i.e. downloaded the launch.jnlp, patched it, then ran with javaws launch.jnlp). When run without the patch, Java fails to download liblinux_x86_64.jar for me, because it only tries the following two URLs, which don’t contain the relevant file: https://192.168.0.207:443/liblinux_x86_64.jar?version-id=1.0.3 and https://192.168.0.207:443/liblinux_x86_64.jar (while the actual path is something like /liblinux_x86_64__V1.0.3.jar.pack.gz).

Read more »

I’ll make this brief, but wanted to document my fix to potentially save someone else some aggravation in the future.

I’ve been working on a c++ project utilizing Google protobuf, and at one point ran into an error during application startup (static initialization):

libprotobuf ERROR google/protobuf/descriptor_database.cc:314] Invalid file descriptor data passed to EncodedDescriptorDatabase::Add().
libprotobuf FATAL google/protobuf/descriptor.cc:862] CHECK failed: generated_database_->Add(encoded_file_descriptor, size):
Abort (core dumped)

Read more »