p14nd4's blog is the best blog ever.

I’ve recently been using self-hosted Gitlab at work, particularly for its CI/CD features. We have a few projects using it already, and I’m trying to expand that utilization. It all went pretty well until I tried building with a new Docker image, and got an error:

ERROR: Job failed: failed to pull image "[..]:latest" with specified policies [always]: Error response from daemon: Head "https://[..]:5050/v2/[..]/manifests/latest": denied: access forbidden (manager.go:237:0s)

The project I most recently set my sights on requires some external tools/resources for the build. While they’re publicly available, the prospect of adding a build script step to download and extract them was unpalatable (waste of bandwidth and time), and while I toyed with the idea of trying to hack them into build cache restoration, I finally relented and just built a new Dockerfile / docker image, and pushed the image to the project’s container repository. (Incidentally, I would’ve loved if .gitlab-ci.yml understood Docker Compose, and I could just mount a volume containing these resources on top of an upstream-maintained/updated Docker image, so please let me know if I missed that or it gets added.)

The Problem

All of that got me to the point of trying to run the build for the first time, which failed quickly with a log like this:

Running with gitlab-runner 16.2.0 (782e15da)
  on gitlab-runner02 Ua6jigiQ, system ID: r_tRd8aLibbXym
Preparing the "docker" executor 00:03
Using Docker executor with image [gitlabHostName]:5050/[project/image path]:latest ...
Pulling docker image [gitlabHostName]:5050/[project/image path]:latest ...
WARNING: Failed to pull image with policy "always": Error response from daemon: Head "https://[gitlabHostName]:5050/v2/[project/image path]/manifests/latest": denied: access forbidden (manager.go:237:0s)
ERROR: Job failed: failed to pull image "[gitlabHostName]:5050/[project/image path]:latest" with specified policies [always]: Error response from daemon: Head "https://[gitlabHostName]:5050/v2/[project/image path]/manifests/latest": denied: access forbidden (manager.go:237:0s)

After trying several things and banging my head on the wall a bit, I eventually spotted a difference between this and a working build: the presence of this line before ‘pulling docker image’:

Authenticating with credentials from job payload (GitLab Registry)

The Solution

Additional searching and scrutinizing the config revealed that in a working build, the image repository path hostname was lowercase, and in the new/broken build, it had capitalization. Sure enough, when I replaced the “gitlabHostName” with “gitlabhostname” (equivalent), the new build started “Authenticating with credentials from job payload (GitLab Registry)” as well, and was able to complete the build.

So, if you’re butting your head against Failed to pull image … denied: access forbidden errors in Gitlab while trying to pull an image from your container registry, check to make sure the hostname specified in the .gitlab-ci.yml image path matches your actual Gitlab container repository and is lower-case. (After seeing this only work with the lower-cased hostname, I was surprised to find that the value in my gitlab.rb registry_external_url does have the capitalization found in the broken build, so it would seem that it’s not even a matter of matching that value, but necessarily being lower-case.)

I’ve recently been debugging an old project I inherited that features a [mostly] Java server and C++ client. I think it was mostly written 10–15 years ago (Java 6 at the latest), and probably not touched a whole lot since then, so while it’s all too easy to complain about any code, let alone old code, and code written by anyone else (and I will), I do also have respect for the fact that it’s moderately complex and has generally worked for a long time.

Although there’s plenty of low-hanging fruit for improvement in the project, the killer has been an issue seen when the system isn’t started up or shut down in its preferred sequence and speed. I never identified a perfect recipe for reproducing the issue, but basically: sometimes the system would get into a state where some clients would connect, but then the server seemed to essentially stop letting anyone new in. The workaround was to stop all clients (luckily the clients in this case were actually other servers within our [customers’] control), stop the server, start the server, and then start up the clients one by one. This is annoying with four servers; it’s a major pain with 40. (For better or worse, nobody currently has more than that.)
Read more »

I just spent a while fighting an exception that was being thrown from within a .NET runtime-generated assembly—specifically from Microsoft.Xml.Serialization.GeneratedAssembly.XmlSerializationWriter{MyClassName}.Write7_{MyClassName}. There were plenty of search results that didn’t apply to my problem, but I didn’t find any that directly addressed the issue I was encountering, so I wanted to write up a quick post to possibly save someone some time in the future.

Read more »

After pfSense upgraded to a newer FreeBSD base (perhaps a year ago), my Thinkpad T41 running pfSense was failing to boot; I believe the errors were something about failing to initialize the device at ata1 (the optical drive); perhaps something about failing to respond to a command. After lots of searching (and poor documentation), I resolved the issue by setting this boot parameter:

hint.ata.1.disabled=1

Once the system booted, I believe I added this to /boot/loader.conf.local to make the fix [slightly more] permanent. (I believe this file still gets wiped out during system updates, so it’s not a perfect solution.)

I’ll update this post with the exact error message if I encounter it again. (I’m finally getting around to writing this post since I’m about to wipe pfSense and try OPNsense, and I don’t want to have to re-discover this solution.)

Edit: As I suspected, OPNsense has the same problem:

(ataprobe0:ata1:0:1:0): ATAPI_IDENTIFY. ACB: a1 00 00 00 00 40 00 00 00 00 00 00
(ataprobe0:ata1:0:1:0): CAM status: Command timeout
(ataprobe0:ata1:0:1:0): Retrying command
(ataprobe0:ata1:0:1:0): ATAPI_IDENTIFY. ACB: a1 00 00 00 00 40 00 00 00 00 00 00
(ataprobe0:ata1:0:1:0): CAM status: Command timeout
(ataprobe0:ata1:0:1:0): Error 5, Retries exhausted

For a demo project I had at work recently, I wanted to incorporate some imagery from the client’s web site. I discovered their web server was configured to generate an index page for the directory containing their images, but I didn’t want to click through each one individually to figure out if it was what I wanted. (An auto-generated index page is something like this, but imagine it having hundreds of entries.)

I was pretty sure that I’d encountered some javascript many years ago that would render a thumbnail gallery for just such an occasion, but I was unable to locate it. Thankfully, I’m a software engineer, and I’m easily distracted by tangential pursuits, so I just wrote my own! If you’re not a nerd, but want to see what it is, copy the text below, visit the index page mentioned above; in your browser address bar, delete the address that’s currently there, manually type javascript: (make sure to get the colon), and then paste the text from below. (Browsers strip the javascript: from the text if you paste it.)

If it works (tested in Chromium 47 and Firefox 43), you should be presented with a little gallery of images! You can also search the web for index of jpg or something similar and test it out there, but you can stumble across some…interesting…things that way. You’ve been warned.

Also, it’s got the potential to crash your browser (or at least really slow things down) if you run it in a really large index. For better or for worse, I don’t do any intelligent loading/unloading of images, so it just ends up rendering a page with ALL OF THE IMAGES from the index. I have no doubt that this can be dramatically improved in a variety of ways; I very infrequently touch JavaScript, so this was a bit of a refresher exercise for me. Pull requests welcome!

One of the sites I run, Techware Labs, uses WordPress for its CMS. (As an aside, we previously used our own home-grown CMS, which had a variety of benefits, but didn’t provide quite as much flexibility as we wanted. Rather than putting significant effort into rewriting or massively upgrading it, we decided to leave that heavy lifting to the pros.) We began using WP around May, 2009 (v2.7.1), since which point 17 major releases have occurred (and significantly more minor releases). We selected the Multi-Page Toolkit plugin to paginate articles (since WordPress just sticks the entire post on a single page by default), which seemed reasonable at the time—it was relatively popular, and had been updated a few months prior. Unfortunately, that update a few months prior [to early 2009] is the last update the plugin has received at the time of this writing.

A lot can happen in seven-plus years, especially in a popular software project like WordPress, but miraculously this plugin mostly kept working through it all. Mostly, however, isn’t quite the same as entirely, and some people get cranky when one thing goes wrong. (Take the Hindenburg, for example; why doesn’t anyone talk about its 62 successful flights?)

Read more »

I got curious today about the CA certificates present on my current phone, the Sony Xperia Z5 Compact (E5823), running Android 5.1.1.

It turns out it’s pretty easy to pull the CA certs from an Android device (even easier than before). Assuming you have adb set up and working with your device already:
$ adb pull /system/etc/security/cacerts cacerts

The certificates are now stored in standard PEM format, so they’re super easy to use in formats we want. I’m running an Ubuntu 15.10 system currently, so I’m using that as a baseline to ‘diff’ the Android CA certs against (using an admittedly cursory check to just match the first line of the cert):
$ for i in cacerts/* ; do if ! $(grep -q $(sed -n 2p $i) /etc/ssl/certs/ca-certificates.crt) ; then echo $i ; fi ; done

This turned up a list of six root certificates trusted by my phone that aren’t trusted by my desktop:
Read more »

I recently tried to use the ELinks console-based web browser with Tor. I found what seemed to be the easiest way to this: torify elinks, but that turned into a journey down the rabbit-hole. (A wiser man than I likely would’ve just used the alternate method suggested there—privoxy—but for the next one of me who comes along, here’s what I found.)
Read more »

Having followed this guide to mounting a box.com account in linux, I tried rsyncing a bunch of photos to my account. It kept bombing out partway through, and subsequent attempts to resume the rsync would re-transfer the same files over and over again.

rsync: rename "/mnt/box/bwca/08/09/.IMG_3084_CR2_embedded.jpg.AigcyF" -> "08/09/IMG_3084_CR2_embedded.jpg": Input/output error (5)
rsync: rename "/mnt/box/bwca/08/09/.IMG_3085.CR2.SEk0pG" -> "08/09/IMG_3085.CR2": Input/output error (5)
rsync: mkstemp "/mnt/box/bwca/08/09/.IMG_3085_CR2_embedded.jpg.injOBG" failed: Permission denied (13)
rsync: mkstemp "/mnt/box/bwca/08/09/.IMG_3086_CR2.jpg.OzVzik" failed: Permission denied (13)

I haven’t solved the issue with rsync dying before the end of the transfer (it’s probably related to this syslog message: mount.davfs: open files exceed max cache size by 25 MiBytes), but I have at least solved the issue with re-transferring files over and over again with each subsequent attempt. The two answers were pretty straightforward, but I figured I’d post them anyway, for posterity’s sake.

Read more »

If you ready my previous post, you already know that I’ve had frustrating issues with lock-ups in Ubuntu 12.04 on my new Dell OptiPlex 9010, and perhaps related oddities with errors reported in memtest86+. I opened a support ticket with Dell, and they shipped me a replacement system, which arrived today. The spoiler: it didn’t fix my problems. However, in the course of determining that fact, I may have further isolated the issue.

Read more »