Gerwim's blog

Enhancing Docker build pipelines based on external tags

2020-11-03 .net docker

Introduction

In my last blog post I’ve described how you would use process isolation in your .NET project.

However, it caused a new problem: you would need to manage multiple Docker images for your own images, e.g.:

  • myregistry.com/mssql/windows:ltsc2016
  • myregistry.com/mssql/windows:ltsc2019
  • myregistry.com/mssql/windows:1903
  • myregistry.com/mssql/windows:1909
  • myregistry.com/mssql/windows:2004
  • myregistry.com/mssql/windows:2009

So, instead of having a single image, you would need to have six times as much now. This post will give you some tools and hints on how to solve this problem.

⚠️ This post will talk about Microsoft and .NET, however the solution is platform and language agnostic. It can be used whether you are using .NET, Python, Linux, FreeBSD, Java etcetera. It does not matter. As long as you want to query a Docker registry API to base your images on.

Retrieving the available Docker tags

First, you would need to know what tags are available so you can populate your list. You could (and this is still a very valid solution!) have this array of tags somewhere locally or hardcoded in your build script. This will depend on your problem and your infrastructure. However, I assume you don’t have anything in place at all yet.

For this very reason (it might be overkill, but I like nice things 😊), I’ve developed an API (which is publicly available on https://api.gerwim.com/dockertags) which you can query to list the available tags for a specific image.

For now, it supports both the Docker Hub and the Microsoft Container Registry. You can also run this API yourself. It’s available as a Docker image and source code is on GitHub.

So, without further ado, here’s an example:

curl 'https://api.gerwim.com/dockertags/v1/tags' \
--header 'registry: mcr.microsoft.com' \
--header 'imageName: dotnet/framework/aspnet' \
--header 'searchRegex: ^4\.8-windowsservercore-(\d{4}|ltsc\d{4})'

will return:

{
    "name": "dotnet/framework/aspnet",
    "tags": [
        "4.8-windowsservercore-1803",
        "4.8-windowsservercore-1903",
        "4.8-windowsservercore-1909",
        "4.8-windowsservercore-2004",
        "4.8-windowsservercore-2009",
        "4.8-windowsservercore-ltsc2016",
        "4.8-windowsservercore-ltsc2019"
    ]
}

The curl command is just a basic HTTP GET. The Powershell equivalent would be:

Invoke-RestMethod -Uri "https://api.gerwim.com/dockertags/v1/tags" `
-Headers @{ `
    "registry" = "mcr.microsoft.com"; `
    "imageName" = "dotnet/framework/aspnet"; `
    "searchRegex" = "^4\.8-windowsservercore-(\d{4}|ltsc\d{4})" `
}
  • It does a HTTP GET to api.gerwim.com
  • Sets three headers: registry, imageName and searchRegex

registry: can be (as of writing) be hub.docker.com or mcr.microsoft.com
imageName: is the image name on the registry
searchRegex: is optional (defaults to (.*)) but will filter the tags based on the regex

Building an ASP.NET image based on Microsoft’s tags

Now, we have the list of available tags we can base our custom image on. Since Microsoft will maintain their list and deprecate older (no longer supported) Windows versions, so we do not have to worry about that.

Assuming you have a custom image, based on 4.8-windowsservercore, your FROM in your dockerfile would look something like this:

FROM mcr.microsoft.com/dotnet/framework/aspnet/4.8-windowsservercore:ltsc2019

We will change this to (if you read my last blog post, this might seem familiar to you 😉):

ARG WINDOWS_VERSION
FROM mcr.microsoft.com/dotnet/framework/aspnet/4.8-windowsservercore:$WINDOWS_VERSION

This means the Docker build command accepts an additional parameter, WINDOWS_VERSION. We can use this argument in our build script to define the tag.

To build and push the Docker images to your registry, we can use the following Powershell script (e.g. build.ps1):

param ([Parameter(Mandatory)]$imageName)

# Retrieve tags from the public API
$response = Invoke-RestMethod -Uri "https://api.gerwim.com/dockertags/v1/tags"-Headers @{"registry" = "mcr.microsoft.com"; "imageName" = "dotnet/framework/aspnet"; "searchRegex" = "^4\.8-windowsservercore-(\d{4}|ltsc\d{4})"}

foreach ($tag in $response.tags) {
    $split = $tag.Split('-'); # split the response (e.g. 4.8-windowsservercore-2009)
    $version = $split[$split.Count - 1]; # get the last string from the array (e.g. 2009)
    Write-Host [$version] Building image
    $imageNameWithVersion = $imageName.Replace("WINDOWS_VERSION", $version); # replace WINDOWS_VERSION by our version, e.g. 2009
    Start-Process -FilePath docker -ArgumentList "build --pull -t $imageNameWithVersion --build-arg WINDOWS_VERSION=$version ." -Wait -NoNewWindow

    # Push image
    Start-Process -FilePath docker -ArgumentList "push $imageNameWithVersion" -Wait -NoNewWindow
}

To run the above script:

powershell -File build.ps1 -imageName "myregistry.com/aspnet-image/windows/WINDOWS_VERSION:latest"

which will build eight images (as of writing):

  • myregistry.com/aspnet-image/windows/1803:latest
  • myregistry.com/aspnet-image/windows/1903:latest
  • myregistry.com/aspnet-image/windows/1909:latest
  • myregistry.com/aspnet-image/windows/2004:latest
  • myregistry.com/aspnet-image/windows/2009:latest
  • myregistry.com/aspnet-image/windows/ltsc2016:latest
  • myregistry.com/aspnet-image/windows/ltsc2019:latest

Real life example using GitLab CI

A real life example using GitLab CI (which is amazing!) is quite easy, but it will depend on your configuration. In my case:

  • I have a virtual machine running Windows Server Core 20H2.
  • Has Docker along with Hyper-V (nested virtualization) installed (which is required to build older versions than 20H2 (which is Windows release ID 2009). For more information, see the compatibility matrix).

My .gitlab-ci.yml file:

stages:
  - prepare
  - build

# generate a dynamic config file
generate-config:
  stage: prepare
  script: powershell -File createconfig.ps1 -imageName "%CI_REGISTRY_IMAGE%/windows/WINDOWS_VERSION:%CI_COMMIT_REF_SLUG%"
  artifacts:
    paths:
      - generated-config.yml

# build the child pipeline (this part will create the actual Docker images)
build-images:
  stage: build
  trigger:
    include:
      - artifact: generated-config.yml
        job: generate-config

My createconfig.ps1 file (which is a customized version of the build.ps1 script above):

param ([Parameter(Mandatory)]$imageName)

$response = Invoke-RestMethod -Uri "https://api.gerwim.com/dockertags/v1/tags"-Headers @{"registry" = "mcr.microsoft.com"; "imageName" = "dotnet/framework/aspnet"; "searchRegex" = "^4\.8-windowsservercore-(\d{4}|ltsc\d{4})"}
$configTemplate = @"
build-WINDOWS_VERSION:
  stage: build
  before_script:
    - docker login -u "%CI_REGISTRY_USER%" -p "%CI_REGISTRY_PASSWORD%" %CI_REGISTRY%
  script:
    - BUILD_COMMAND
    - PUSH_COMMAND
"@

foreach ($tag in $response.tags) {
    $split = $tag.Split('-');  # split the response (e.g. 4.8-windowsservercore-2009)
    $version = $split[$split.Count - 1]; # get the last string from the array (e.g. 2009)
    Write-Host [$version] Generating config
    # Start replacing magic strings
    $imageNameWithVersion = $imageName.Replace("WINDOWS_VERSION", $version);
    $template = $configTemplate.Replace("WINDOWS_VERSION", $version);
    $template = $template.Replace("BUILD_COMMAND", "docker build --pull -t $imageNameWithVersion --build-arg WINDOWS_VERSION=$version .");
    $template = $template.Replace("PUSH_COMMAND", "docker push $imageNameWithVersion");
    
    # Append the template variable (with the replaced strings) to a local file
    Add-Content -Path generated-config.yml -Value $template
}

This will result in a pipeline like: gitlab-pipeline.png

The most important part is the magic string WINDOWS_VERSION in the imageName parameter. This string, WINDOWS_VERSION, is replaced in the Powershell file and passed along as a Docker build argument to fetch and use the correct base image.

Of course, if you use a different CI solution (or don’t use Windows at all), you will need to use a different solution, but the gist is:

  • Get the available Docker tags using a HTTP GET
  • Build each image using the correct base version, by using an argument in the FROM line, passed by --build-arg. E.g.:

Dockerfile:

ARG WINDOWS_VERSION
FROM mcr.microsoft.com/dotnet/framework/aspnet/4.8-windowsservercore:$WINDOWS_VERSION

Build command:

docker build -t myimage --build-arg WINDOWS_VERSION=ltsc2019 .

What about non Microsoft or .NET?

In the example above the argument WINDOWS_VERSION is used, but it could be anything and has nothing to do with Microsoft or Windows itself. It’s called WINDOWS_VERSION for clarity.

This solution could be used to build your custom image based on anything. E.g. build against all available Ubuntu LTS tags:

curl 'https://api.gerwim.com/dockertags/v1/tags' \
--header 'registry: hub.docker.com' \
--header 'imageName: ubuntu' \
--header 'searchRegex: (^1\d*[68]\.04$)|(^2\d*[02468]\.04$)'

returns:

{
  "name": "ubuntu",
  "tags": [
    "16.04",
    "18.04",
    "20.04"
  ]
}

which would be used in your build script.

Conclusion

Hopefully you gained insight on how to build your Docker images based on external tags so you can further enhance your build pipeline(s) 🎉.

Have any questions or have feedback? Let me know on Twitter!

© 2024 Gerwim Feiken