r/docker • • 10d ago

Question over general docker usage/concept

Greetings,

I would like to share a problem that I'm facing with Docker, looking for some ideas.

In our company we've started to use docker hosted in an Azure DevOps cloud ubuntu machine, there we have an Azure self-hosted agent installed. We work only with firmware projects and this is our build server. We have this setup for 3 years.

Usually, I generate the docker images on demand according with the project requirements.
In our firmware projects hosted in git we have a yaml pipeline configuration where we set the image tag so that the proper container can be raised to process the build.

We have a requirement from safety standards (we must keep the project artifacts/infrastructure from 10 to 15 years), because of that I keep all the dependencies I need to reproduce/re-create an image as debian installers (for offline installation). I've created a script which runs prior the build process, which creates a yaml manifest containing all the image packages and include this manifest also in the image (at /etc/manifest.yaml) for runtime check. Then it's possible to verify the content via the manifest hash. Examples of dependencies we have are cmake, unit test tools, static code analyzer, and so on.
We decided to keep everything in a single image because most of the tools require the compiler, so that the image complexity keeps lower as possible.

As we have different projects using the same images, we keep the Dockerfiles under version control, where the image name is the compiler version (the most significant dependency for our projects), something like this:

docker-<compiler>-<version>
  ubuntu-24.04
    Dockerfile
    build-image.sh
    manifest.yaml
  ubuntu-22.04
    Dockerfile
    ...

When a new image is generated, I tag the commit with the same tag I use in the container (e.g commit tag ubuntu24.04-v1.0-0, for a created container named gcc-13.3.0:ubuntu24.04-v1.0-0).

As our concept is based on tags, and the docker tags are not the most reliable thing in the world, I keep also the tagged image name as a LABEL within the image.

At this point I ask myself if we are going in the right direction and the increased complexity in the toolchain management is because of the concept we've decided to go for, or if it's just what we have to do in order to be compliant with such safety requirements. Or even I ask if we are drifting from the docker concept or using the tool in a wrong way.

I would really like to hear opinions, especially from other colleagues in similar situation.

Thanks in advance.

Note: I know that I can save the image in a registry, in parallel I'm trying to find solutions (if possible) to reduce storage costs (each image ~20GB).

4 Upvotes

11 comments sorted by

2

u/Lumethys 10d ago

You sure you understand what an image is? Why do you need to "keep track of artifact used to create an image"?

An image IS the artifact used to create a container.

It's like saying "I need to create a version control tool to so that i can re-create my git repo history"

1

u/vaugsantos90 10d ago

We keep track of the artifacts to reproduce the container in the future, if needed.
Of course we can store the image and load it, but given the amount of images we may have in the timespan of 15 years, we want to lower the storage costs.

4

u/xanyook 10d ago

Your pipeline should push your image in a registry like Nexus for example. Then just backup Nexus VM for your 15 years retention with your RTO/RPO policy.

1

u/Perfect-Escape-3904 10d ago

I’m not in the same position as you, but maybe the way I see it is that docker should be a great asset. Why worry about labels etc, can you not just keep the built image?

Or is your issue more about how you tie the built images to certain ci runs so you know what image built what code?

2

u/vaugsantos90 10d ago

We have also this option, but anyways we want to reproduce the images, to reduce the storage costs (each image has like 20GB - proprietary compiler and other tools).

2

u/crackjiver 10d ago

If you don't already have one, look at getting an artifact repository tool like JFrog Artifactory.

It can host all the OS packages and dependencies in an efficient storage medium. The files are stored in a flat path structure and named with the checksum of the content so it's only stored once, despite being linked to multiple download paths.

It is also a docker/oci container registry, build record, SBOM and vulnerability scanner. I love it.

It's annoying when you come to rebuild an old image based on Ubuntu 14.04 and the apt lists are no longer supported. Being able to pull the debs out of artifactory cache is a life saver.

2

u/RobotJonesDad 9d ago

Do you pull ANYTHING from the internet during your image build? I've had stuff disappear completely and without warning.

If your images are built in a stack of images, then the disk space is tremendously smaller because most images will share most layers inside docker. You take your base layer OS, then use that to create an image with all the tools. Build the final image on top of that. Now your individual images are really just the size difference between final images, but the bulk of the stuff is the same. Store them in a suitable registry and they will only store each layer once, regardless of how many images you have.

Now your images capture everything you need with no external dependencies.

1

u/Perfect-Escape-3904 10d ago

How many different bytes per image, you could look at archive options that deduplicate, assuming from build current to build vnext you have 1% rate of change

1

u/crackjiver 10d ago

Having a pattern that's easy to follow is good.

Having config/build process in source control is good

Having build dependencies captured is good.

Having git tags that relate to the container images is good, having them be immutable would be better.

We use the 'git describe --tags --long --always' value when tagging container images because it includes part of the commit hash string and can't be moved or deleted.

This way the container images tag is directly linked with an immutable git reference so that, in theory, it can be rebuilt from scratch if needed.

Compliance, over long time scales, is tricky because things change rapidly in computing. There's more than one container standard. Is Docker going to be available in 10 years? Are your x86 686 Compilers from 10 years ago still able to run on today's hardware (we had to wrap them in "set arch" commands to convince them that data types were still 32bit compatible)

2

u/vaugsantos90 10d ago

The usage of LABELS with the image tag is an attempt to make the relationship between image and repository most reliable possible.
Regarding the compliance, that's indeed a challenge, but unfortunately it's a regulatory thing we have to be compliant with. For the old projects we used to keep old laptop/PCs/virtual machines to achieve such compliance, but backup of Docker images seems to be better idea ;)

2

u/Fun-Adhesiveness-362 10d ago

Hi there!

I don't see a problem with keeping the dependencies needed for an offline rebuild. That perfectly makes sense.

One thing I'd change is how you identify the image used by each build. Copying the tag into a LABEL doesn't stop that tag from pointing to another image later. Keep the readable tag, but record the image digest alongside the source commit in your build records. At least that's how we do it in our own stacks.

For rebuilding offline, keep the base image as well as the Dockerfile and dependency installers. Pinning FROM to a digest identifies the exact version, but you still need a retained copy if it disappears from the original registry.

If you have any questions or it's not clear what I suggested, please let me know and I'd be more than happy to jump into more technical details.