GitLab Continuous Integration enables you to fully automate image building, QA testing, and eventually, deploying the container to a review environment or even the production environment. But, as always, combining technology and full automation has its challenges, and one of them is cleaning up your "mess".
My pipeline looks like this:
- Stage 1 - build: Building my production Docker container and pushing it to the Docker registry
- Stage 2 - QA: Pull the Docker container and run some tests
- Stage 3 - Review: Deploy the Docker container to the review environment
- Stage 4 - Production: Deploy the Docker container to the production environment (only "master")
Stage 3, deploying a Docker container to a review environment is used to QA test the application by the product owner and other stakeholders before it goes into "master". The downside of this approach is that we are forced to push all images to the Docker registry because you don't really know which Docker image is used for the review environment. And when you work with a lot of people on separate features in separate branches, you can imagine that the number of pipelines can become huge which results in a lot of development images in the Docker registry.
Unfortunately, the Docker Registry has only an API. The Docker command-line tool is only capable of pushing and pulling images from the Docker registry but isn't capable of removing them. End result: a full disk and a lot of outdated images.
GitLab implements a GUI for the Docker registry. This can be used when you have a small number of Docker images. But if you're building 10 images an hour, manually removing them using the GitLab Docker registry GUI is just not working. Solution: Use the Docker registry API to delete images.
Retrieve an authentication token
I found out, it is quite hard to find documentation how to retrieve an API token from Gitlab when you're using the GitLab JWT authentication. But after some 2-3 hours of "Googling", I found some documentation on the Docker Docs about the token authentication implementation in the Docker registry. The scope is also very important, and details about the scope I found on the token scope documentation page.
End result using curl:
curl --user 'username:password' 'https://gitlab.domain.com/jwt/auth?client_id=docker&offline_token=true&service=container_registry&scope=repository:your-repo-name:push,pull'
Some explanation:
- --user is used for basic authentication using your personal account credentials
- https://gitlab.domain.com/jwt/auth is the JWT authentication URL
- The client_id is always docker
- offline_token should be true (but I couldn't really figure out what it does...)
- service is always container_registry
- GitLab only supported the repository scope (as far I could test). Example: repository:customer/application:push,pull
At the moment, only push and pull are supported. Otherwise, you won't retrieve a valid token. This is an issue because this won't allow you to delete images... An issue and merge request are already created to resolve this in GitLab 9.5.
The result of the call will be a JSON string containing a token. This token will be used as a bearer token in the communication with the Docker registry API.
Use the Docker registry API to retrieve a list of tags
The Docker registry API has a set of endpoints. One of them is to retrieve a list of tags for an image:
curl -vvv -H 'Accept: application/vnd.docker.distribution.manifest.v2+json' -H 'Authorization: Bearer your-token' https://registry.domain.com:5000/v2/your-repo-name/tags/list
Some explanation:
- -vvv is to debug the request
- We'll set the correct content-type for this request using -H 'Accept: application/vnd.docker.distribution.manifest.v2+json'
- Because our Docker registry requires authentication, we pass the token retrieved in the previous request using -H 'Authorization: Bearer your-token'
- The base URL to the Docker registry is https://registry.domain.com:5000/v2/
- And to retrieve a list of tags for a specific repository and image, use your-repo-name/tags/list
The result of this call is again a JSON string containg a list of tags.
To retrieve more details about a tag your can use the following call (you need this to retrieve the tag reference):
curl -vvv -H 'Accept: application/vnd.docker.distribution.manifest.v2+json' -H 'Authorization: Bearer your-token' https://registry.domain.com:5000/v2/your-repo-name/manifests/tag-reference
Remove a tag or image
At the moment, the call is not possible in conjunction with GitLab 9.4 or earlier because of the bug previously mentioned. But, when that bug is fixed, you can remove a tag using the following call:
curl -vvv -X DELETE -H 'Accept: application/vnd.docker.distribution.manifest.v2+json' -H 'Authorization: Bearer your-token' https://registry.domain.com:5000/v2/your-repo-name/manifests/tag-reference
A full explanation about the available API endpoints is available at the Docker docs: https://docs.docker.com/registry/spec/api/
Thanks for reading, and if you have questions or feedback, leave a comment!