mirror of https://github.com/apache/kafka.git
KAFKA-15881: Update Release Process for Docker images documentation (#14990)
Reviewers: Manikumar Reddy <manikumar.reddy@gmail.com>
This commit is contained in:
Vedarth Sharma
GitHub
parent
5b478aebfd
commit
c5ee82cab4
125
docker/README.md
125
docker/README.md
|
|
@ -1,14 +1,71 @@
|
|||
Docker Images
|
||||
=============
|
||||
|
||||
Introduction
|
||||
------------
|
||||
|
||||
This directory contains scripts to build, test, push and promote docker image for kafka.
|
||||
All fo the steps can be either performed locally or by using Github Actions.
|
||||
|
||||
Github Actions
|
||||
--------------
|
||||
|
||||
- Github Actions can be accessed in Actions tab of the repository.
|
||||
- Please ensure that whichever Github Action Workflow you are triggering, you are doing it for the correct `branch`.
|
||||
- For all Github Actions Workflows for docker images, you'll get the option to select the branch as a dropdown box.
|
||||
|
||||
Repository Setup
|
||||
----------------
|
||||
|
||||
Make sure the `DOCKERHUB_USER` and `DOCKERHUB_TOKEN` secrets are added and made available to Github Actions in Github Repository settings. This is required for pushing the docker image.
|
||||
|
||||
Bulding image and running tests using github actions
|
||||
----------------------------------------------------
|
||||
|
||||
- This is the recommended way to build, test and get a CVE report for the docker image.
|
||||
- Just choose the image type and provide kafka url to `Docker Build Test` workflow. It will generate a test report and CVE report that can be shared with the community.
|
||||
|
||||
- kafka-url - This is the url to download kafka tarball from. For example kafka tarball url from (https://archive.apache.org/dist/kafka). For building RC image this will be an RC tarball url.
|
||||
|
||||
- image-type - This is the type of image that we intend to build. This will be dropdown menu type selection in the workflow. `jvm` image type is for official docker image (to be hosted on apache/kafka) as described in [KIP-975](https://cwiki.apache.org/confluence/display/KAFKA/KIP-975%3A+Docker+Image+for+Apache+Kafka)
|
||||
|
||||
- Example:-
|
||||
To build and test a jvm image type ensuring kafka to be containerised should be https://archive.apache.org/dist/kafka/3.6.0/kafka_2.13-3.6.0.tgz (it is recommended to use scala 2.13 binary tarball), following inputs in github actions workflow are recommended.
|
||||
```
|
||||
image_type: jvm
|
||||
kafka_url: https://archive.apache.org/dist/kafka/3.6.0/kafka_2.13-3.6.0.tgz
|
||||
```
|
||||
|
||||
Creating a Release Candidate using github actions
|
||||
-------------------------------------------------
|
||||
|
||||
- This is the recommended way to push an RC docker image.
|
||||
- Go to `Build and Push Release Candidate Docker Image` Github Actions Workflow.
|
||||
- Choose the `image_type` and and provide `kafka_url` that needs to be containerised in the `rc_docker_image` that will be pushed to github.
|
||||
- Example:-
|
||||
If you want to push a jvm image which contains kafka from https://archive.apache.org/dist/kafka/3.6.0/kafka_2.13-3.6.0.tgz to dockerhub under the namespace apache, repo name as kafka and image tag as 3.6.0-rc1 then following values need to be added in Github Actions Workflow:-
|
||||
```
|
||||
image_type: jvm
|
||||
kafka_url: https://archive.apache.org/dist/kafka/3.6.0/kafka_2.13-3.6.0.tgz
|
||||
rc_docker_image: apache/kafka:3.6.0-rc0
|
||||
```
|
||||
|
||||
Promoting a Release Candidate using github actions
|
||||
--------------------------------------------------
|
||||
|
||||
- This is the recommended way to promote an RC docker image.
|
||||
- Go to `Promote Release Candidate Docker Image` Github Actions Workflow.
|
||||
- Choose the RC docker image (`rc_docker_image`) that you want to promote and where it needs to be pushed to (`promoted_docker_image`), i.e. the final docker image release.
|
||||
- Example:-
|
||||
If you want to promote apache/kafka:3.6.0-rc0 RC docker image to apache/kafka:3.6.0 then following parameters can be provided to the workflow.
|
||||
```
|
||||
rc_docker_image: apache/kafka:3.6.0-rc0
|
||||
promoted_docker_image: apache/kafka:3.6.0
|
||||
```
|
||||
|
||||
Local Setup
|
||||
-----------
|
||||
|
||||
Make sure you have python (>= 3.7.x) and java (>= 17) (java needed only for running tests) installed before running the tests and scripts.
|
||||
|
||||
Run `pip install -r requirements.txt` to get all the requirements for running the scripts.
|
||||
|
|
@ -17,92 +74,44 @@ Make sure you have docker installed with support for buildx enabled. (For pushin
|
|||
|
||||
Bulding image and running tests locally
|
||||
---------------------------------------
|
||||
|
||||
- `docker_build_test.py` script builds and tests the docker image.
|
||||
- kafka binary tarball url along with image name, tag and type is needed to build the image. For detailed usage description check `python docker_build_test.py --help`.
|
||||
- Sanity tests for the docker image are present in test/docker_sanity_test.py.
|
||||
- By default image will be built and tested, but if you only want to build the image, pass `--build` (or `-b`) flag and if you only want to test the given image pass `--test` (or `-t`) flag.
|
||||
- An html test report will be generated after the tests are executed containing the results.
|
||||
|
||||
Example command:-
|
||||
To build and test an image named test under kafka namespace with 3.6.0 tag and jvm image type ensuring kafka to be containerised should be https://downloads.apache.org/kafka/3.6.0/kafka_2.13-3.6.0.tgz (it is recommended to use scala 2.13 binary tarball), following command can be used
|
||||
- Example :- To build and test an image named test under kafka namespace with 3.6.0 tag and jvm image type ensuring kafka to be containerised should be https://downloads.apache.org/kafka/3.6.0/kafka_2.13-3.6.0.tgz (it is recommended to use scala 2.13 binary tarball), following command can be used
|
||||
```
|
||||
python docker_build_test.py kafka/test --image-tag=3.6.0 --image-type=jvm --kafka-url=https://archive.apache.org/dist/kafka/3.6.0/kafka_2.13-3.6.0.tgz
|
||||
```
|
||||
|
||||
Bulding image and running tests using github actions
|
||||
----------------------------------------------------
|
||||
This is the recommended way to build, test and get a CVE report for the docker image.
|
||||
Just choose the image type and provide kafka url to `Docker Build Test` workflow. It will generate a test report and CVE report that can be shared with the community.
|
||||
|
||||
kafka-url - This is the url to download kafka tarball from. For example kafka tarball url from (https://archive.apache.org/dist/kafka). For building RC image this will be an RC tarball url.
|
||||
|
||||
image-type - This is the type of image that we intend to build. This will be dropdown menu type selection in the workflow. `jvm` image type is for official docker image (to be hosted on apache/kafka) as described in [KIP-975](https://cwiki.apache.org/confluence/display/KAFKA/KIP-975%3A+Docker+Image+for+Apache+Kafka)
|
||||
|
||||
Example command:-
|
||||
To build and test a jvm image type ensuring kafka to be containerised should be https://archive.apache.org/dist/kafka/3.6.0/kafka_2.13-3.6.0.tgz (it is recommended to use scala 2.13 binary tarball), following inputs in github actions workflow are recommended.
|
||||
```
|
||||
image_type: jvm
|
||||
kafka_url: https://archive.apache.org/dist/kafka/3.6.0/kafka_2.13-3.6.0.tgz
|
||||
```
|
||||
|
||||
Creating a Release Candidate
|
||||
----------------------------
|
||||
|
||||
- `docker_release.py` script builds a multi-architecture image and pushes it to provided docker registry.
|
||||
- Ensure you are logged in to the docker registry before triggering the script.
|
||||
- kafka binary tarball url along with image name (in the format `<registry>/<namespace>/<image_name>:<image_tag>`) and type is needed to build the image. For detailed usage description check `python docker_release.py --help`.
|
||||
|
||||
Example command:-
|
||||
To push an image named test under kafka dockerhub namespace with 3.6.0 tag and jvm image type ensuring kafka to be containerised should be https://archive.apache.org/dist/kafka/3.6.0/kafka_2.13-3.6.0.tgz (it is recommended to use scala 2.13 binary tarball), following command can be used. (Make sure you have push access to the docker repo)
|
||||
- Example:- To push an image named test under kafka dockerhub namespace with 3.6.0 tag and jvm image type ensuring kafka to be containerised should be https://archive.apache.org/dist/kafka/3.6.0/kafka_2.13-3.6.0.tgz (it is recommended to use scala 2.13 binary tarball), following command can be used. (Make sure you have push access to the docker repo)
|
||||
```
|
||||
# kafka/test is an example repo. Please replace with the docker hub repo you have push access to.
|
||||
|
||||
python docker_release.py kafka/test:3.6.0 --kafka-url https://archive.apache.org/dist/kafka/3.6.0/kafka_2.13-3.6.0.tgz
|
||||
```
|
||||
|
||||
Please note that we use docker buildx for preparing the multi-architecture image and pushing it to docker registry. It's possible to encounter build failures because of buildx. Please retry the command in case some buildx related error occurs.
|
||||
|
||||
Creating a Release Candidate using github actions
|
||||
-------------------------------------------------
|
||||
This is the recommended way to push an RC docker image.
|
||||
Go to `Build and Push Release Candidate Docker Image` Github Actions Workflow.
|
||||
Choose the `image_type` and and provide `kafka_url` that needs to be containerised in the `rc_docker_image` that will be pushed to github.
|
||||
|
||||
Example:-
|
||||
If you want to push a jvm image which contains kafka from https://archive.apache.org/dist/kafka/3.6.0/kafka_2.13-3.6.0.tgz to dockerhub under the namespace apache, repo name as kafka and image tag as 3.6.0-rc1 then following values need to be added in Github Actions Workflow:-
|
||||
```
|
||||
image_type: jvm
|
||||
kafka_url: https://archive.apache.org/dist/kafka/3.6.0/kafka_2.13-3.6.0.tgz
|
||||
rc_docker_image: apache/kafka:3.6.0-rc0
|
||||
```
|
||||
- Please note that we use docker buildx for preparing the multi-architecture image and pushing it to docker registry. It's possible to encounter build failures because of buildx. Please retry the command in case some buildx related error occurs.
|
||||
|
||||
Promoting a Release Candidate
|
||||
-----------------------------
|
||||
`docker_promote.py` provides an interactive way to pull an RC Docker image and promote it to required dockerhub repo.
|
||||
|
||||
Promoting a Release Candidate using github actions
|
||||
--------------------------------------------------
|
||||
This is the recommended way to promote an RC docker image.
|
||||
Go to `Promote Release Candidate Docker Image` Github Actions Workflow.
|
||||
Choose the RC docker image (`rc_docker_image`) that you want to promote and where it needs to be pushed to (`promoted_docker_image`), i.e. the final docker image release.
|
||||
- It's not recommended to promote the docker image locally, as we have github actions doing it in a convenient way, but if needed following command can be used to promote a Release Candidate image.
|
||||
- Example:- If you want to promote RC image apache/kafka:3.6.0-rc0 to apache/kafka:3.6.0, following command can be used
|
||||
|
||||
Example:-
|
||||
If you want to promote apache/kafka:3.6.0-rc0 RC docker image to apache/kafka:3.6.0 then following parameters can be provided to the workflow.
|
||||
```
|
||||
rc_docker_image: apache/kafka:3.6.0-rc0
|
||||
promoted_docker_image: apache/kafka:3.6.0
|
||||
# Ensure docker buildx is enabled in your system and you have access to apache/kafka
|
||||
docker buildx imagetools create --tag apache/kafka:3.6.0 apache/kafka:3.6.0-rc0
|
||||
```
|
||||
|
||||
Using the image in a docker container
|
||||
-------------------------------------
|
||||
Please check [this](./examples/README.md) for usage guide of the docker image.
|
||||
|
||||
Steps to release docker image
|
||||
-----------------------------
|
||||
- Make sure you have executed `release.py` script to prepare RC tarball in apache sftp server.
|
||||
- Use the RC tarball url (make sure you choose scala 2.13 version) as input kafka url to build docker image and run sanity tests.
|
||||
- Trigger github actions workflow using the RC branch, provide RC tarball url as kafka url.
|
||||
- This will generate test report and CVE report for docker images.
|
||||
- If the reports look fine, RC docker image can be built and published.
|
||||
- Execute `docker_release.py` script to build and publish RC docker image in your dockerhub account.
|
||||
- Share the RC docker image, test report and CVE report with the community in RC vote email.
|
||||
- Once approved and ready, take help from someone in PMC to trigger `docker_promote.py` script and promote the RC docker image to apache/kafka dockerhub repo
|
||||
Please check [this](./examples/README.md) for usage guide of the docker image.
|
||||
|
|
|
|||
|
|
@ -1,85 +0,0 @@
|
|||
#!/usr/bin/env python
|
||||
|
||||
# Licensed to the Apache Software Foundation (ASF) under one or more
|
||||
# contributor license agreements. See the NOTICE file distributed with
|
||||
# this work for additional information regarding copyright ownership.
|
||||
# The ASF licenses this file to You under the Apache License, Version 2.0
|
||||
# (the "License"); you may not use this file except in compliance with
|
||||
# the License. You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
"""
|
||||
Python script to promote an rc image.
|
||||
|
||||
Follow the interactive guide to pull an RC image and promote it desired dockerhub repository.
|
||||
|
||||
Usage: docker_promote.py
|
||||
|
||||
Interactive utility to promote a docker image
|
||||
"""
|
||||
|
||||
import requests
|
||||
from getpass import getpass
|
||||
from common import execute, get_input
|
||||
|
||||
def login():
|
||||
execute(["docker", "login"])
|
||||
|
||||
def pull(rc_image, promotion_image):
|
||||
execute(["docker", "pull", "--platform=linux/amd64", rc_image])
|
||||
execute(["docker", "tag", rc_image, f"{promotion_image}-amd64"])
|
||||
execute(["docker", "pull", "--platform=linux/arm64", rc_image])
|
||||
execute(["docker", "tag", rc_image, f"{promotion_image}-arm64"])
|
||||
|
||||
def push(promotion_image):
|
||||
execute(["docker", "push", f"{promotion_image}-amd64"])
|
||||
execute(["docker", "push", f"{promotion_image}-arm64"])
|
||||
|
||||
def push_manifest(promotion_image):
|
||||
execute(["docker", "manifest", "create", promotion_image,
|
||||
"--amend", f"{promotion_image}-amd64",
|
||||
"--amend", f"{promotion_image}-arm64"])
|
||||
|
||||
execute(["docker", "manifest", "push", promotion_image])
|
||||
|
||||
def remove(promotion_image_namespace, promotion_image_name, promotion_image_tag, token):
|
||||
if requests.delete(f"https://hub.docker.com/v2/repositories/{promotion_image_namespace}/{promotion_image_name}/tags/{promotion_image_tag}-amd64", headers={"Authorization": f"JWT {token}"}).status_code != 204:
|
||||
raise SystemError(f"Failed to delete redundant images from dockerhub. Please make sure {promotion_image_namespace}/{promotion_image_name}:{promotion_image_tag}-amd64 is removed from dockerhub")
|
||||
if requests.delete(f"https://hub.docker.com/v2/repositories/{promotion_image_namespace}/{promotion_image_name}/tags/{promotion_image_tag}-arm64", headers={"Authorization": f"JWT {token}"}).status_code != 204:
|
||||
raise SystemError(f"Failed to delete redundant images from dockerhub. Please make sure {promotion_image_namespace}/{promotion_image_name}:{promotion_image_tag}-arm64 is removed from dockerhub")
|
||||
execute(["docker", "rmi", f"{promotion_image_namespace}/{promotion_image_name}:{promotion_image_tag}-amd64"])
|
||||
execute(["docker", "rmi", f"{promotion_image_namespace}/{promotion_image_name}:{promotion_image_tag}-arm64"])
|
||||
|
||||
if __name__ == "__main__":
|
||||
login()
|
||||
username = get_input("Enter dockerhub username: ")
|
||||
password = getpass("Enter dockerhub password: ")
|
||||
|
||||
token = (requests.post("https://hub.docker.com/v2/users/login/", json={"username": username, "password": password})).json()['token']
|
||||
if len(token) == 0:
|
||||
raise PermissionError("Dockerhub login failed")
|
||||
|
||||
rc_image = get_input("Enter the RC docker image that you want to pull (in the format <registry>/<namespace>/<image_name>:<image_tag>): ")
|
||||
promotion_image_namespace = get_input("Enter the dockerhub namespace that the rc image needs to be promoted to [example: apache]: ")
|
||||
promotion_image_name = get_input("Enter the dockerhub image name that the rc image needs to be promoted to [example: kafka]: ")
|
||||
promotion_image_tag = get_input("Enter the dockerhub image tag that the rc image needs to be promoted to [example: latest]: ")
|
||||
promotion_image = f"{promotion_image_namespace}/{promotion_image_name}:{promotion_image_tag}"
|
||||
|
||||
print(f"Docker image {rc_image} will be pulled and pushed to {promotion_image}")
|
||||
|
||||
proceed = input("Should we proceed? [y/N]: ")
|
||||
if proceed == "y":
|
||||
pull(rc_image, promotion_image)
|
||||
push(promotion_image)
|
||||
push_manifest(promotion_image)
|
||||
remove(promotion_image_namespace, promotion_image_name, promotion_image_tag, token)
|
||||
print("The image has been promoted successfully. The promoted image should be accessible in dockerhub")
|
||||
else:
|
||||
print("Image promotion aborted")
|
||||
|
|
@ -104,7 +104,7 @@ Single Node
|
|||
# Run from root of the repo
|
||||
$ docker compose -f docker/examples/jvm/single-node/ssl/docker-compose.yml up
|
||||
```
|
||||
- To produce messages using client scripts:-
|
||||
- To produce messages using client scripts (Ensure that java version >= 17):-
|
||||
```
|
||||
# Run from root of the repo
|
||||
$ bin/kafka-console-producer.sh --topic test --bootstrap-server localhost:9093 --producer.config ./docker/examples/fixtures/client-secrets/client-ssl.properties
|
||||
|
|
@ -119,7 +119,7 @@ Single Node
|
|||
# Run from root of the repo
|
||||
$ docker compose -f docker/examples/jvm/single-node/file-input/docker-compose.yml up
|
||||
```
|
||||
- To produce messages using client scripts:-
|
||||
- To produce messages using client scripts (Ensure that java version >= 17):-
|
||||
```
|
||||
# Run from root of the repo
|
||||
$ bin/kafka-console-producer.sh --topic test --bootstrap-server localhost:9093 --producer.config ./docker/examples/fixtures/client-secrets/client-ssl.properties
|
||||
|
|
@ -159,7 +159,7 @@ Multi Node Cluster
|
|||
# Run from root of the repo
|
||||
$ docker compose -f docker/examples/jvm/cluster/combined/ssl/docker-compose.yml up
|
||||
```
|
||||
- To produce messages using client scripts:-
|
||||
- To produce messages using client scripts (Ensure that java version >= 17):-
|
||||
```
|
||||
# Run from root of the repo
|
||||
$ bin/kafka-console-producer.sh --topic test --bootstrap-server localhost:29093 --producer.config ./docker/examples/fixtures/client-secrets/client-ssl.properties
|
||||
|
|
@ -188,7 +188,7 @@ Multi Node Cluster
|
|||
# Run from root of the repo
|
||||
$ docker compose -f docker/examples/jvm/cluster/isolated/ssl/docker-compose.yml up
|
||||
```
|
||||
- To produce messages using client scripts:-
|
||||
- To produce messages using client scripts (Ensure that java version >= 17):-
|
||||
```
|
||||
# Run from root of the repo
|
||||
$ bin/kafka-console-producer.sh --topic test --bootstrap-server localhost:29093 --producer.config ./docker/examples/fixtures/client-secrets/client-ssl.properties
|
||||
|
|
|
|||
|
|
@ -759,9 +759,9 @@ https://kafka.apache.org/KEYS
|
|||
* Release artifacts to be voted upon (source and binary):
|
||||
https://home.apache.org/~%(apache_id)s/kafka-%(rc_tag)s/
|
||||
|
||||
<USE docker/README.md FOR STEPS TO GENERATE DOCKER IMAGE>
|
||||
<USE docker/README.md FOR STEPS TO CREATE RELEASE CANDIDATE DOCKER IMAGE>
|
||||
* Docker release artifact to be voted upon:
|
||||
<USERNAME>/<IMAGE_NAME>:<IMAGE:TAG>
|
||||
apache/kafka:%(rc_tag)s
|
||||
|
||||
* Maven artifacts to be voted upon:
|
||||
https://repository.apache.org/content/groups/staging/org/apache/kafka/
|
||||
|
|
@ -781,6 +781,9 @@ https://kafka.apache.org/%(docs_version)s/protocol.html
|
|||
* Successful Jenkins builds for the %(dev_branch)s branch:
|
||||
Unit/integration tests: https://ci-builds.apache.org/job/Kafka/job/kafka/job/%(dev_branch)s/<BUILD NUMBER>/
|
||||
System tests: https://jenkins.confluent.io/job/system-test-kafka/job/%(dev_branch)s/<BUILD_NUMBER>/
|
||||
|
||||
<USE docker/README.md FOR STEPS TO RUN DOCKER BUILD TEST GITHUB ACTIONS>
|
||||
* Successful Docker Image Github Actions Pipeline for %(dev_branch)s branch:
|
||||
Docker Build Test Pipeline: https://github.com/apache/kafka/actions/runs/<RUN_NUMBER>
|
||||
|
||||
/**************************************
|
||||
|
|
|
|||
Loading…
Reference in New Issue