mirror of https://github.com/apache/kafka.git
KAFKA-16932: Add documentation for the native docker image (#16338)
This PR contains the the following documentation changes for the native docker image: in the docker/README.md: How to build, release and promote the native docker image. in the tests/README.md: How to run system tests by bringing up kafka in the native mode. added docker/native/README.md added html changes for the kafka-site added native docker image support in the docker compose files examples. Testing: Tested all the docker compose files with both the docker images - jvm and native Tested the html changes locally with the kafka-site Reviewers: Manikumar Reddy <manikumar.reddy@gmail.com>, Vedarth Sharma <vesharma@confluent.io>
This commit is contained in:
Krishna Agarwal
GitHub
parent
4e846038a6
commit
bcf781230e
|
|
@ -4,8 +4,10 @@ Docker Images
|
|||
Introduction
|
||||
------------
|
||||
|
||||
This directory contains scripts to build, test, push and promote docker image for kafka. It also contains scripts to build, test the JVM-based Docker Official Image, and generate a PR template for the same to be raised under the Docker official images repo.
|
||||
All of the steps can be either performed locally or by using Github Actions.
|
||||
This directory contains:
|
||||
- Scripts to build, test, push and promote docker images for apache kafka - both `jvm` based and graalvm based `native` kafka
|
||||
- Scripts to build, test the JVM-based Docker Official Image, and generate a PR template for the same to be raised under the Docker official images repo.
|
||||
All of the steps can be either performed locally or by using Github Actions.
|
||||
|
||||
Github Actions
|
||||
--------------
|
||||
|
|
@ -27,28 +29,42 @@ Building image and running tests using github actions
|
|||
|
||||
- kafka-url - This is the url to download kafka tarball from. For example kafka tarball url from (https://archive.apache.org/dist/kafka). For building RC image this will be an RC tarball url.
|
||||
|
||||
- image-type - This is the type of image that we intend to build. This will be dropdown menu type selection in the workflow. `jvm` image type is for official docker image (to be hosted on apache/kafka) as described in [KIP-975](https://cwiki.apache.org/confluence/display/KAFKA/KIP-975%3A+Docker+Image+for+Apache+Kafka)
|
||||
- image-type - This is the type of image that we intend to build. This will be dropdown menu type selection in the workflow.
|
||||
- `jvm` image type is for official docker image (to be hosted on apache/kafka) as described in [KIP-975](https://cwiki.apache.org/confluence/display/KAFKA/KIP-975%3A+Docker+Image+for+Apache+Kafka)
|
||||
- `native` image type is for graalvm based `native` kafka docker image (to be hosted on apache/kafka-native) as described in [KIP-974](https://cwiki.apache.org/confluence/display/KAFKA/KIP-974%3A+Docker+Image+for+GraalVM+based+Native+Kafka+Broker#KIP974:DockerImageforGraalVMbasedNativeKafkaBroker-ImageNaming)
|
||||
|
||||
- Example:-
|
||||
- Example(jvm):-
|
||||
To build and test a jvm image type ensuring kafka to be containerised should be https://archive.apache.org/dist/kafka/3.6.0/kafka_2.13-3.6.0.tgz (it is recommended to use scala 2.13 binary tarball), following inputs in github actions workflow are recommended.
|
||||
```
|
||||
image_type: jvm
|
||||
kafka_url: https://archive.apache.org/dist/kafka/3.6.0/kafka_2.13-3.6.0.tgz
|
||||
```
|
||||
|
||||
- Example(native):-
|
||||
To build and test a native image type ensuring kafka to be containerised should be https://archive.apache.org/dist/kafka/3.8.0/kafka_2.13-3.8.0.tgz (it is recommended to use scala 2.13 binary tarball), following inputs in github actions workflow are recommended.
|
||||
```
|
||||
image_type: native
|
||||
kafka_url: https://archive.apache.org/dist/kafka/3.8.0/kafka_2.13-3.8.0.tgz
|
||||
```
|
||||
Creating a Release Candidate using github actions
|
||||
-------------------------------------------------
|
||||
|
||||
- This is the recommended way to push an RC docker image.
|
||||
- Go to `Build and Push Release Candidate Docker Image` Github Actions Workflow.
|
||||
- Choose the `image_type` and provide `kafka_url` that needs to be containerised in the `rc_docker_image` that will be pushed to github.
|
||||
- Example:-
|
||||
- Example(jvm):-
|
||||
If you want to push a jvm image which contains kafka from https://archive.apache.org/dist/kafka/3.6.0/kafka_2.13-3.6.0.tgz to dockerhub under the namespace apache, repo name as kafka and image tag as 3.6.0-rc1 then following values need to be added in Github Actions Workflow:-
|
||||
```
|
||||
image_type: jvm
|
||||
kafka_url: https://archive.apache.org/dist/kafka/3.6.0/kafka_2.13-3.6.0.tgz
|
||||
rc_docker_image: apache/kafka:3.6.0-rc0
|
||||
```
|
||||
- Example(native):-
|
||||
If you want to push a native image which contains kafka from https://archive.apache.org/dist/kafka/3.8.0/kafka_2.13-3.8.0.tgz to dockerhub under the namespace apache, repo name as kafka-native and image tag as 3.8.0-rc0 then following values need to be added in Github Actions Workflow:-
|
||||
```
|
||||
image_type: native
|
||||
kafka_url: https://archive.apache.org/dist/kafka/3.8.0/kafka_2.13-3.8.0.tgz
|
||||
rc_docker_image: apache/kafka-native:3.8.0-rc0
|
||||
```
|
||||
|
||||
Promoting a Release Candidate using github actions
|
||||
--------------------------------------------------
|
||||
|
|
@ -56,12 +72,18 @@ Promoting a Release Candidate using github actions
|
|||
- This is the recommended way to promote an RC docker image.
|
||||
- Go to `Promote Release Candidate Docker Image` Github Actions Workflow.
|
||||
- Choose the RC docker image (`rc_docker_image`) that you want to promote and where it needs to be pushed to (`promoted_docker_image`), i.e. the final docker image release.
|
||||
- Example:-
|
||||
- Example(jvm):-
|
||||
If you want to promote apache/kafka:3.6.0-rc0 RC docker image to apache/kafka:3.6.0 then following parameters can be provided to the workflow.
|
||||
```
|
||||
rc_docker_image: apache/kafka:3.6.0-rc0
|
||||
promoted_docker_image: apache/kafka:3.6.0
|
||||
```
|
||||
- Example(native):-
|
||||
If you want to promote apache/kafka-native:3.8.0-rc0 RC docker image to apache/kafka-native:3.8.0 then following parameters can be provided to the workflow.
|
||||
```
|
||||
rc_docker_image: apache/kafka-native:3.8.0-rc0
|
||||
promoted_docker_image: apache/kafka-native:3.8.0
|
||||
```
|
||||
|
||||
Cron job for checking CVEs in supported docker images
|
||||
-----------------------------------------------------
|
||||
|
|
@ -100,10 +122,14 @@ Building image and running tests locally
|
|||
- Sanity tests for the docker image are present in test/docker_sanity_test.py.
|
||||
- By default image will be built and tested, but if you only want to build the image, pass `--build` (or `-b`) flag and if you only want to test the given image pass `--test` (or `-t`) flag.
|
||||
- An html test report will be generated after the tests are executed containing the results.
|
||||
- Example :- To build and test an image named test under kafka namespace with 3.6.0 tag and jvm image type ensuring kafka to be containerised should be https://downloads.apache.org/kafka/3.6.0/kafka_2.13-3.6.0.tgz (it is recommended to use scala 2.13 binary tarball), following command can be used
|
||||
- Example(jvm) :- To build and test an image named test under kafka namespace with 3.6.0 tag and jvm image type ensuring kafka to be containerised should be https://downloads.apache.org/kafka/3.6.0/kafka_2.13-3.6.0.tgz (it is recommended to use scala 2.13 binary tarball), following command can be used
|
||||
```
|
||||
python docker_build_test.py kafka/test --image-tag=3.6.0 --image-type=jvm --kafka-url=https://archive.apache.org/dist/kafka/3.6.0/kafka_2.13-3.6.0.tgz
|
||||
```
|
||||
- Example(native) :- To build and test an image named test under kafka namespace with 3.8.0 tag and native image type ensuring kafka to be containerised should be https://downloads.apache.org/kafka/3.8.0/kafka_2.13-3.8.0.tgz (it is recommended to use scala 2.13 binary tarball), following command can be used
|
||||
```
|
||||
python docker_build_test.py kafka/test --image-tag=3.8.0 --image-type=native --kafka-url=https://archive.apache.org/dist/kafka/3.8.0/kafka_2.13-3.8.0.tgz
|
||||
```
|
||||
|
||||
Creating a Release Candidate
|
||||
----------------------------
|
||||
|
|
@ -111,11 +137,17 @@ Creating a Release Candidate
|
|||
- `docker_release.py` script builds a multi-architecture image and pushes it to provided docker registry.
|
||||
- Ensure you are logged in to the docker registry before triggering the script.
|
||||
- kafka binary tarball url along with image name (in the format `<registry>/<namespace>/<image_name>:<image_tag>`) and type is needed to build the image. For detailed usage description check `python docker_release.py --help`.
|
||||
- Example:- To push an image named test under kafka dockerhub namespace with 3.6.0 tag and jvm image type ensuring kafka to be containerised should be https://archive.apache.org/dist/kafka/3.6.0/kafka_2.13-3.6.0.tgz (it is recommended to use scala 2.13 binary tarball), following command can be used. (Make sure you have push access to the docker repo)
|
||||
- Example(jvm):- To push an image named test under kafka dockerhub namespace with 3.6.0 tag and jvm image type ensuring kafka to be containerised should be https://archive.apache.org/dist/kafka/3.6.0/kafka_2.13-3.6.0.tgz (it is recommended to use scala 2.13 binary tarball), following command can be used. (Make sure you have push access to the docker repo)
|
||||
```
|
||||
# kafka/test is an example repo. Please replace with the docker hub repo you have push access to.
|
||||
|
||||
python docker_release.py kafka/test:3.6.0 --kafka-url https://archive.apache.org/dist/kafka/3.6.0/kafka_2.13-3.6.0.tgz
|
||||
python docker_release.py kafka/test:3.6.0 --kafka-url --image-type=jvm https://archive.apache.org/dist/kafka/3.6.0/kafka_2.13-3.6.0.tgz
|
||||
```
|
||||
- Example(native):- To push an image named test under kafka-native dockerhub namespace with 3.8.0 tag and native image type ensuring kafka to be containerised should be https://archive.apache.org/dist/kafka/3.8.0/kafka_2.13-3.8.0.tgz (it is recommended to use scala 2.13 binary tarball), following command can be used. (Make sure you have push access to the docker repo)
|
||||
```
|
||||
# kafka-native/test is an example repo. Please replace with the docker hub repo you have push access to.
|
||||
|
||||
python docker_release.py kafka-native/test:3.8.0 --kafka-url --image-type=native https://archive.apache.org/dist/kafka/3.8.0/kafka_2.13-3.8.0.tgz
|
||||
```
|
||||
|
||||
- Please note that we use docker buildx for preparing the multi-architecture image and pushing it to docker registry. It's possible to encounter build failures because of buildx. Please retry the command in case some buildx related error occurs.
|
||||
|
|
@ -124,12 +156,18 @@ Promoting a Release Candidate
|
|||
-----------------------------
|
||||
|
||||
- It's not recommended to promote the docker image locally, as we have github actions doing it in a convenient way, but if needed following command can be used to promote a Release Candidate image.
|
||||
- Example:- If you want to promote RC image apache/kafka:3.6.0-rc0 to apache/kafka:3.6.0, following command can be used
|
||||
- Example(jvm):- If you want to promote RC image apache/kafka:3.6.0-rc0 to apache/kafka:3.6.0, following command can be used
|
||||
|
||||
```
|
||||
# Ensure docker buildx is enabled in your system and you have access to apache/kafka
|
||||
docker buildx imagetools create --tag apache/kafka:3.6.0 apache/kafka:3.6.0-rc0
|
||||
```
|
||||
- Example(native):- If you want to promote RC image apache/kafka-native:3.8.0-rc0 to apache/kafka-native:3.8.0, following command can be used
|
||||
|
||||
```
|
||||
# Ensure docker buildx is enabled in your system and you have access to apache/kafka-native
|
||||
docker buildx imagetools create --tag apache/kafka-native:3.8.0 apache/kafka-native:3.8.0-rc0
|
||||
```
|
||||
|
||||
Using the image in a docker container
|
||||
-------------------------------------
|
||||
|
|
@ -172,5 +210,6 @@ python generate_kafka_pr_template.py --image-type=jvm
|
|||
```
|
||||
|
||||
- kafka-version - This is the version to create the Docker official images static Dockerfile and assets for, as well as the version to build and test the Docker official image for.
|
||||
- image-type - This is the type of image that we intend to build. This will be dropdown menu type selection in the workflow. `jvm` image type is for official docker image (to be hosted on apache/kafka) as described in [KIP-975](https://cwiki.apache.org/confluence/display/KAFKA/KIP-975%3A+Docker+Image+for+Apache+Kafka). As of now [KIP-1028](https://cwiki.apache.org/confluence/display/KAFKA/KIP-1028%3A+Docker+Official+Image+for+Apache+Kafka) only aims to release JVM based Docker Official Images.
|
||||
- image-type - This is the type of image that we intend to build. This will be dropdown menu type selection in the workflow. `jvm` image type is for official docker image (to be hosted on apache/kafka) as described in [KIP-975](https://cwiki.apache.org/confluence/display/KAFKA/KIP-975%3A+Docker+Image+for+Apache+Kafka).
|
||||
- **NOTE:** As of now [KIP-1028](https://cwiki.apache.org/confluence/display/KAFKA/KIP-1028%3A+Docker+Official+Image+for+Apache+Kafka) only aims to release JVM based Docker Official Images and not GraalVM based native Apache Kafka docker image.
|
||||
|
||||
|
|
|
|||
|
|
@ -58,7 +58,7 @@ Running in SSL mode
|
|||
- Recommended way to run in ssl mode is by mounting secrets on `/etc/kafka/secrets` in docker container and providing configs following through environment variables (`KAFKA_SSL_KEYSTORE_FILENAME`, `KAFKA_SSL_KEYSTORE_CREDENTIALS`, `KAFKA_SSL_KEY_CREDENTIALS`, `KAFKA_SSL_TRUSTSTORE_FILENAME` and `KAFKA_SSL_TRUSTSTORE_CREDENTIALS`) to let the docker image scripts extract passwords and populate correct paths in server.properties.
|
||||
- Please ensure appropriate `KAFKA_ADVERTISED_LISTENERS` are provided through environment variables to enable SSL mode in Kafka server, i.e. it should contain an `SSL` listener.
|
||||
- Alternatively property file input can be used to provide ssl properties.
|
||||
- Make sure you set location of truststore and keystore correctly when using file input. See example for file input in `jvm/single-node/file-input` for better clarity.
|
||||
- Make sure you set location of truststore and keystore correctly when using file input. See example for file input in `docker-compose-files/single-node/file-input` for better clarity.
|
||||
- Note that advertised.listeners property needs to be provided along with SSL properties in file input and cannot be provided through environment variable separately.
|
||||
- In conclusion, ssl properties with advertised.listeners should be treated as a group and provided in file input or environment variables in it's entirety.
|
||||
- In case ssl properties are provided both through file input and environment variables, environment variable properties will override the file input properties, just as mentioned in the beginning of this section.
|
||||
|
|
@ -66,7 +66,15 @@ Running in SSL mode
|
|||
Examples
|
||||
--------
|
||||
|
||||
- `jvm` directory contains docker compose files for some example configs to run `apache/kafka` docker image.
|
||||
- `docker-compose-files` directory contains docker compose files for some example configs to run `apache/kafka` OR `apache/kafka-native` docker image.
|
||||
- Pass the `IMAGE` variable with the Docker Compose file to specify which Docker image to use for bringing up the containers.
|
||||
```
|
||||
# to bring up containers using apache/kafka docker image
|
||||
IMAGE=apache/kafka:latest <docker compose command>
|
||||
|
||||
# to bring up containers using apache/kafka-native docker image
|
||||
IMAGE=apache/kafka-native:latest <docker compose command>
|
||||
```
|
||||
- Run the commands from root of the repository.
|
||||
- Checkout `single-node` examples for quick small examples to play around with.
|
||||
- `cluster` contains multi node examples, for `combined` mode as well as `isolated` mode.
|
||||
|
|
@ -77,7 +85,7 @@ Single Node
|
|||
-----------
|
||||
|
||||
- These examples are for understanding various ways inputs can be provided and kafka can be configured in docker container.
|
||||
- Examples are present inside `jvm/single-node` directory.
|
||||
- Examples are present inside `docker-compose-files/single-node` directory.
|
||||
- Plaintext:-
|
||||
- This is the simplest compose file.
|
||||
- We are using environment variables purely for providing configs.
|
||||
|
|
@ -88,7 +96,12 @@ Single Node
|
|||
- To run the example:-
|
||||
```
|
||||
# Run from root of the repo
|
||||
$ docker compose -f docker/examples/jvm/single-node/plaintext/docker-compose.yml up
|
||||
|
||||
# JVM based Apache Kafka Docker Image
|
||||
$ IMAGE=apache/kafka:latest docker compose -f docker/examples/docker-compose-files/single-node/plaintext/docker-compose.yml up
|
||||
|
||||
# GraalVM based Native Apache Kafka Docker Image
|
||||
$ IMAGE=apache/kafka-native:latest docker compose -f docker/examples/docker-compose-files/single-node/plaintext/docker-compose.yml up
|
||||
```
|
||||
- To produce messages using client scripts:-
|
||||
```
|
||||
|
|
@ -102,7 +115,12 @@ Single Node
|
|||
- To run the example:-
|
||||
```
|
||||
# Run from root of the repo
|
||||
$ docker compose -f docker/examples/jvm/single-node/ssl/docker-compose.yml up
|
||||
|
||||
# JVM based Apache Kafka Docker Image
|
||||
$ IMAGE=apache/kafka:latest docker compose -f docker/examples/docker-compose-files/single-node/ssl/docker-compose.yml up
|
||||
|
||||
# GraalVM based Native Apache Kafka Docker Image
|
||||
$ IMAGE=apache/kafka-native:latest docker compose -f docker/examples/docker-compose-files/single-node/ssl/docker-compose.yml up
|
||||
```
|
||||
- To produce messages using client scripts (Ensure that java version >= 17):-
|
||||
```
|
||||
|
|
@ -117,7 +135,12 @@ Single Node
|
|||
- To run the example:-
|
||||
```
|
||||
# Run from root of the repo
|
||||
$ docker compose -f docker/examples/jvm/single-node/file-input/docker-compose.yml up
|
||||
|
||||
# JVM based Apache Kafka Docker Image
|
||||
$ IMAGE=apache/kafka:latest docker compose -f docker/examples/docker-compose-files/single-node/file-input/docker-compose.yml up
|
||||
|
||||
# GraalVM based Native Apache Kafka Docker Image
|
||||
$ IMAGE=apache/kafka-native:latest docker compose -f docker/examples/docker-compose-files/single-node/file-input/docker-compose.yml up
|
||||
```
|
||||
- To produce messages using client scripts (Ensure that java version >= 17):-
|
||||
```
|
||||
|
|
@ -130,7 +153,7 @@ Multi Node Cluster
|
|||
|
||||
- These examples are for real world usecases where multiple nodes of kafka are required.
|
||||
- Combined:-
|
||||
- Examples are present in `jvm/cluster/combined` directory.
|
||||
- Examples are present in `docker-compose-files/cluster/combined` directory.
|
||||
- Plaintext:-
|
||||
- Each broker must expose a unique port to host machine.
|
||||
- For example broker-1, broker2 and broker3 are listening on port 9092, they're exposing it to the host via ports 29092, 39092 and 49092 respectively.
|
||||
|
|
@ -143,7 +166,12 @@ Multi Node Cluster
|
|||
- To run the example:-
|
||||
```
|
||||
# Run from root of the repo
|
||||
$ docker compose -f docker/examples/jvm/cluster/combined/plaintext/docker-compose.yml up
|
||||
|
||||
# JVM based Apache Kafka Docker Image
|
||||
$ IMAGE=apache/kafka:latest docker compose -f docker/examples/docker-compose-files/cluster/combined/plaintext/docker-compose.yml up
|
||||
|
||||
# GraalVM based Native Apache Kafka Docker Image
|
||||
$ IMAGE=apache/kafka-native:latest docker compose -f docker/examples/docker-compose-files/cluster/combined/plaintext/docker-compose.yml up
|
||||
```
|
||||
- To access using client script:-
|
||||
```
|
||||
|
|
@ -157,7 +185,12 @@ Multi Node Cluster
|
|||
- To run the example:-
|
||||
```
|
||||
# Run from root of the repo
|
||||
$ docker compose -f docker/examples/jvm/cluster/combined/ssl/docker-compose.yml up
|
||||
|
||||
# JVM based Apache Kafka Docker Image
|
||||
$ IMAGE=apache/kafka:latest docker compose -f docker/examples/docker-compose-files/cluster/combined/ssl/docker-compose.yml up
|
||||
|
||||
# GraalVM based Native Apache Kafka Docker Image
|
||||
$ IMAGE=apache/kafka-native:latest docker compose -f docker/examples/docker-compose-files/cluster/combined/ssl/docker-compose.yml up
|
||||
```
|
||||
- To produce messages using client scripts (Ensure that java version >= 17):-
|
||||
```
|
||||
|
|
@ -165,7 +198,7 @@ Multi Node Cluster
|
|||
$ bin/kafka-console-producer.sh --topic test --bootstrap-server localhost:29093 --producer.config ./docker/examples/fixtures/client-secrets/client-ssl.properties
|
||||
```
|
||||
- Isolated:-
|
||||
- Examples are present in `jvm/cluster/isolated` directory.
|
||||
- Examples are present in `docker-compose-files/cluster/isolated` directory.
|
||||
- Plaintext:-
|
||||
- Here controllers and brokers are configured separately.
|
||||
- It's a good practice to define that brokers depend on controllers.
|
||||
|
|
@ -173,7 +206,12 @@ Multi Node Cluster
|
|||
- To run the example:-
|
||||
```
|
||||
# Run from root of the repo
|
||||
$ docker compose -f docker/examples/jvm/cluster/isolated/plaintext/docker-compose.yml up
|
||||
|
||||
# JVM based Apache Kafka Docker Image
|
||||
$ IMAGE=apache/kafka:latest docker compose -f docker/examples/docker-compose-files/cluster/isolated/plaintext/docker-compose.yml up
|
||||
|
||||
# GraalVM based Native Apache Kafka Docker Image
|
||||
$ IMAGE=apache/kafka-native:latest docker compose -f docker/examples/docker-compose-files/cluster/isolated/plaintext/docker-compose.yml up
|
||||
```
|
||||
- To access using client script:-
|
||||
```
|
||||
|
|
@ -186,7 +224,12 @@ Multi Node Cluster
|
|||
- To run the example:-
|
||||
```
|
||||
# Run from root of the repo
|
||||
$ docker compose -f docker/examples/jvm/cluster/isolated/ssl/docker-compose.yml up
|
||||
|
||||
# JVM based Apache Kafka Docker Image
|
||||
$ IMAGE=apache/kafka:latest docker compose -f docker/examples/docker-compose-files/cluster/isolated/ssl/docker-compose.yml up
|
||||
|
||||
# GraalVM based Native Apache Kafka Docker Image
|
||||
$ IMAGE=apache/kafka-native:latest docker compose -f docker/examples/docker-compose-files/cluster/isolated/ssl/docker-compose.yml up
|
||||
```
|
||||
- To produce messages using client scripts (Ensure that java version >= 17):-
|
||||
```
|
||||
|
|
|
|||
|
|
@ -17,7 +17,7 @@
|
|||
version: '2'
|
||||
services:
|
||||
kafka-1:
|
||||
image: apache/kafka:latest
|
||||
image: ${IMAGE}
|
||||
hostname: kafka-1
|
||||
container_name: kafka-1
|
||||
ports:
|
||||
|
|
@ -39,7 +39,7 @@ services:
|
|||
KAFKA_LOG_DIRS: '/tmp/kraft-combined-logs'
|
||||
|
||||
kafka-2:
|
||||
image: apache/kafka:latest
|
||||
image: ${IMAGE}
|
||||
hostname: kafka-2
|
||||
container_name: kafka-2
|
||||
ports:
|
||||
|
|
@ -61,7 +61,7 @@ services:
|
|||
KAFKA_LOG_DIRS: '/tmp/kraft-combined-logs'
|
||||
|
||||
kafka-3:
|
||||
image: apache/kafka:latest
|
||||
image: ${IMAGE}
|
||||
hostname: kafka-3
|
||||
container_name: kafka-3
|
||||
ports:
|
||||
|
|
@ -17,7 +17,7 @@
|
|||
version: '2'
|
||||
services:
|
||||
kafka-1:
|
||||
image: apache/kafka:latest
|
||||
image: ${IMAGE}
|
||||
hostname: kafka-1
|
||||
container_name: kafka-1
|
||||
ports:
|
||||
|
|
@ -49,7 +49,7 @@ services:
|
|||
KAFKA_SSL_ENDPOINT_IDENTIFICATION_ALGORITHM: ""
|
||||
|
||||
kafka-2:
|
||||
image: apache/kafka:latest
|
||||
image: ${IMAGE}
|
||||
hostname: kafka-2
|
||||
container_name: kafka-2
|
||||
ports:
|
||||
|
|
@ -81,7 +81,7 @@ services:
|
|||
KAFKA_SSL_ENDPOINT_IDENTIFICATION_ALGORITHM: ""
|
||||
|
||||
kafka-3:
|
||||
image: apache/kafka:latest
|
||||
image: ${IMAGE}
|
||||
hostname: kafka-3
|
||||
container_name: kafka-3
|
||||
ports:
|
||||
|
|
@ -17,7 +17,7 @@
|
|||
version: '2'
|
||||
services:
|
||||
controller-1:
|
||||
image: apache/kafka:latest
|
||||
image: ${IMAGE}
|
||||
environment:
|
||||
KAFKA_NODE_ID: 1
|
||||
KAFKA_PROCESS_ROLES: 'controller'
|
||||
|
|
@ -33,7 +33,7 @@ services:
|
|||
KAFKA_LOG_DIRS: '/tmp/kraft-combined-logs'
|
||||
|
||||
controller-2:
|
||||
image: apache/kafka:latest
|
||||
image: ${IMAGE}
|
||||
environment:
|
||||
KAFKA_NODE_ID: 2
|
||||
KAFKA_PROCESS_ROLES: 'controller'
|
||||
|
|
@ -49,7 +49,7 @@ services:
|
|||
KAFKA_LOG_DIRS: '/tmp/kraft-combined-logs'
|
||||
|
||||
controller-3:
|
||||
image: apache/kafka:latest
|
||||
image: ${IMAGE}
|
||||
environment:
|
||||
KAFKA_NODE_ID: 3
|
||||
KAFKA_PROCESS_ROLES: 'controller'
|
||||
|
|
@ -65,7 +65,7 @@ services:
|
|||
KAFKA_LOG_DIRS: '/tmp/kraft-combined-logs'
|
||||
|
||||
kafka-1:
|
||||
image: apache/kafka:latest
|
||||
image: ${IMAGE}
|
||||
ports:
|
||||
- 29092:9092
|
||||
hostname: kafka-1
|
||||
|
|
@ -91,7 +91,7 @@ services:
|
|||
- controller-3
|
||||
|
||||
kafka-2:
|
||||
image: apache/kafka:latest
|
||||
image: ${IMAGE}
|
||||
ports:
|
||||
- 39092:9092
|
||||
hostname: kafka-2
|
||||
|
|
@ -117,7 +117,7 @@ services:
|
|||
- controller-3
|
||||
|
||||
kafka-3:
|
||||
image: apache/kafka:latest
|
||||
image: ${IMAGE}
|
||||
ports:
|
||||
- 49092:9092
|
||||
hostname: kafka-3
|
||||
|
|
@ -17,7 +17,7 @@
|
|||
version: '2'
|
||||
services:
|
||||
controller-1:
|
||||
image: apache/kafka:latest
|
||||
image: ${IMAGE}
|
||||
environment:
|
||||
KAFKA_NODE_ID: 1
|
||||
KAFKA_PROCESS_ROLES: 'controller'
|
||||
|
|
@ -33,7 +33,7 @@ services:
|
|||
KAFKA_LOG_DIRS: '/tmp/kraft-combined-logs'
|
||||
|
||||
controller-2:
|
||||
image: apache/kafka:latest
|
||||
image: ${IMAGE}
|
||||
environment:
|
||||
KAFKA_NODE_ID: 2
|
||||
KAFKA_PROCESS_ROLES: 'controller'
|
||||
|
|
@ -49,7 +49,7 @@ services:
|
|||
KAFKA_LOG_DIRS: '/tmp/kraft-combined-logs'
|
||||
|
||||
controller-3:
|
||||
image: apache/kafka:latest
|
||||
image: ${IMAGE}
|
||||
environment:
|
||||
KAFKA_NODE_ID: 3
|
||||
KAFKA_PROCESS_ROLES: 'controller'
|
||||
|
|
@ -65,7 +65,7 @@ services:
|
|||
KAFKA_LOG_DIRS: '/tmp/kraft-combined-logs'
|
||||
|
||||
kafka-1:
|
||||
image: apache/kafka:latest
|
||||
image: ${IMAGE}
|
||||
hostname: kafka-1
|
||||
container_name: kafka-1
|
||||
ports:
|
||||
|
|
@ -101,7 +101,7 @@ services:
|
|||
- controller-3
|
||||
|
||||
kafka-2:
|
||||
image: apache/kafka:latest
|
||||
image: ${IMAGE}
|
||||
hostname: kafka-2
|
||||
container_name: kafka-2
|
||||
ports:
|
||||
|
|
@ -137,7 +137,7 @@ services:
|
|||
- controller-3
|
||||
|
||||
kafka-3:
|
||||
image: apache/kafka:latest
|
||||
image: ${IMAGE}
|
||||
hostname: kafka-3
|
||||
container_name: kafka-3
|
||||
ports:
|
||||
|
|
@ -17,7 +17,7 @@
|
|||
version: '2'
|
||||
services:
|
||||
broker:
|
||||
image: apache/kafka:latest
|
||||
image: ${IMAGE}
|
||||
hostname: broker
|
||||
container_name: broker
|
||||
ports:
|
||||
|
|
@ -17,7 +17,7 @@
|
|||
version: '2'
|
||||
services:
|
||||
broker:
|
||||
image: apache/kafka:latest
|
||||
image: ${IMAGE}
|
||||
hostname: broker
|
||||
container_name: broker
|
||||
ports:
|
||||
|
|
@ -17,7 +17,7 @@
|
|||
version: '2'
|
||||
services:
|
||||
broker:
|
||||
image: apache/kafka:latest
|
||||
image: ${IMAGE}
|
||||
hostname: broker
|
||||
container_name: broker
|
||||
ports:
|
||||
|
|
@ -0,0 +1,31 @@
|
|||
# GraalVM Based Native Apache Kafka Docker Image
|
||||
|
||||
## Introduction
|
||||
- The Native Apache Kafka Docker Image can launch brokers with sub-second startup time and minimal memory footprint by leveraging native Kafka executable.
|
||||
- The native Kafka executable is built by compiling Apache Kafka code ahead-of-time using the [GraalVM native-image tool](https://www.graalvm.org/jdk21/reference-manual/native-image/).
|
||||
- This image is experimental and intended for local development and testing purposes only; it is not recommended for production use.
|
||||
- This is introduced with [KIP-974](https://cwiki.apache.org/confluence/display/KAFKA/KIP-974%3A+Docker+Image+for+GraalVM+based+Native+Kafka+Broker).
|
||||
|
||||
## Native-Image reachability metadata
|
||||
The native-image tool performs static analysis while building a native binary to determine the dynamic features(the dynamic language features of the JVM, including reflection and resource handling, compute the dynamically-accessed program elements such as invoked methods or resource URLs at runtime), but it cannot always exhaustively predict all uses.
|
||||
To ensure inclusion of these elements into the native binary, reachability metadata should be provided to the native-image builder.
|
||||
These metadata configs are provided in the `native-image-configs` directory.
|
||||
|
||||
## How to generate reachability metadata
|
||||
GraalVM provide an option to create these configs automatically by running the application normally with the [native-image agent](https://www.graalvm.org/latest/reference-manual/native-image/metadata/AutomaticMetadataCollection/) attached.
|
||||
The configs in the `native-image-configs` directory are generated by running existing Apache Kafka System Tests(using GraalVM JIT and attaching the native-image agent to the process) as they are quite exhaustive.
|
||||
|
||||
## Limitations of Native Apache Kafka executable
|
||||
- Dynamic Features: For any new dynamic feature added or modified in Apache Kafka, the corresponding metadata configurations must be added or updated in the `native-image-configs`. As of now these configs are static.
|
||||
- Runtime Jars: Native Kafka Executable does not support capabilities that require a new runtime jar provided by the user, as the jar's information is not available during build time. In such cases, a new native Kafka binary must be built, including that jar in the classpath during the build.
|
||||
- Garbage Collection: This implementation uses the community edition of GraalVM, which supports only `serial` GC. Therefore, Native Apache Kafka supports `serial` GC and not `G1` GC.
|
||||
|
||||
## How to use this image in docker container
|
||||
Check out [this](../examples/README.md) guide.
|
||||
|
||||
## How to build, test and release this docker image
|
||||
Check out [this](../README.md) guide.
|
||||
|
||||
## How to run system tests on native Apache Kafka
|
||||
Check out [this](../../tests/README.md#running-tests-using-docker) guide.
|
||||
|
||||
|
|
@ -20,33 +20,36 @@
|
|||
</script>
|
||||
|
||||
<script id="docker-template" type="text/x-handlebars-template">
|
||||
<h4 class="anchor-heading">Introduction</h4>
|
||||
<h4 class="anchor-heading">JVM Based Apache Kafka Docker Image</h4>
|
||||
|
||||
<p>
|
||||
<a href="https://www.docker.com/">Docker</a> is a popular container runtime. Docker images for Apache Kafka can be found on <a href="https://hub.docker.com/r/apache/kafka">Docker Hub</a> and are available from version 3.7.0.
|
||||
<a href="https://www.docker.com/">Docker</a> is a popular container runtime. Docker images for the JVM based Apache Kafka can be found on <a href="https://hub.docker.com/r/apache/kafka">Docker Hub</a> and are available from version 3.7.0.
|
||||
</p>
|
||||
|
||||
<h4 class="anchor-heading">Getting the kafka docker image</h4>
|
||||
<ul>
|
||||
<li> Docker image can be pulled from Docker Hub using the following command: </li>
|
||||
<pre class="line-numbers"><code class="language-bash">$ docker pull apache/kafka:{{fullDotVersion}}</code></pre>
|
||||
<li> If you want to fetch the latest version of the docker image use following command: </li>
|
||||
<pre class="line-numbers"><code class="language-bash">$ docker pull apache/kafka:latest</code></pre>
|
||||
<li> To start the Kafka container using this docker image with default configs and on default port 9092: </li>
|
||||
<pre class="line-numbers"><code class="language-bash">$ docker run -p 9092:9092 apache/kafka:{{fullDotVersion}}</code></pre>
|
||||
</ul>
|
||||
|
||||
<h4 class="anchor-heading">GraalVM Based Native Apache Kafka Docker Image</h4>
|
||||
|
||||
<p>
|
||||
Docker image can be pulled from Docker Hub using the following command:-
|
||||
Docker images for the GraalVM Based Native Apache Kafka can be found on <a href="https://hub.docker.com/r/apache/kafka-native">Docker Hub</a> and are available from version 3.8.0.<br>
|
||||
NOTE: This image is experimental and intended for local development and testing purposes only; it is not recommended for production use.
|
||||
</p>
|
||||
|
||||
<pre class="line-numbers"><code class="language-bash">$ docker pull apache/kafka:{{fullDotVersion}}</code></pre>
|
||||
|
||||
<p>
|
||||
If you want to fetch the latest version of the docker image use following command:-
|
||||
</p>
|
||||
|
||||
<pre class="line-numbers"><code class="language-bash">$ docker pull apache/kafka:latest</code></pre>
|
||||
|
||||
<h4 class="anchor-heading">Start kafka with default configs</h4>
|
||||
|
||||
<p>
|
||||
Run docker image on default port 9092:-
|
||||
</p>
|
||||
|
||||
<pre class="line-numbers"><code class="language-bash">$ docker run -p 9092:9092 apache/kafka:{{fullDotVersion}}</code></pre>
|
||||
<ul>
|
||||
<li> Docker image can be pulled from Docker Hub using the following command: </li>
|
||||
<pre class="line-numbers"><code class="language-bash">$ docker pull apache/kafka-native:{{fullDotVersion}}</code></pre>
|
||||
<li> If you want to fetch the latest version of the docker image use following command: </li>
|
||||
<pre class="line-numbers"><code class="language-bash">$ docker pull apache/kafka-native:latest</code></pre>
|
||||
<li> To start the Kafka container using this docker image with default configs and on default port 9092: </li>
|
||||
<pre class="line-numbers"><code class="language-bash">$ docker run -p 9092:9092 apache/kafka-native:{{fullDotVersion}}</code></pre>
|
||||
</ul>
|
||||
|
||||
<h4 class="anchor-heading">Usage guide</h4>
|
||||
|
||||
|
|
|
|||
|
|
@ -98,19 +98,23 @@ $ bin/kafka-server-start.sh config/server.properties</code></pre>
|
|||
|
||||
<pre class="line-numbers"><code class="language-bash">$ bin/kafka-server-start.sh config/kraft/server.properties</code></pre>
|
||||
|
||||
<h5>Using docker image</h5>
|
||||
<h5>Using JVM Based Apache Kafka Docker Image</h5>
|
||||
|
||||
<p>
|
||||
Get the docker image
|
||||
</p>
|
||||
<ul>
|
||||
<li> Get the docker image: </li>
|
||||
<pre class="line-numbers"><code class="language-bash">$ docker pull apache/kafka:{{fullDotVersion}}</code></pre>
|
||||
<li> Start the kafka docker container: </li>
|
||||
<pre class="line-numbers"><code class="language-bash">$ docker run -p 9092:9092 apache/kafka:{{fullDotVersion}}</code></pre>
|
||||
</ul>
|
||||
|
||||
<pre class="line-numbers"><code class="language-bash">$ docker pull apache/kafka:{{fullDotVersion}}</code></pre>
|
||||
<h5>Using GraalVM Based Native Apache Kafka Docker Image</h5>
|
||||
|
||||
<p>
|
||||
Start the kafka docker container
|
||||
</p>
|
||||
|
||||
<pre class="line-numbers"><code class="language-bash">$ docker run -p 9092:9092 apache/kafka:{{fullDotVersion}}</code></pre>
|
||||
<ul>
|
||||
<li> Get the docker image: </li>
|
||||
<pre class="line-numbers"><code class="language-bash">$ docker pull apache/kafka-native:{{fullDotVersion}}</code></pre>
|
||||
<li> Start the kafka docker container: </li>
|
||||
<pre class="line-numbers"><code class="language-bash">$ docker run -p 9092:9092 apache/kafka-native:{{fullDotVersion}}</code></pre>
|
||||
</ul>
|
||||
|
||||
<p>
|
||||
Once the Kafka server has successfully launched, you will have a basic Kafka environment running and ready to use.
|
||||
|
|
|
|||
|
|
@ -790,8 +790,9 @@ https://kafka.apache.org/KEYS
|
|||
https://home.apache.org/~%(apache_id)s/kafka-%(rc_tag)s/
|
||||
|
||||
<USE docker/README.md FOR STEPS TO CREATE RELEASE CANDIDATE DOCKER IMAGE>
|
||||
* Docker release artifact to be voted upon:
|
||||
* Docker release artifact to be voted upon(apache/kafka-native is supported from 3.8+ release.):
|
||||
apache/kafka:%(rc_tag)s
|
||||
apache/kafka-native:%(rc_tag)s
|
||||
|
||||
* Maven artifacts to be voted upon:
|
||||
https://repository.apache.org/content/groups/staging/org/apache/kafka/
|
||||
|
|
@ -813,7 +814,11 @@ Unit/integration tests: https://ci-builds.apache.org/job/Kafka/job/kafka/job/%(d
|
|||
System tests: https://jenkins.confluent.io/job/system-test-kafka/job/%(dev_branch)s/<BUILD_NUMBER>/
|
||||
|
||||
<USE docker/README.md FOR STEPS TO RUN DOCKER BUILD TEST GITHUB ACTIONS>
|
||||
* Successful Docker Image Github Actions Pipeline for %(dev_branch)s branch:
|
||||
* Successful JVM based Apache Kafka Docker Image Github Actions Pipeline for %(dev_branch)s branch:
|
||||
Docker Build Test Pipeline: https://github.com/apache/kafka/actions/runs/<RUN_NUMBER>
|
||||
|
||||
* Successful GraalVM based Native Apache Kafka Docker Image Github Actions Pipeline for %(dev_branch)s branch:
|
||||
* NOTE: GraalVM based Native Apache Kafka Docker Image is supported from 3.8+ release.
|
||||
Docker Build Test Pipeline: https://github.com/apache/kafka/actions/runs/<RUN_NUMBER>
|
||||
|
||||
/**************************************
|
||||
|
|
|
|||
|
|
@ -51,6 +51,19 @@ bash tests/docker/ducker-ak up -j 'openjdk:11'; tests/docker/run_tests.sh
|
|||
```
|
||||
REBUILD="t" bash tests/docker/run_tests.sh
|
||||
```
|
||||
* Run tests with Kafka in `native` mode
|
||||
- To run tests with Kafka in `native` mode, pass `{"kafka_mode": "native"}` to the ducktape globals. This will bring up ducker nodes with the native Kafka binary inside them and use it to start Kafka while running the tests.
|
||||
```
|
||||
_DUCKTAPE_OPTIONS="--globals '{\"kafka_mode\":\"native\"}'" TC_PATHS="tests/kafkatest/tests/" bash tests/docker/run_tests.sh
|
||||
```
|
||||
- To only bring up ducker nodes with kafka native binary inside it.
|
||||
```
|
||||
bash tests/docker/ducker-ak up -m native
|
||||
```
|
||||
- To run tests with Kafka in `native` mode using `ducker-ak test`(Make sure that the ducker nodes are already up with kafka native binary inside it).
|
||||
```
|
||||
tests/docker/ducker-ak test tests/kafkatest/tests/client/compression_test.py -- --globals '{\"kafka_mode\":\"native\"}'
|
||||
```
|
||||
* Debug tests in VS Code:
|
||||
- Run test with `--debug` flag (can be before or after file name):
|
||||
```
|
||||
|
|
|
|||
Loading…
Reference in New Issue