mirror of https://github.com/apache/kafka.git
KAFKA-15879: Add documentation and examples for docker image (#14938)
Reviewers: Manikumar Reddy <manikumar.reddy@gmail.com>
This commit is contained in:
Vedarth Sharma
GitHub
parent
75ad884161
commit
0cd2be8718
|
|
@ -209,7 +209,8 @@ if (repo != null) {
|
|||
'**/generated/**',
|
||||
'clients/src/test/resources/serializedData/*',
|
||||
'docker/resources/utility/go.sum',
|
||||
'docker/test/fixtures/secrets/*'
|
||||
'docker/test/fixtures/secrets/*',
|
||||
'docker/examples/fixtures/secrets/*'
|
||||
])
|
||||
}
|
||||
} else {
|
||||
|
|
|
|||
|
|
@ -94,32 +94,7 @@ promoted_docker_image: apache/kafka:3.6.0
|
|||
|
||||
Using the image in a docker container
|
||||
-------------------------------------
|
||||
- The image uses the kafka downloaded from provided kafka url
|
||||
- The image can be run in a container in default mode by running
|
||||
`docker run -p 9092:9092 <image-name:tag>`
|
||||
- Default configs run kafka in kraft mode with plaintext listners on 9092 port.
|
||||
- Once user provided config properties are provided default configs will get replaced.
|
||||
- User can provide kafka configs following two ways:-
|
||||
- By mounting folder containing property files
|
||||
- Mount the folder containing kafka property files to `/mnt/shared/config`
|
||||
- These files will replace the default config files
|
||||
- Using environment variables
|
||||
- Kafka properties defined via env variables will override properties defined in file input
|
||||
- If properties are provided via environment variables only, default configs will be replaced by user provided properties
|
||||
- Input format for env variables:-
|
||||
- Replace . with _
|
||||
- Replace _ with __(double underscore)
|
||||
- Replace - with ___(triple underscore)
|
||||
- Prefix the result with KAFKA_
|
||||
- Examples:
|
||||
- For abc.def, use KAFKA_ABC_DEF
|
||||
- For abc-def, use KAFKA_ABC___DEF
|
||||
- For abc_def, use KAFKA_ABC__DEF
|
||||
- Hence order of precedence of properties is the following:-
|
||||
- Env variable (highest)
|
||||
- File input
|
||||
- Default configs (only when there is no user provided config)
|
||||
- Any env variable that is commonly used in starting kafka(for example, CLUSTER_ID) can be supplied to docker container and it will be available when kafka starts
|
||||
Please check [this](./examples/README.md) for usage guide of the docker image.
|
||||
|
||||
Steps to release docker image
|
||||
-----------------------------
|
||||
|
|
|
|||
|
|
@ -0,0 +1,197 @@
|
|||
Kafka Docker Image Usage Guide
|
||||
==============================
|
||||
|
||||
Introduction
|
||||
------------
|
||||
|
||||
This document contains usage guide as well as examples for docker image. Docker compose files are provided in this directory for the example use cases.
|
||||
|
||||
Kafka server can be started using following ways:-
|
||||
- Default configs
|
||||
- File input
|
||||
- Environment variables
|
||||
|
||||
Running on default configs
|
||||
--------------------------
|
||||
|
||||
If no user provided configs are passed to docker container or configs provided are empty, default configs will be used (configs that are packaged in kafka tarball). If any user provided config is provided, default configs will not be used.
|
||||
|
||||
Use file input for providing configs
|
||||
------------------------------------
|
||||
|
||||
- This method requires users to provide path to a local folder which contains kafka property files and mount it to docker container using docker volume.
|
||||
- It replaces the default config file present in docker container.
|
||||
- Mount the folder containing kafka property files to `/mnt/shared/config` in docker container.
|
||||
- Command `docker run --volume path/to/property/folder:/mnt/shared/config -p 9092:9092 apache/kafka:{{fullDotVersion}}` can be used to mount the folder containing property files.
|
||||
- Property files will be only read by the docker container.
|
||||
|
||||
Using Environment Variables
|
||||
---------------------------
|
||||
|
||||
- Kafka property defined via env variables will override the value of that property defined in file input and default config.
|
||||
- If properties are provided via environment variables only, default configs will be replaced by user provided properties.
|
||||
- To construct the environment key variable name for server.properties configs, following steps can be followed:-
|
||||
- Replace . with _
|
||||
- Replace _ with __(double underscore)
|
||||
- Replace - with ___(triple underscore)
|
||||
- Prefix the result with KAFKA_
|
||||
- Examples:
|
||||
- For abc.def, use KAFKA_ABC_DEF
|
||||
- For abc-def, use KAFKA_ABC___DEF
|
||||
- For abc_def, use KAFKA_ABC__DEF
|
||||
|
||||
- To provide configs to log4j property files, following points should be considered:-
|
||||
- log4j properties provided via environment variables will be appended to the default properties file (log4j properties files bundled with kafka)
|
||||
- `KAFKA_LOG4J_ROOT_LOGLEVEL` can be provided to set the value of log4j.rootLogger in log4j.properties and tools-log4j.properties
|
||||
- log4j loggers can be added to log4j.properties by setting them in KAFKA_LOG4J_LOGGERS environment variable in a single comma separated string
|
||||
- Example:
|
||||
- Assuming that KAFKA_LOG4J_LOGGERS='property1=value1,property2=value2' environment variable is provided to docker container.
|
||||
- log4j.logger.property1=value1 and log4j.logger.property2=value2 will be added to the log4j.properties file inside docker container.
|
||||
|
||||
- Environment variables commonly used in Kafka can be provided via environment variables, for example `CLUSTER_ID`.
|
||||
- Command `docker run --env CONFIG_NAME=CONFIG_VALUE -p 9092:9092 apache/kafka:{{fullDotVersion}}` can be used to provide environment variables to docker container
|
||||
- Note that it is recommended to use docker compose files to provide configs using environment variables.
|
||||
|
||||
Running in SSL mode
|
||||
-------------------
|
||||
|
||||
- Recommended way to run in ssl mode is by mounting secrets on `/etc/kafka/secrets` in docker container and providing configs following through environment variables (`KAFKA_SSL_KEYSTORE_FILENAME`, `KAFKA_SSL_KEYSTORE_CREDENTIALS`, `KAFKA_SSL_KEY_CREDENTIALS`, `KAFKA_SSL_TRUSTSTORE_FILENAME` and `KAFKA_SSL_TRUSTSTORE_CREDENTIALS`) to let the docker image scripts extract passwords and populate correct paths in server.properties.
|
||||
- Please ensure appropriate `KAFKA_ADVERTISED_LISTENERS` are provided through environment variables to enable SSL mode in Kafka server, i.e. it should contain an `SSL` listener.
|
||||
- Alternatively property file input can be used to provide ssl properties.
|
||||
- Make sure you set location of truststore and keystore correctly when using file input. See example for file input in `jvm/single-node/file-input` for better clarity.
|
||||
- Note that advertised.listeners property needs to be provided along with SSL properties in file input and cannot be provided through environment variable separately.
|
||||
- In conclusion, ssl properties with advertised.listeners should be treated as a group and provided in file input or environment variables in it's entirety.
|
||||
- In case ssl properties are provided both through file input and environment variables, environment variable properties will override the file input properties, just as mentioned in the beginning of this section.
|
||||
|
||||
Examples
|
||||
--------
|
||||
|
||||
- `jvm` directory contains docker compose files for some example configs to run `apache/kafka` docker image.
|
||||
- Run the commands from root of the repository.
|
||||
- Checkout `single-node` examples for quick small examples to play around with.
|
||||
- `cluster` contains multi node examples, for `combined` mode as well as `isolated` mode.
|
||||
- Kafka server running on docker container can be accessed using cli scripts or your own client code.
|
||||
- Make sure jars are built, if you decide to use cli scripts of this repo.
|
||||
|
||||
Single Node
|
||||
-----------
|
||||
|
||||
- These examples are for understanding various ways inputs can be provided and kafka can be configured in docker container.
|
||||
- Examples are present inside `jvm/single-node` directory.
|
||||
- Plaintext:-
|
||||
- This is the simplest compose file.
|
||||
- We are using environment variables purely for providing configs.
|
||||
- `KAFKA_LISTENERS` is getting supplied. But if it was not provided, defaulting would have kicked in and we would have used `KAFKA_ADVERTISED_LISTENERS` to generate `KAFKA_LISTENERS`, by replacing the host with `0.0.0.0`.
|
||||
- Note that we have provided a `CLUSTER_ID`, but it's not mandatory as there is a default cluster id present in container.
|
||||
- We had to provide `KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR` and set it explicitly to 1, because if we don't provide it default value provided by kafka will be taken which is 3.
|
||||
- We have also set hostname of the container. It can be kept same as the container name for clarity.
|
||||
- To run the example:-
|
||||
```
|
||||
# Run from root of the repo
|
||||
$ docker compose -f docker/examples/jvm/single-node/plaintext/docker-compose.yml up
|
||||
```
|
||||
- To produce messages using client scripts:-
|
||||
```
|
||||
# Run from root of the repo
|
||||
$ bin/kafka-console-producer.sh --topic test --bootstrap-server localhost:9092
|
||||
```
|
||||
- SSL:-
|
||||
- Note that here we are using environment variables to pass configs.
|
||||
- Notice how secrets folder is mounted to docker container.
|
||||
- In case of environment variable it is mandatory to keep the files in `/etc/kafka/secrets` folder in docker container, given that the path of the files will be derived from that, as we are just providing file names in other SSL configs.
|
||||
- To run the example:-
|
||||
```
|
||||
# Run from root of the repo
|
||||
$ docker compose -f docker/examples/jvm/single-node/ssl/docker-compose.yml up
|
||||
```
|
||||
- To produce messages using client scripts:-
|
||||
```
|
||||
# Run from root of the repo
|
||||
$ bin/kafka-console-producer.sh --topic test --bootstrap-server localhost:9093 --producer.config ./docker/examples/fixtures/client-secrets/client-ssl.properties
|
||||
```
|
||||
- File Input:-
|
||||
- Here ssl configs are provided via file input.
|
||||
- Notice that now full file path is needed for the configs.
|
||||
- Note that there is extra volume mount now.
|
||||
- Configs provided via environment variable will override the file input configs.
|
||||
- To run the example:-
|
||||
```
|
||||
# Run from root of the repo
|
||||
$ docker compose -f docker/examples/jvm/single-node/file-input/docker-compose.yml up
|
||||
```
|
||||
- To produce messages using client scripts:-
|
||||
```
|
||||
# Run from root of the repo
|
||||
$ bin/kafka-console-producer.sh --topic test --bootstrap-server localhost:9093 --producer.config ./docker/examples/fixtures/client-secrets/client-ssl.properties
|
||||
```
|
||||
|
||||
Multi Node Cluster
|
||||
------------------
|
||||
|
||||
- These examples are for real world usecases where multiple nodes of kafka are required.
|
||||
- Combined:-
|
||||
- Examples are present in `jvm/cluster/combined` directory.
|
||||
- Plaintext:-
|
||||
- Each broker must expose a unique port to host machine.
|
||||
- For example broker-1, broker2 and broker3 are listening on port 9092, they're exposing it to the host via ports 29092, 39092 and 49092 respectively.
|
||||
- Here important thing to note is that to ensure that kafka brokers are accessible both to clients as well as to each other we have introduced an additional listener.
|
||||
- PLAINTEXT is supposed to be listener accessible to other brokers.
|
||||
- The inter broker listener advertised by the brokers is exposed on container's hostname. This is done so that brokers can find each other in docker network.
|
||||
- PLAINTEXT_HOST is supposed to be listener accessible to the clients.
|
||||
- The port advertised for host machine is done on localhost, as this is the domain (in this example) that client will use to connect with kafka running inside docker container.
|
||||
- Here we take advantage of hostname set for each broker and set the listner accordingly.
|
||||
- To run the example:-
|
||||
```
|
||||
# Run from root of the repo
|
||||
$ docker compose -f docker/examples/jvm/cluster/combined/plaintext/docker-compose.yml up
|
||||
```
|
||||
- To access using client script:-
|
||||
```
|
||||
# Run from root of the repo
|
||||
$ bin/kafka-console-producer.sh --topic quickstart-events --bootstrap-server localhost:29092
|
||||
```
|
||||
- SSL:-
|
||||
- Similar to Plaintext example, for inter broker communication in SSL mode, SSL-INTERNAL is required.
|
||||
- Note that `KAFKA_SSL_ENDPOINT_IDENTIFICATION_ALGORITHM` is set to empty as hostname was not set in credentials. This won't be needed in production usecases.
|
||||
- Also note that for example we have used the same credentials for all brokers. Make sure each broker has it's own secrets.
|
||||
- To run the example:-
|
||||
```
|
||||
# Run from root of the repo
|
||||
$ docker compose -f docker/examples/jvm/cluster/combined/ssl/docker-compose.yml up
|
||||
```
|
||||
- To produce messages using client scripts:-
|
||||
```
|
||||
# Run from root of the repo
|
||||
$ bin/kafka-console-producer.sh --topic test --bootstrap-server localhost:29093 --producer.config ./docker/examples/fixtures/client-secrets/client-ssl.properties
|
||||
```
|
||||
- Isolated:-
|
||||
- Examples are present in `jvm/cluster/isolated` directory.
|
||||
- Plaintext:-
|
||||
- Here controllers and brokers are configured separately.
|
||||
- It's a good practice to define that brokers depend on controllers.
|
||||
- In this case also we have same listeners setup as mentioned in combined case.
|
||||
- To run the example:-
|
||||
```
|
||||
# Run from root of the repo
|
||||
$ docker compose -f docker/examples/jvm/cluster/isolated/plaintext/docker-compose.yml up
|
||||
```
|
||||
- To access using client script:-
|
||||
```
|
||||
# Run from root of the repo
|
||||
$ bin/kafka-console-producer.sh --topic quickstart-events --bootstrap-server localhost:29092
|
||||
```
|
||||
- SSL:-
|
||||
- Pretty much same as combined example, with controllers and brokers separated now.
|
||||
- Note that `SSL-INTERNAL` is only for inter broker communication and controllers are using `PLAINTEXT`.
|
||||
- To run the example:-
|
||||
```
|
||||
# Run from root of the repo
|
||||
$ docker compose -f docker/examples/jvm/cluster/isolated/ssl/docker-compose.yml up
|
||||
```
|
||||
- To produce messages using client scripts:-
|
||||
```
|
||||
# Run from root of the repo
|
||||
$ bin/kafka-console-producer.sh --topic test --bootstrap-server localhost:29093 --producer.config ./docker/examples/fixtures/client-secrets/client-ssl.properties
|
||||
```
|
||||
|
||||
- Note that the examples are meant to be tried one at a time, make sure you close an example server before trying out the other to avoid conflicts.
|
||||
|
|
@ -0,0 +1,26 @@
|
|||
# Licensed to the Apache Software Foundation (ASF) under one or more
|
||||
# contributor license agreements. See the NOTICE file distributed with
|
||||
# this work for additional information regarding copyright ownership.
|
||||
# The ASF licenses this file to You under the Apache License, Version 2.0
|
||||
# (the "License"); you may not use this file except in compliance with
|
||||
# the License. You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
security.protocol=SSL
|
||||
# Provide path to the truststore file
|
||||
ssl.truststore.location=./docker/examples/fixtures/secrets/kafka.truststore.jks
|
||||
# This the password for the example keystores and truststores
|
||||
ssl.truststore.password=abcdefgh
|
||||
# Provide path to the keystore file
|
||||
ssl.keystore.location=./docker/examples/fixtures/client-secrets/client.keystore.jks
|
||||
ssl.keystore.password=abcdefgh
|
||||
ssl.key.password=abcdefgh
|
||||
ssl.client.auth=required
|
||||
ssl.endpoint.identification.algorithm=
|
||||
Binary file not shown.
|
|
@ -0,0 +1,31 @@
|
|||
# Licensed to the Apache Software Foundation (ASF) under one or more
|
||||
# contributor license agreements. See the NOTICE file distributed with
|
||||
# this work for additional information regarding copyright ownership.
|
||||
# The ASF licenses this file to You under the Apache License, Version 2.0
|
||||
# (the "License"); you may not use this file except in compliance with
|
||||
# the License. You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
advertised.listeners=PLAINTEXT_HOST://localhost:9092,SSL://localhost:9093,PLAINTEXT://broker:19092
|
||||
controller.listener.names=CONTROLLER
|
||||
group.initial.rebalance.delay.ms=0
|
||||
inter.broker.listener.name=PLAINTEXT
|
||||
listener.security.protocol.map=PLAINTEXT:PLAINTEXT,SSL:SSL,CONTROLLER:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
|
||||
log.dirs=/tmp/kraft-combined-logs
|
||||
offsets.topic.replication.factor=1
|
||||
process.roles=broker
|
||||
ssl.client.auth=required
|
||||
ssl.key.password=abcdefgh
|
||||
ssl.keystore.location=/etc/kafka/secrets/kafka01.keystore.jks
|
||||
ssl.keystore.password=abcdefgh
|
||||
ssl.truststore.location=/etc/kafka/secrets/kafka.truststore.jks
|
||||
ssl.truststore.password=abcdefgh
|
||||
transaction.state.log.min.isr=1
|
||||
transaction.state.log.replication.factor=1
|
||||
Binary file not shown.
Binary file not shown.
|
|
@ -0,0 +1 @@
|
|||
abcdefgh
|
||||
|
|
@ -0,0 +1 @@
|
|||
abcdefgh
|
||||
|
|
@ -0,0 +1 @@
|
|||
abcdefgh
|
||||
|
|
@ -0,0 +1,83 @@
|
|||
# Licensed to the Apache Software Foundation (ASF) under one or more
|
||||
# contributor license agreements. See the NOTICE file distributed with
|
||||
# this work for additional information regarding copyright ownership.
|
||||
# The ASF licenses this file to You under the Apache License, Version 2.0
|
||||
# (the "License"); you may not use this file except in compliance with
|
||||
# the License. You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
---
|
||||
version: '2'
|
||||
services:
|
||||
kafka-1:
|
||||
image: apache/kafka:latest
|
||||
hostname: kafka-1
|
||||
container_name: kafka-1
|
||||
ports:
|
||||
- 29092:9092
|
||||
environment:
|
||||
KAFKA_NODE_ID: 1
|
||||
KAFKA_PROCESS_ROLES: 'broker,controller'
|
||||
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: 'CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT'
|
||||
KAFKA_CONTROLLER_QUORUM_VOTERS: '1@kafka-1:9093,2@kafka-2:9093,3@kafka-3:9093'
|
||||
KAFKA_LISTENERS: 'PLAINTEXT://:19092,CONTROLLER://:9093,PLAINTEXT_HOST://:9092'
|
||||
KAFKA_INTER_BROKER_LISTENER_NAME: 'PLAINTEXT'
|
||||
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka-1:19092,PLAINTEXT_HOST://localhost:29092
|
||||
KAFKA_CONTROLLER_LISTENER_NAMES: 'CONTROLLER'
|
||||
CLUSTER_ID: '4L6g3nShT-eMCtK--X86sw'
|
||||
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
|
||||
KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
|
||||
KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
|
||||
KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
|
||||
KAFKA_LOG_DIRS: '/tmp/kraft-combined-logs'
|
||||
|
||||
kafka-2:
|
||||
image: apache/kafka:latest
|
||||
hostname: kafka-2
|
||||
container_name: kafka-2
|
||||
ports:
|
||||
- 39092:9092
|
||||
environment:
|
||||
KAFKA_NODE_ID: 2
|
||||
KAFKA_PROCESS_ROLES: 'broker,controller'
|
||||
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: 'CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT'
|
||||
KAFKA_CONTROLLER_QUORUM_VOTERS: '1@kafka-1:9093,2@kafka-2:9093,3@kafka-3:9093'
|
||||
KAFKA_LISTENERS: 'PLAINTEXT://:19092,CONTROLLER://:9093,PLAINTEXT_HOST://:9092'
|
||||
KAFKA_INTER_BROKER_LISTENER_NAME: 'PLAINTEXT'
|
||||
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka-2:19092,PLAINTEXT_HOST://localhost:39092
|
||||
KAFKA_CONTROLLER_LISTENER_NAMES: 'CONTROLLER'
|
||||
CLUSTER_ID: '4L6g3nShT-eMCtK--X86sw'
|
||||
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
|
||||
KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
|
||||
KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
|
||||
KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
|
||||
KAFKA_LOG_DIRS: '/tmp/kraft-combined-logs'
|
||||
|
||||
kafka-3:
|
||||
image: apache/kafka:latest
|
||||
hostname: kafka-3
|
||||
container_name: kafka-3
|
||||
ports:
|
||||
- 49092:9092
|
||||
environment:
|
||||
KAFKA_NODE_ID: 3
|
||||
KAFKA_PROCESS_ROLES: 'broker,controller'
|
||||
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: 'CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT'
|
||||
KAFKA_CONTROLLER_QUORUM_VOTERS: '1@kafka-1:9093,2@kafka-2:9093,3@kafka-3:9093'
|
||||
KAFKA_LISTENERS: 'PLAINTEXT://:19092,CONTROLLER://:9093,PLAINTEXT_HOST://:9092'
|
||||
KAFKA_INTER_BROKER_LISTENER_NAME: 'PLAINTEXT'
|
||||
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka-3:19092,PLAINTEXT_HOST://localhost:49092
|
||||
KAFKA_CONTROLLER_LISTENER_NAMES: 'CONTROLLER'
|
||||
CLUSTER_ID: '4L6g3nShT-eMCtK--X86sw'
|
||||
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
|
||||
KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
|
||||
KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
|
||||
KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
|
||||
KAFKA_LOG_DIRS: '/tmp/kraft-combined-logs'
|
||||
|
|
@ -0,0 +1,113 @@
|
|||
# Licensed to the Apache Software Foundation (ASF) under one or more
|
||||
# contributor license agreements. See the NOTICE file distributed with
|
||||
# this work for additional information regarding copyright ownership.
|
||||
# The ASF licenses this file to You under the Apache License, Version 2.0
|
||||
# (the "License"); you may not use this file except in compliance with
|
||||
# the License. You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
---
|
||||
version: '2'
|
||||
services:
|
||||
kafka-1:
|
||||
image: apache/kafka:latest
|
||||
hostname: kafka-1
|
||||
container_name: kafka-1
|
||||
ports:
|
||||
- 29093:9093
|
||||
volumes:
|
||||
- ../../../../fixtures/secrets:/etc/kafka/secrets
|
||||
environment:
|
||||
KAFKA_NODE_ID: 1
|
||||
KAFKA_PROCESS_ROLES: 'broker,controller'
|
||||
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: 'SSL:SSL,CONTROLLER:PLAINTEXT,SSL-INTERNAL:SSL'
|
||||
KAFKA_CONTROLLER_QUORUM_VOTERS: '1@kafka-1:29092,2@kafka-2:29092,3@kafka-3:29092'
|
||||
KAFKA_LISTENERS: 'SSL-INTERNAL://:19093,CONTROLLER://kafka-1:29092,SSL://:9093'
|
||||
KAFKA_INTER_BROKER_LISTENER_NAME: 'SSL-INTERNAL'
|
||||
KAFKA_SECURITY_PROTOCOL: SSL
|
||||
KAFKA_ADVERTISED_LISTENERS: SSL-INTERNAL://kafka-1:19093,SSL://localhost:29093
|
||||
KAFKA_CONTROLLER_LISTENER_NAMES: 'CONTROLLER'
|
||||
CLUSTER_ID: '4L6g3nShT-eMCtK--X86sw'
|
||||
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
|
||||
KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
|
||||
KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
|
||||
KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
|
||||
KAFKA_LOG_DIRS: '/tmp/kraft-combined-logs'
|
||||
KAFKA_SSL_KEYSTORE_FILENAME: 'kafka01.keystore.jks'
|
||||
KAFKA_SSL_KEYSTORE_CREDENTIALS: 'kafka_keystore_creds'
|
||||
KAFKA_SSL_KEY_CREDENTIALS: 'kafka_ssl_key_creds'
|
||||
KAFKA_SSL_TRUSTSTORE_FILENAME: 'kafka.truststore.jks'
|
||||
KAFKA_SSL_TRUSTSTORE_CREDENTIALS: 'kafka_truststore_creds'
|
||||
KAFKA_SSL_CLIENT_AUTH: 'required'
|
||||
KAFKA_SSL_ENDPOINT_IDENTIFICATION_ALGORITHM: ""
|
||||
|
||||
kafka-2:
|
||||
image: apache/kafka:latest
|
||||
hostname: kafka-2
|
||||
container_name: kafka-2
|
||||
ports:
|
||||
- 39093:9093
|
||||
volumes:
|
||||
- ../../../../fixtures/secrets:/etc/kafka/secrets
|
||||
environment:
|
||||
KAFKA_NODE_ID: 2
|
||||
KAFKA_PROCESS_ROLES: 'broker,controller'
|
||||
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: 'SSL:SSL,CONTROLLER:PLAINTEXT,SSL-INTERNAL:SSL'
|
||||
KAFKA_CONTROLLER_QUORUM_VOTERS: '1@kafka-1:29092,2@kafka-2:29092,3@kafka-3:29092'
|
||||
KAFKA_LISTENERS: 'SSL-INTERNAL://:19093,CONTROLLER://kafka-2:29092,SSL://:9093'
|
||||
KAFKA_INTER_BROKER_LISTENER_NAME: 'SSL-INTERNAL'
|
||||
KAFKA_SECURITY_PROTOCOL: SSL
|
||||
KAFKA_ADVERTISED_LISTENERS: SSL-INTERNAL://kafka-2:19093,SSL://localhost:39093
|
||||
KAFKA_CONTROLLER_LISTENER_NAMES: 'CONTROLLER'
|
||||
CLUSTER_ID: '4L6g3nShT-eMCtK--X86sw'
|
||||
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
|
||||
KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
|
||||
KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
|
||||
KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
|
||||
KAFKA_LOG_DIRS: '/tmp/kraft-combined-logs'
|
||||
KAFKA_SSL_KEYSTORE_FILENAME: 'kafka01.keystore.jks'
|
||||
KAFKA_SSL_KEYSTORE_CREDENTIALS: 'kafka_keystore_creds'
|
||||
KAFKA_SSL_KEY_CREDENTIALS: 'kafka_ssl_key_creds'
|
||||
KAFKA_SSL_TRUSTSTORE_FILENAME: 'kafka.truststore.jks'
|
||||
KAFKA_SSL_TRUSTSTORE_CREDENTIALS: 'kafka_truststore_creds'
|
||||
KAFKA_SSL_CLIENT_AUTH: 'required'
|
||||
KAFKA_SSL_ENDPOINT_IDENTIFICATION_ALGORITHM: ""
|
||||
|
||||
kafka-3:
|
||||
image: apache/kafka:latest
|
||||
hostname: kafka-3
|
||||
container_name: kafka-3
|
||||
ports:
|
||||
- 49093:9093
|
||||
volumes:
|
||||
- ../../../../fixtures/secrets:/etc/kafka/secrets
|
||||
environment:
|
||||
KAFKA_NODE_ID: 3
|
||||
KAFKA_PROCESS_ROLES: 'broker,controller'
|
||||
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: 'SSL:SSL,CONTROLLER:PLAINTEXT,SSL-INTERNAL:SSL'
|
||||
KAFKA_CONTROLLER_QUORUM_VOTERS: '1@kafka-1:29092,2@kafka-2:29092,3@kafka-3:29092'
|
||||
KAFKA_LISTENERS: 'SSL-INTERNAL://:19093,CONTROLLER://kafka-3:29092,SSL://:9093'
|
||||
KAFKA_INTER_BROKER_LISTENER_NAME: 'SSL-INTERNAL'
|
||||
KAFKA_SECURITY_PROTOCOL: SSL
|
||||
KAFKA_ADVERTISED_LISTENERS: SSL-INTERNAL://kafka-3:19093,SSL://localhost:49093
|
||||
KAFKA_CONTROLLER_LISTENER_NAMES: 'CONTROLLER'
|
||||
CLUSTER_ID: '4L6g3nShT-eMCtK--X86sw'
|
||||
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
|
||||
KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
|
||||
KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
|
||||
KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
|
||||
KAFKA_LOG_DIRS: '/tmp/kraft-combined-logs'
|
||||
KAFKA_SSL_KEYSTORE_FILENAME: 'kafka01.keystore.jks'
|
||||
KAFKA_SSL_KEYSTORE_CREDENTIALS: 'kafka_keystore_creds'
|
||||
KAFKA_SSL_KEY_CREDENTIALS: 'kafka_ssl_key_creds'
|
||||
KAFKA_SSL_TRUSTSTORE_FILENAME: 'kafka.truststore.jks'
|
||||
KAFKA_SSL_TRUSTSTORE_CREDENTIALS: 'kafka_truststore_creds'
|
||||
KAFKA_SSL_CLIENT_AUTH: 'required'
|
||||
KAFKA_SSL_ENDPOINT_IDENTIFICATION_ALGORITHM: ""
|
||||
|
|
@ -0,0 +1,143 @@
|
|||
# Licensed to the Apache Software Foundation (ASF) under one or more
|
||||
# contributor license agreements. See the NOTICE file distributed with
|
||||
# this work for additional information regarding copyright ownership.
|
||||
# The ASF licenses this file to You under the Apache License, Version 2.0
|
||||
# (the "License"); you may not use this file except in compliance with
|
||||
# the License. You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
---
|
||||
version: '2'
|
||||
services:
|
||||
controller-1:
|
||||
image: apache/kafka:latest
|
||||
environment:
|
||||
KAFKA_NODE_ID: 1
|
||||
KAFKA_PROCESS_ROLES: 'controller'
|
||||
KAFKA_CONTROLLER_QUORUM_VOTERS: '1@controller-1:9093,2@controller-2:9093,3@controller-3:9093'
|
||||
KAFKA_INTER_BROKER_LISTENER_NAME: 'PLAINTEXT'
|
||||
KAFKA_CONTROLLER_LISTENER_NAMES: 'CONTROLLER'
|
||||
KAFKA_LISTENERS: 'CONTROLLER://:9093'
|
||||
CLUSTER_ID: '4L6g3nShT-eMCtK--X86sw'
|
||||
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
|
||||
KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
|
||||
KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
|
||||
KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
|
||||
KAFKA_LOG_DIRS: '/tmp/kraft-combined-logs'
|
||||
|
||||
controller-2:
|
||||
image: apache/kafka:latest
|
||||
environment:
|
||||
KAFKA_NODE_ID: 2
|
||||
KAFKA_PROCESS_ROLES: 'controller'
|
||||
KAFKA_CONTROLLER_QUORUM_VOTERS: '1@controller-1:9093,2@controller-2:9093,3@controller-3:9093'
|
||||
KAFKA_INTER_BROKER_LISTENER_NAME: 'PLAINTEXT'
|
||||
KAFKA_CONTROLLER_LISTENER_NAMES: 'CONTROLLER'
|
||||
KAFKA_LISTENERS: 'CONTROLLER://:9093'
|
||||
CLUSTER_ID: '4L6g3nShT-eMCtK--X86sw'
|
||||
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
|
||||
KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
|
||||
KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
|
||||
KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
|
||||
KAFKA_LOG_DIRS: '/tmp/kraft-combined-logs'
|
||||
|
||||
controller-3:
|
||||
image: apache/kafka:latest
|
||||
environment:
|
||||
KAFKA_NODE_ID: 3
|
||||
KAFKA_PROCESS_ROLES: 'controller'
|
||||
KAFKA_CONTROLLER_QUORUM_VOTERS: '1@controller-1:9093,2@controller-2:9093,3@controller-3:9093'
|
||||
KAFKA_INTER_BROKER_LISTENER_NAME: 'PLAINTEXT'
|
||||
KAFKA_CONTROLLER_LISTENER_NAMES: 'CONTROLLER'
|
||||
KAFKA_LISTENERS: 'CONTROLLER://:9093'
|
||||
CLUSTER_ID: '4L6g3nShT-eMCtK--X86sw'
|
||||
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
|
||||
KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
|
||||
KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
|
||||
KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
|
||||
KAFKA_LOG_DIRS: '/tmp/kraft-combined-logs'
|
||||
|
||||
kafka-1:
|
||||
image: apache/kafka:latest
|
||||
ports:
|
||||
- 29092:9092
|
||||
hostname: kafka-1
|
||||
container_name: kafka-1
|
||||
environment:
|
||||
KAFKA_NODE_ID: 4
|
||||
KAFKA_PROCESS_ROLES: 'broker'
|
||||
KAFKA_CONTROLLER_QUORUM_VOTERS: '1@controller-1:9093,2@controller-2:9093,3@controller-3:9093'
|
||||
KAFKA_LISTENERS: 'PLAINTEXT://:19092,PLAINTEXT_HOST://:9092'
|
||||
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: 'CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT'
|
||||
KAFKA_INTER_BROKER_LISTENER_NAME: 'PLAINTEXT'
|
||||
KAFKA_ADVERTISED_LISTENERS: 'PLAINTEXT://kafka-1:19092,PLAINTEXT_HOST://localhost:29092'
|
||||
KAFKA_CONTROLLER_LISTENER_NAMES: 'CONTROLLER'
|
||||
CLUSTER_ID: '4L6g3nShT-eMCtK--X86sw'
|
||||
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
|
||||
KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
|
||||
KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
|
||||
KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
|
||||
KAFKA_LOG_DIRS: '/tmp/kraft-combined-logs'
|
||||
depends_on:
|
||||
- controller-1
|
||||
- controller-2
|
||||
- controller-3
|
||||
|
||||
kafka-2:
|
||||
image: apache/kafka:latest
|
||||
ports:
|
||||
- 39092:9092
|
||||
hostname: kafka-2
|
||||
container_name: kafka-2
|
||||
environment:
|
||||
KAFKA_NODE_ID: 5
|
||||
KAFKA_PROCESS_ROLES: 'broker'
|
||||
KAFKA_CONTROLLER_QUORUM_VOTERS: '1@controller-1:9093,2@controller-2:9093,3@controller-3:9093'
|
||||
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: 'CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT'
|
||||
KAFKA_LISTENERS: 'PLAINTEXT://:19092,PLAINTEXT_HOST://:9092'
|
||||
KAFKA_INTER_BROKER_LISTENER_NAME: 'PLAINTEXT'
|
||||
KAFKA_ADVERTISED_LISTENERS: 'PLAINTEXT://kafka-2:19092,PLAINTEXT_HOST://localhost:39092'
|
||||
KAFKA_CONTROLLER_LISTENER_NAMES: 'CONTROLLER'
|
||||
CLUSTER_ID: '4L6g3nShT-eMCtK--X86sw'
|
||||
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
|
||||
KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
|
||||
KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
|
||||
KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
|
||||
KAFKA_LOG_DIRS: '/tmp/kraft-combined-logs'
|
||||
depends_on:
|
||||
- controller-1
|
||||
- controller-2
|
||||
- controller-3
|
||||
|
||||
kafka-3:
|
||||
image: apache/kafka:latest
|
||||
ports:
|
||||
- 49092:9092
|
||||
hostname: kafka-3
|
||||
container_name: kafka-3
|
||||
environment:
|
||||
KAFKA_NODE_ID: 6
|
||||
KAFKA_PROCESS_ROLES: 'broker'
|
||||
KAFKA_CONTROLLER_QUORUM_VOTERS: '1@controller-1:9093,2@controller-2:9093,3@controller-3:9093'
|
||||
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: 'CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT'
|
||||
KAFKA_LISTENERS: 'PLAINTEXT://:19092,PLAINTEXT_HOST://:9092'
|
||||
KAFKA_INTER_BROKER_LISTENER_NAME: 'PLAINTEXT'
|
||||
KAFKA_ADVERTISED_LISTENERS: 'PLAINTEXT://kafka-3:19092,PLAINTEXT_HOST://localhost:49092'
|
||||
KAFKA_CONTROLLER_LISTENER_NAMES: 'CONTROLLER'
|
||||
CLUSTER_ID: '4L6g3nShT-eMCtK--X86sw'
|
||||
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
|
||||
KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
|
||||
KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
|
||||
KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
|
||||
KAFKA_LOG_DIRS: '/tmp/kraft-combined-logs'
|
||||
depends_on:
|
||||
- controller-1
|
||||
- controller-2
|
||||
- controller-3
|
||||
|
|
@ -0,0 +1,173 @@
|
|||
# Licensed to the Apache Software Foundation (ASF) under one or more
|
||||
# contributor license agreements. See the NOTICE file distributed with
|
||||
# this work for additional information regarding copyright ownership.
|
||||
# The ASF licenses this file to You under the Apache License, Version 2.0
|
||||
# (the "License"); you may not use this file except in compliance with
|
||||
# the License. You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
---
|
||||
version: '2'
|
||||
services:
|
||||
controller-1:
|
||||
image: apache/kafka:latest
|
||||
environment:
|
||||
KAFKA_NODE_ID: 1
|
||||
KAFKA_PROCESS_ROLES: 'controller'
|
||||
KAFKA_CONTROLLER_QUORUM_VOTERS: '1@controller-1:29092,2@controller-2:29092,3@controller-3:29092'
|
||||
KAFKA_INTER_BROKER_LISTENER_NAME: 'PLAINTEXT'
|
||||
KAFKA_CONTROLLER_LISTENER_NAMES: 'CONTROLLER'
|
||||
KAFKA_LISTENERS: 'CONTROLLER://:29092'
|
||||
CLUSTER_ID: '4L6g3nShT-eMCtK--X86sw'
|
||||
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
|
||||
KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
|
||||
KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
|
||||
KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
|
||||
KAFKA_LOG_DIRS: '/tmp/kraft-combined-logs'
|
||||
|
||||
controller-2:
|
||||
image: apache/kafka:latest
|
||||
environment:
|
||||
KAFKA_NODE_ID: 2
|
||||
KAFKA_PROCESS_ROLES: 'controller'
|
||||
KAFKA_CONTROLLER_QUORUM_VOTERS: '1@controller-1:29092,2@controller-2:29092,3@controller-3:29092'
|
||||
KAFKA_INTER_BROKER_LISTENER_NAME: 'PLAINTEXT'
|
||||
KAFKA_CONTROLLER_LISTENER_NAMES: 'CONTROLLER'
|
||||
KAFKA_LISTENERS: 'CONTROLLER://:29092'
|
||||
CLUSTER_ID: '4L6g3nShT-eMCtK--X86sw'
|
||||
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
|
||||
KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
|
||||
KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
|
||||
KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
|
||||
KAFKA_LOG_DIRS: '/tmp/kraft-combined-logs'
|
||||
|
||||
controller-3:
|
||||
image: apache/kafka:latest
|
||||
environment:
|
||||
KAFKA_NODE_ID: 3
|
||||
KAFKA_PROCESS_ROLES: 'controller'
|
||||
KAFKA_CONTROLLER_QUORUM_VOTERS: '1@controller-1:29092,2@controller-2:29092,3@controller-3:29092'
|
||||
KAFKA_INTER_BROKER_LISTENER_NAME: 'PLAINTEXT'
|
||||
KAFKA_CONTROLLER_LISTENER_NAMES: 'CONTROLLER'
|
||||
KAFKA_LISTENERS: 'CONTROLLER://:29092'
|
||||
CLUSTER_ID: '4L6g3nShT-eMCtK--X86sw'
|
||||
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
|
||||
KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
|
||||
KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
|
||||
KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
|
||||
KAFKA_LOG_DIRS: '/tmp/kraft-combined-logs'
|
||||
|
||||
kafka-1:
|
||||
image: apache/kafka:latest
|
||||
hostname: kafka-1
|
||||
container_name: kafka-1
|
||||
ports:
|
||||
- 29093:9093
|
||||
volumes:
|
||||
- ../../../../fixtures/secrets:/etc/kafka/secrets
|
||||
environment:
|
||||
KAFKA_NODE_ID: 4
|
||||
KAFKA_PROCESS_ROLES: 'broker'
|
||||
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: 'SSL:SSL,CONTROLLER:PLAINTEXT,SSL-INTERNAL:SSL'
|
||||
KAFKA_LISTENERS: 'SSL-INTERNAL://:19093,SSL://:9093'
|
||||
KAFKA_CONTROLLER_QUORUM_VOTERS: '1@controller-1:29092,2@controller-2:29092,3@controller-3:29092'
|
||||
KAFKA_INTER_BROKER_LISTENER_NAME: 'SSL-INTERNAL'
|
||||
KAFKA_SECURITY_PROTOCOL: SSL
|
||||
KAFKA_ADVERTISED_LISTENERS: SSL-INTERNAL://kafka-1:19093,SSL://localhost:29093
|
||||
KAFKA_CONTROLLER_LISTENER_NAMES: 'CONTROLLER'
|
||||
CLUSTER_ID: '4L6g3nShT-eMCtK--X86sw'
|
||||
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
|
||||
KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
|
||||
KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
|
||||
KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
|
||||
KAFKA_LOG_DIRS: '/tmp/kraft-combined-logs'
|
||||
KAFKA_SSL_KEYSTORE_FILENAME: 'kafka01.keystore.jks'
|
||||
KAFKA_SSL_KEYSTORE_CREDENTIALS: 'kafka_keystore_creds'
|
||||
KAFKA_SSL_KEY_CREDENTIALS: 'kafka_ssl_key_creds'
|
||||
KAFKA_SSL_TRUSTSTORE_FILENAME: 'kafka.truststore.jks'
|
||||
KAFKA_SSL_TRUSTSTORE_CREDENTIALS: 'kafka_truststore_creds'
|
||||
KAFKA_SSL_CLIENT_AUTH: 'required'
|
||||
KAFKA_SSL_ENDPOINT_IDENTIFICATION_ALGORITHM: ""
|
||||
depends_on:
|
||||
- controller-1
|
||||
- controller-2
|
||||
- controller-3
|
||||
|
||||
kafka-2:
|
||||
image: apache/kafka:latest
|
||||
hostname: kafka-2
|
||||
container_name: kafka-2
|
||||
ports:
|
||||
- 39093:9093
|
||||
volumes:
|
||||
- ../../../../fixtures/secrets:/etc/kafka/secrets
|
||||
environment:
|
||||
KAFKA_NODE_ID: 5
|
||||
KAFKA_PROCESS_ROLES: 'broker'
|
||||
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: 'SSL:SSL,CONTROLLER:PLAINTEXT,SSL-INTERNAL:SSL'
|
||||
KAFKA_CONTROLLER_QUORUM_VOTERS: '1@controller-1:29092,2@controller-2:29092,3@controller-3:29092'
|
||||
KAFKA_LISTENERS: 'SSL-INTERNAL://:19093,SSL://:9093'
|
||||
KAFKA_INTER_BROKER_LISTENER_NAME: 'SSL-INTERNAL'
|
||||
KAFKA_SECURITY_PROTOCOL: SSL
|
||||
KAFKA_ADVERTISED_LISTENERS: SSL-INTERNAL://kafka-2:19093,SSL://localhost:39093
|
||||
KAFKA_CONTROLLER_LISTENER_NAMES: 'CONTROLLER'
|
||||
CLUSTER_ID: '4L6g3nShT-eMCtK--X86sw'
|
||||
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
|
||||
KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
|
||||
KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
|
||||
KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
|
||||
KAFKA_LOG_DIRS: '/tmp/kraft-combined-logs'
|
||||
KAFKA_SSL_KEYSTORE_FILENAME: 'kafka01.keystore.jks'
|
||||
KAFKA_SSL_KEYSTORE_CREDENTIALS: 'kafka_keystore_creds'
|
||||
KAFKA_SSL_KEY_CREDENTIALS: 'kafka_ssl_key_creds'
|
||||
KAFKA_SSL_TRUSTSTORE_FILENAME: 'kafka.truststore.jks'
|
||||
KAFKA_SSL_TRUSTSTORE_CREDENTIALS: 'kafka_truststore_creds'
|
||||
KAFKA_SSL_CLIENT_AUTH: 'required'
|
||||
KAFKA_SSL_ENDPOINT_IDENTIFICATION_ALGORITHM: ""
|
||||
depends_on:
|
||||
- controller-1
|
||||
- controller-2
|
||||
- controller-3
|
||||
|
||||
kafka-3:
|
||||
image: apache/kafka:latest
|
||||
hostname: kafka-3
|
||||
container_name: kafka-3
|
||||
ports:
|
||||
- 49093:9093
|
||||
volumes:
|
||||
- ../../../../fixtures/secrets:/etc/kafka/secrets
|
||||
environment:
|
||||
KAFKA_NODE_ID: 6
|
||||
KAFKA_PROCESS_ROLES: 'broker'
|
||||
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: 'SSL:SSL,CONTROLLER:PLAINTEXT,SSL-INTERNAL:SSL'
|
||||
KAFKA_CONTROLLER_QUORUM_VOTERS: '1@controller-1:29092,2@controller-2:29092,3@controller-3:29092'
|
||||
KAFKA_LISTENERS: 'SSL-INTERNAL://:19093,SSL://:9093'
|
||||
KAFKA_INTER_BROKER_LISTENER_NAME: 'SSL-INTERNAL'
|
||||
KAFKA_SECURITY_PROTOCOL: SSL
|
||||
KAFKA_ADVERTISED_LISTENERS: SSL-INTERNAL://kafka-3:19093,SSL://localhost:49093
|
||||
KAFKA_CONTROLLER_LISTENER_NAMES: 'CONTROLLER'
|
||||
CLUSTER_ID: '4L6g3nShT-eMCtK--X86sw'
|
||||
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
|
||||
KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
|
||||
KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
|
||||
KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
|
||||
KAFKA_LOG_DIRS: '/tmp/kraft-combined-logs'
|
||||
KAFKA_SSL_KEYSTORE_FILENAME: 'kafka01.keystore.jks'
|
||||
KAFKA_SSL_KEYSTORE_CREDENTIALS: 'kafka_keystore_creds'
|
||||
KAFKA_SSL_KEY_CREDENTIALS: 'kafka_ssl_key_creds'
|
||||
KAFKA_SSL_TRUSTSTORE_FILENAME: 'kafka.truststore.jks'
|
||||
KAFKA_SSL_TRUSTSTORE_CREDENTIALS: 'kafka_truststore_creds'
|
||||
KAFKA_SSL_CLIENT_AUTH: 'required'
|
||||
KAFKA_SSL_ENDPOINT_IDENTIFICATION_ALGORITHM: ""
|
||||
depends_on:
|
||||
- controller-1
|
||||
- controller-2
|
||||
- controller-3
|
||||
|
|
@ -0,0 +1,37 @@
|
|||
# Licensed to the Apache Software Foundation (ASF) under one or more
|
||||
# contributor license agreements. See the NOTICE file distributed with
|
||||
# this work for additional information regarding copyright ownership.
|
||||
# The ASF licenses this file to You under the Apache License, Version 2.0
|
||||
# (the "License"); you may not use this file except in compliance with
|
||||
# the License. You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
---
|
||||
version: '2'
|
||||
services:
|
||||
broker:
|
||||
image: apache/kafka:latest
|
||||
hostname: broker
|
||||
container_name: broker
|
||||
ports:
|
||||
- "9092:9092"
|
||||
- "9093:9093"
|
||||
volumes:
|
||||
- ../../../fixtures/secrets:/etc/kafka/secrets
|
||||
- ../../../fixtures/file-input:/mnt/shared/config
|
||||
environment:
|
||||
# Environment variables used by kafka scripts will be needed in case of File input.
|
||||
CLUSTER_ID: '4L6g3nShT-eMCtK--X86sw'
|
||||
# Set properties not provided in the file input
|
||||
KAFKA_NODE_ID: 1
|
||||
KAFKA_CONTROLLER_QUORUM_VOTERS: '1@broker:29093'
|
||||
KAFKA_LISTENERS: 'PLAINTEXT_HOST://:9092,SSL://:9093,CONTROLLER://:29093,PLAINTEXT://:19092'
|
||||
# Override an existing property
|
||||
KAFKA_PROCESS_ROLES: 'broker,controller'
|
||||
|
|
@ -0,0 +1,39 @@
|
|||
# Licensed to the Apache Software Foundation (ASF) under one or more
|
||||
# contributor license agreements. See the NOTICE file distributed with
|
||||
# this work for additional information regarding copyright ownership.
|
||||
# The ASF licenses this file to You under the Apache License, Version 2.0
|
||||
# (the "License"); you may not use this file except in compliance with
|
||||
# the License. You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
---
|
||||
version: '2'
|
||||
services:
|
||||
broker:
|
||||
image: apache/kafka:latest
|
||||
hostname: broker
|
||||
container_name: broker
|
||||
ports:
|
||||
- '9092:9092'
|
||||
environment:
|
||||
KAFKA_NODE_ID: 1
|
||||
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: 'CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT'
|
||||
KAFKA_ADVERTISED_LISTENERS: 'PLAINTEXT_HOST://localhost:9092,PLAINTEXT://broker:19092'
|
||||
KAFKA_PROCESS_ROLES: 'broker,controller'
|
||||
KAFKA_CONTROLLER_QUORUM_VOTERS: '1@broker:29093'
|
||||
KAFKA_LISTENERS: 'CONTROLLER://:29093,PLAINTEXT_HOST://:9092,PLAINTEXT://:19092'
|
||||
KAFKA_INTER_BROKER_LISTENER_NAME: 'PLAINTEXT'
|
||||
KAFKA_CONTROLLER_LISTENER_NAMES: 'CONTROLLER'
|
||||
CLUSTER_ID: '4L6g3nShT-eMCtK--X86sw'
|
||||
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
|
||||
KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
|
||||
KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
|
||||
KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
|
||||
KAFKA_LOG_DIRS: '/tmp/kraft-combined-logs'
|
||||
|
|
@ -0,0 +1,48 @@
|
|||
# Licensed to the Apache Software Foundation (ASF) under one or more
|
||||
# contributor license agreements. See the NOTICE file distributed with
|
||||
# this work for additional information regarding copyright ownership.
|
||||
# The ASF licenses this file to You under the Apache License, Version 2.0
|
||||
# (the "License"); you may not use this file except in compliance with
|
||||
# the License. You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
---
|
||||
version: '2'
|
||||
services:
|
||||
broker:
|
||||
image: apache/kafka:latest
|
||||
hostname: broker
|
||||
container_name: broker
|
||||
ports:
|
||||
- '9093:9093'
|
||||
volumes:
|
||||
- ../../../fixtures/secrets:/etc/kafka/secrets
|
||||
environment:
|
||||
KAFKA_NODE_ID: 1
|
||||
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: 'SSL:SSL,CONTROLLER:PLAINTEXT,SSL-INTERNAL:SSL'
|
||||
KAFKA_ADVERTISED_LISTENERS: 'SSL://localhost:9093,SSL-INTERNAL://broker:19093'
|
||||
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
|
||||
KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
|
||||
KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
|
||||
KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
|
||||
KAFKA_PROCESS_ROLES: 'broker,controller'
|
||||
KAFKA_CONTROLLER_QUORUM_VOTERS: '1@broker:29093'
|
||||
KAFKA_LISTENERS: 'SSL://:9093,CONTROLLER://:29093,SSL-INTERNAL://:19093'
|
||||
KAFKA_INTER_BROKER_LISTENER_NAME: 'SSL-INTERNAL'
|
||||
KAFKA_CONTROLLER_LISTENER_NAMES: 'CONTROLLER'
|
||||
KAFKA_LOG_DIRS: '/tmp/kraft-combined-logs'
|
||||
CLUSTER_ID: '4L6g3nShT-eMCtK--X86sw'
|
||||
KAFKA_SSL_KEYSTORE_FILENAME: 'kafka01.keystore.jks'
|
||||
KAFKA_SSL_KEYSTORE_CREDENTIALS: 'kafka_keystore_creds'
|
||||
KAFKA_SSL_KEY_CREDENTIALS: 'kafka_ssl_key_creds'
|
||||
KAFKA_SSL_TRUSTSTORE_FILENAME: 'kafka.truststore.jks'
|
||||
KAFKA_SSL_TRUSTSTORE_CREDENTIALS: 'kafka_truststore_creds'
|
||||
KAFKA_SSL_CLIENT_AUTH: 'required'
|
||||
KAFKA_SSL_ENDPOINT_IDENTIFICATION_ALGORITHM: ""
|
||||
|
|
@ -0,0 +1,58 @@
|
|||
<!--
|
||||
Licensed to the Apache Software Foundation (ASF) under one or more
|
||||
contributor license agreements. See the NOTICE file distributed with
|
||||
this work for additional information regarding copyright ownership.
|
||||
The ASF licenses this file to You under the Apache License, Version 2.0
|
||||
(the "License"); you may not use this file except in compliance with
|
||||
the License. You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing, software
|
||||
distributed under the License is distributed on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
See the License for the specific language governing permissions and
|
||||
limitations under the License.
|
||||
-->
|
||||
|
||||
<script>
|
||||
<!--#include virtual="js/templateData.js" -->
|
||||
</script>
|
||||
|
||||
<script id="docker-template" type="text/x-handlebars-template">
|
||||
<h4 class="anchor-heading">Introduction</h4>
|
||||
|
||||
<p>
|
||||
<a href="https://www.docker.com/">Docker</a> is a popular container runtime. Docker images for Apache Kafka can be found on <a href="https://hub.docker.com/r/apache/kafka">Docker Hub</a> and are available from version 3.7.0.
|
||||
</p>
|
||||
|
||||
<h4 class="anchor-heading">Getting the kafka docker image</h4>
|
||||
|
||||
<p>
|
||||
Docker image can be pulled from Docker Hub using the following command:-
|
||||
</p>
|
||||
|
||||
<pre class="line-numbers"><code class="language-bash">$ docker pull apache/kafka:{{fullDotVersion}}</code></pre>
|
||||
|
||||
<p>
|
||||
If you want to fetch the latest version of the docker image use following command:-
|
||||
</p>
|
||||
|
||||
<pre class="line-numbers"><code class="language-bash">$ docker pull apache/kafka:latest</code></pre>
|
||||
|
||||
<h4 class="anchor-heading">Start kafka with default configs</h4>
|
||||
|
||||
<p>
|
||||
Run docker image on default port 9092:-
|
||||
</p>
|
||||
|
||||
<pre class="line-numbers"><code class="language-bash">$ docker run -p 9092:9092 apache/kafka:{{fullDotVersion}}</code></pre>
|
||||
|
||||
<h4 class="anchor-heading">Usage guide</h4>
|
||||
|
||||
<p>
|
||||
Detailed instructions for using the docker image are mentioned <a href="https://github.com/apache/kafka/blob/trunk/docker/examples/README.md">here</a>.
|
||||
</p>
|
||||
</script>
|
||||
|
||||
<div class="p-docker"></div>
|
||||
|
|
@ -72,6 +72,8 @@
|
|||
<!--#include virtual="ecosystem.html" -->
|
||||
<h3 class="anchor-heading"><a id="upgrade" class="anchor-link"></a><a href="#upgrade">1.5 Upgrading From Previous Versions</a></h3>
|
||||
<!--#include virtual="upgrade.html" -->
|
||||
<h3 class="anchor-heading"><a id="docker" class="anchor-link"></a><a href="#docker">1.6 Docker</a></h3>
|
||||
<!--#include virtual="docker.html" -->
|
||||
|
||||
<h2 class="anchor-heading"><a id="api" class="anchor-link"></a><a href="#api">2. APIs</a></h2>
|
||||
|
||||
|
|
|
|||
|
|
@ -76,6 +76,10 @@ $ bin/kafka-server-start.sh config/server.properties</code></pre>
|
|||
Kafka with KRaft
|
||||
</h5>
|
||||
|
||||
<p>Kafka can be run using KRaft mode using local scripts and downloaded files or the docker image. Follow one of the sections below but not both to start the kafka server.</p>
|
||||
|
||||
<h5>Using downloaded files</h5>
|
||||
|
||||
<p>
|
||||
Generate a Cluster UUID
|
||||
</p>
|
||||
|
|
@ -94,6 +98,20 @@ $ bin/kafka-server-start.sh config/server.properties</code></pre>
|
|||
|
||||
<pre class="line-numbers"><code class="language-bash">$ bin/kafka-server-start.sh config/kraft/server.properties</code></pre>
|
||||
|
||||
<h5>Using docker image</h5>
|
||||
|
||||
<p>
|
||||
Get the docker image
|
||||
</p>
|
||||
|
||||
<pre class="line-numbers"><code class="language-bash">$ docker pull apache/kafka:{{fullDotVersion}}</code></pre>
|
||||
|
||||
<p>
|
||||
Start the kafka docker container
|
||||
</p>
|
||||
|
||||
<pre class="line-numbers"><code class="language-bash">$ docker run -p 9092:9092 apache/kafka:{{fullDotVersion}}</code></pre>
|
||||
|
||||
<p>
|
||||
Once the Kafka server has successfully launched, you will have a basic Kafka environment running and ready to use.
|
||||
</p>
|
||||
|
|
|
|||
|
|
@ -27,6 +27,7 @@
|
|||
<li><a href="#quickstart">1.3 Quick Start</a>
|
||||
<li><a href="#ecosystem">1.4 Ecosystem</a>
|
||||
<li><a href="#upgrade">1.5 Upgrading</a>
|
||||
<li><a href="#docker">1.6 Docker</a>
|
||||
</ul>
|
||||
</li>
|
||||
<li><a href="#api">2. APIs</a>
|
||||
|
|
|
|||
Loading…
Reference in New Issue