root
/
kafka
mirror of https://github.com/apache/kafka.git
1
0
Fork 0
Code Issues Actions 2 Packages Projects Releases Wiki Activity

KAFKA-13731: Allow standalone workers to be started without providing any connector configurations (#11890) 

Reviewers: Yash Mayya <yash.mayya@gmail.com>, Mickael Maison <mickael.maison@gmail.com>
This commit is contained in:
Chris Egerton 2022-11-30 11:49:24 -05:00 committed by GitHub
parent 9baa5023d3
commit 548348c9e7
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
2 changed files with 7 additions and 7 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
connect/runtime/src/main/java/org/apache/kafka/connect/cli/ConnectStandalone.java
View File

@ -58,8 +58,8 @@ public class ConnectStandalone {
public static void main(String[] args) {
if (args.length < 2 || Arrays.asList(args).contains("--help")) {
log.info("Usage: ConnectStandalone worker.properties connector1.properties [connector2.properties ...]");
if (args.length < 1 || Arrays.asList(args).contains("--help")) {
log.info("Usage: ConnectStandalone worker.properties [connector1.properties connector2.properties ...]");
Exit.exit(1);
}

10
docs/connect.html
View File

@ -41,7 +41,7 @@
<p>In standalone mode all work is performed in a single process. This configuration is simpler to setup and get started with and may be useful in situations where only one worker makes sense (e.g. collecting log files), but it does not benefit from some of the features of Kafka Connect such as fault tolerance. You can start a standalone process with the following command:</p>
<pre class="brush: bash;">
&gt; bin/connect-standalone.sh config/connect-standalone.properties connector1.properties [connector2.properties ...]</pre>
&gt; bin/connect-standalone.sh config/connect-standalone.properties [connector1.properties connector2.properties ...]</pre>
<p>The first parameter is the configuration for the worker. This includes settings such as the Kafka connection parameters, serialization format, and how frequently to commit offsets. The provided example should work well with a local cluster running with the default configuration provided by <code>config/server.properties</code>. It will require tweaking to use with a different configuration or production deployment. All workers (both standalone and distributed) require a few configs:</p>
<ul>
@ -60,7 +60,7 @@
<p>Starting with 2.3.0, client configuration overrides can be configured individually per connector by using the prefixes <code>producer.override.</code> and <code>consumer.override.</code> for Kafka sources or Kafka sinks respectively. These overrides are included with the rest of the connector's configuration properties.</p>
<p>The remaining parameters are connector configuration files. You may include as many as you want, but all will execute within the same process (on different threads).</p>
<p>The remaining parameters are connector configuration files. You may include as many as you want, but all will execute within the same process (on different threads). You can also choose not to specify any connector configuration files on the command line, and instead use the REST API to create connectors at runtime after your standalone worker starts.</p>
<p>Distributed mode handles automatic balancing of work, allows you to scale up (or down) dynamically, and offers fault tolerance both in the active tasks and for configuration and offset commit data. Execution is very similar to standalone mode:</p>
@ -82,7 +82,7 @@
<h4><a id="connect_configuring" href="#connect_configuring">Configuring Connectors</a></h4>
<p>Connector configurations are simple key-value mappings. For standalone mode these are defined in a properties file and passed to the Connect process on the command line. In distributed mode, they will be included in the JSON payload for the request that creates (or modifies) the connector.</p>
<p>Connector configurations are simple key-value mappings. In both standalone and distributed mode, they are included in the JSON payload for the REST request that creates (or modifies) the connector. In standalone mode these can also be defined in a properties file and passed to the Connect process on the command line.</p>
<p>Most configurations are connector dependent, so they can't be outlined here. However, there are a few common options:</p>
@ -254,7 +254,7 @@ predicates.IsBar.pattern=bar</pre>
<h4><a id="connect_rest" href="#connect_rest">REST API</a></h4>
<p>Since Kafka Connect is intended to be run as a service, it also provides a REST API for managing connectors. The REST API server can be configured using the <code>listeners</code> configuration option.
<p>Since Kafka Connect is intended to be run as a service, it also provides a REST API for managing connectors. This REST API is available in both standalone and distributed mode. The REST API server can be configured using the <code>listeners</code> configuration option.
This field should contain a list of listeners in the following format: <code>protocol://host:port,protocol2://host2:port2</code>. Currently supported protocols are <code>http</code> and <code>https</code>.
For example:</p>
@ -284,7 +284,7 @@ listeners=http://localhost:8080,https://localhost:8443</pre>
<li><code>ssl.client.auth</code></li>
</ul>
<p>The REST API is used not only by users to monitor / manage Kafka Connect. It is also used for the Kafka Connect cross-cluster communication. Requests received on the follower nodes REST API will be forwarded to the leader node REST API.
<p>The REST API is used not only by users to monitor / manage Kafka Connect. In distributed mode, it is also used for the Kafka Connect cross-cluster communication. Some requests received on the follower nodes REST API will be forwarded to the leader node REST API.
In case the URI under which is given host reachable is different from the URI which it listens on, the configuration options <code>rest.advertised.host.name</code>, <code>rest.advertised.port</code> and <code>rest.advertised.listener</code>
can be used to change the URI which will be used by the follower nodes to connect with the leader. When using both HTTP and HTTPS listeners, the <code>rest.advertised.listener</code> option can be also used to define which listener
will be used for the cross-cluster communication. When using HTTPS for communication between nodes, the same <code>ssl.*</code> or <code>listeners.https</code> options will be used to configure the HTTPS client.</p>