mirror of https://github.com/apache/kafka.git
f6acfb0891
This is an initial patch implementing the basics of Copycat for KIP-26. The intent here is to start a review of the key pieces of the core API and get a reasonably functional, baseline, non-distributed implementation of Copycat in place to get things rolling. The current patch has a number of known issues that need to be addressed before a final version: * Some build-related issues. Specifically, requires some locally-installed dependencies (see below), ignores checkstyle for the runtime data library because it's lifted from Avro currently and likely won't last in its current form, and some Gradle task dependencies aren't quite right because I haven't gotten rid of the dependency on `core` (which should now be an easy patch since new consumer groups are in a much better state). * This patch currently depends on some Confluent trunk code because I prototyped with our Avro serializers w/ schema-registry support. We need to figure out what we want to provide as an example built-in set of serializers. Unlike core Kafka where we could ignore the issue, providing only ByteArray or String serializers, this is pretty central to how Copycat works. * This patch uses a hacked up version of Avro as its runtime data format. Not sure if we want to go through the entire API discussion just to get some basic code committed, so I filed KAFKA-2367 to handle that separately. The core connector APIs and the runtime data APIs are entirely orthogonal. * This patch needs some updates to get aligned with recent new consumer changes (specifically, I'm aware of the ConcurrentModificationException issue on exit). More generally, the new consumer is in flux but Copycat depends on it, so there are likely to be some negative interactions. * The layout feels a bit awkward to me right now because I ported it from a Maven layout. We don't have nearly the same level of granularity in Kafka currently (core and clients, plus the mostly ignored examples, log4j-appender, and a couple of contribs). We might want to reorganize, although keeping data+api separate from runtime and connector plugins is useful for minimizing dependencies. * There are a variety of other things (e.g., I'm not happy with the exception hierarchy/how they are currently handled, TopicPartition doesn't really need to be duplicated unless we want Copycat entirely isolated from the Kafka APIs, etc), but I expect those we'll cover in the review. Before commenting on the patch, it's probably worth reviewing https://issues.apache.org/jira/browse/KAFKA-2365 and https://issues.apache.org/jira/browse/KAFKA-2366 to get an idea of what I had in mind for a) what we ultimately want with all the Copycat patches and b) what we aim to cover in this initial patch. My hope is that we can use a WIP patch (after the current obvious deficiencies are addressed) while recognizing that we want to make iterative progress with a bunch of subsequent PRs. Author: Ewen Cheslack-Postava <me@ewencp.org> Reviewers: Ismael Juma, Gwen Shapira Closes #99 from ewencp/copycat and squashes the following commits: |
||
|---|---|---|
| bin | ||
| checkstyle | ||
| clients/src | ||
| config | ||
| contrib | ||
| copycat | ||
| core/src | ||
| dev-utils | ||
| examples | ||
| gradle | ||
| log4j-appender/src | ||
| system_test | ||
| tests | ||
| tools/src/main/java/org/apache/kafka/clients/tools | ||
| vagrant | ||
| .gitignore | ||
| .reviewboardrc | ||
| CONTRIBUTING.md | ||
| HEADER | ||
| LICENSE | ||
| NOTICE | ||
| README.md | ||
| Vagrantfile | ||
| build.gradle | ||
| doap_Kafka.rdf | ||
| gradle.properties | ||
| gradlew | ||
| gradlew.bat | ||
| kafka-merge-pr.py | ||
| kafka-patch-review.py | ||
| scala.gradle | ||
| settings.gradle | ||
| wrapper.gradle | ||
README.md
Apache Kafka
See our web site for details on the project.
You need to have gradle installed.
First bootstrap and download the wrapper
cd kafka_source_dir
gradle
Now everything else will work
Building a jar and running it
./gradlew jar
Follow instuctions in http://kafka.apache.org/documentation.html#quickstart
Building source jar
./gradlew srcJar
Building javadocs and scaladocs
./gradlew javadoc
./gradlew javadocJar # builds a jar from the javadocs
./gradlew scaladoc
./gradlew scaladocJar # builds a jar from the scaladocs
./gradlew docsJar # builds both javadoc and scaladoc jar
Running unit tests
./gradlew test
Forcing re-running unit tests w/o code change
./gradlew cleanTest test
Running a particular unit test
./gradlew -Dtest.single=RequestResponseSerializationTest core:test
Running a particular test method within a unit test
./gradlew core:test --tests kafka.api.test.ProducerFailureHandlingTest.testCannotSendToInternalTopic
./gradlew clients:test --tests org.apache.kafka.clients.producer.MetadataTest.testMetadataUpdateWaitTime
Running a particular unit test with log4j output
change the log4j setting in either clients/src/test/resources/log4j.properties or core/src/test/resources/log4j.properties
./gradlew -i -Dtest.single=RequestResponseSerializationTest core:test
Building a binary release gzipped tar ball
./gradlew clean
./gradlew releaseTarGz
The above command will fail if you haven't set up the signing key. To bypass signing the artifact, you can run
./gradlew releaseTarGz -x signArchives
The release file can be found inside ./core/build/distributions/.
Cleaning the build
./gradlew clean
Running a task on a particular version of Scala (either 2.10.5 or 2.11.7)
(If building a jar with a version other than 2.10, need to set SCALA_BINARY_VERSION variable or change it in bin/kafka-run-class.sh to run quick start.)
./gradlew -PscalaVersion=2.11.7 jar
./gradlew -PscalaVersion=2.11.7 test
./gradlew -PscalaVersion=2.11.7 releaseTarGz
Running a task for a specific project
This is for 'core', 'contrib:hadoop-consumer', 'contrib:hadoop-producer', 'examples' and 'clients' ./gradlew core:jar ./gradlew core:test
Listing all gradle tasks
./gradlew tasks
Building IDE project
./gradlew eclipse
./gradlew idea
Building the jar for all scala versions and for all projects
./gradlew jarAll
Running unit tests for all scala versions and for all projects
./gradlew testAll
Building a binary release gzipped tar ball for all scala versions
./gradlew releaseTarGzAll
Publishing the jar for all version of Scala and for all projects to maven
./gradlew uploadArchivesAll
Please note for this to work you should create/update ~/.gradle/gradle.properties and assign the following variables
mavenUrl=
mavenUsername=
mavenPassword=
signing.keyId=
signing.password=
signing.secretKeyRingFile=
Publishing the jars without signing to a local repository
./gradlew -Dorg.gradle.project.skipSigning=true -Dorg.gradle.project.mavenUrl=file://path/to/repo uploadArchivesAll
Building the test jar
./gradlew testJar
Determining how transitive dependencies are added
./gradlew core:dependencies --configuration runtime
Running checkstyle on the java code
./gradlew checkstyleMain checkstyleTest
Running in Vagrant
See vagrant/README.md.
Contribution
Apache Kafka is interested in building the community; we would welcome any thoughts or patches. You can reach us on the Apache mailing lists.
To contribute follow the instructions here:
We also welcome patches for the website and documentation which can be found here: