Various converters (AvroConverter and JsonConverter) produce a
SchemaAndValue consisting of a logical schema type and a java.util.Date.
This is a fix for SchemaProjector to properly handle the Date.
Author: Robert Yokota <rayokota@gmail.com>
Reviewers: Konstantine Karantasis <konstantine@confluent.io>, Ewen Cheslack-Postava <ewen@confluent.io>
Closes#5736 from rayokota/KAFKA-7476
(cherry picked from commit 3edd8e7333)
Signed-off-by: Ewen Cheslack-Postava <me@ewencp.org>
In some tests, the check monitoring the JMX tool log output doesn’t quite wait long enough before failing. Increasing the timeout from 10 to 20 seconds.
This is a port of #5605 for the 11.3 branch
Reviewers: John Roesler <john@confluent.io>, Matthias J. Sax <matthias@confluent.io>, Guozhang Wang <wangguoz@gmail.com>
This ensures that the whole test suite is executed even
if there are failures. It currently stops at a module
boundary if there are any failures. There is a discussion
to change the gradle default to stop after the first test failure:
https://github.com/gradle/gradle/issues/6513
`--continue` is recommended for CI in that discussion.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Dong Lin <lindong28@gmail.com>
Closes#5599 from ijuma/jenkins-script-continue
ACL updates currently get `(currentAcls, currentVersion)` for the resource from ZK and do a conditional update using `(currentAcls+newAcl, currentVersion)`. This supports concurrent atomic updates if the resource path already exists in ZK. If the path doesn't exist, we currently do a conditional createOrUpdate using `(newAcl, 0)`. So two brokers adding acls using `(newAcl1, 0)` and `(newAcl2, 0)` will result in one broker creating the path and setting newAcl1, while the other broker can potentially update the path with `(newAcl2, 0)`, losing newAcl1. The timing window is very small, but we have seen intermittent failures in `SimpleAclAuthorizerTest.testHighConcurrencyModificationOfResourceAcls` as a result of this window.
ZooKeeper listener for change notifications should be created before loading the ACL cache to avoid timing window if acls are modified when broker is starting up.
Reviewers: Jun Rao <junrao@gmail.com>, Ismael Juma <ismael@confluent.io>
Maven Central dropped support for all versions but
TLS 1.2, so dependency resolution fails if Gradle
builds run with JDK 7. 2.0 and trunk require JDK 8,
but every other version is affected. Gradle 4.8.1
fixes the issue by enabling TLS 1.2 by default even
when JDK 7 is used.
Reviewers: Rajini Sivaram <rajinisivaram@googlemail.com>
https://issues.apache.org/jira/browse/KAFKA-7058
* Summary of testing strategy: Added new unit test
Author: Gunnar Morling <gunnar.morling@googlemail.com>
Reviewers: Randall Hauch <rhauch@gmail.com>, Ewen Cheslack-Postava <ewen@confluent.io>
Closes#5225 from gunnarmorling/KAFKA-7058
(cherry picked from commit be846d833c)
Signed-off-by: Ewen Cheslack-Postava <me@ewencp.org>
This is a cherry-pick PR from #5207
1. add the committed offsets to checkpointable offset map.
2. add the restoration integration test for the source KTable case.
Now that we support re-initializing state stores, we need to clear the segments when the store is closed so that they can be re-opened.
Author: Damian Guy <damian.guy@gmail.com>
Reviewers: Bill Bejeck <bbejeck@gmail.com>, Guozhang Wang <wangguoz@gmail.com>, Ted Yu <yuzhihong@gmail.com>
Closes#4324 from dguy/kafka-6360
This could be backported to older branches to reduce the extra log warning messages there, too.
Running Connect system tests in this branch builder job: https://jenkins.confluent.io/job/system-test-kafka-branch-builder/1773/
Author: Randall Hauch <rhauch@gmail.com>
Reviewers: Konstantine Karantasis <konstantine@confluent.io>, Ewen Cheslack-Postava <ewen@confluent.io>
Closes#5151 from rhauch/kafka-7009
(cherry picked from commit 7a1f555676)
Signed-off-by: Ewen Cheslack-Postava <me@ewencp.org>
As titled, not starting new transaction since during restoration producer would have not activity and hence may cause txn expiration. Also delay starting new txn in resuming until initializing topology.
Reviewers: Matthias J. Sax <mjsax@apache.org>, Bill Bejeck <bill@confluent.io>
Initialize topology after state store restoration.
Although IMHO updating some of the existing tests demonstrates the correct order of operations, I'll probably add an integration test, but I wanted to get this PR in for feedback on the approach.
Author: Bill Bejeck <bill@confluent.io>
Reviewers: Guozhang Wang <wangguoz@gmail.com>, Matthias J. Sax <mjsax@apache.org>
Closes#4415 from bbejeck/KAFKA-6205_restore_state_stores_before_initializing_topology
minor log4j edits
As Frederic reported on mailing list under the subject "kafka-streams Invalid transition attempted from state READY to state ABORTING_TRANSACTION", producer#abortTransaction should only be called when transactionInFlight is true.
Reviewers: Guozhang Wang <wangguoz@gmail.com>, Matthias J. Sax <matthias@confluent.io>
Previously, we failed to remove sensors from the parentSensors map, effectively a memory leak.
Add a test to verify that removed sensors get removed from the underlying registry as well as the parentSensors map.
Reviewers: Guozhang Wang <wangguoz@gmail.com>
This is a change to improve resource cleanup for sink tasks and source tasks. Now `Task.stop()` is called from both `WorkerSinkTask.close()` and `WorkerSourceTask.close()`.
It is called from `WorkerXXXTask.close()` since this method is called in the `finally` block of `WorkerTask.run()`, and Connect developers use `stop()` to clean up resources.
Author: Robert Yokota <rayokota@gmail.com>
Reviewers: Randall Hauch <rhauch@gmail.com>, Ewen Cheslack-Postava <ewen@confluent.io>
Closes#5020 from rayokota/K6566-improve-connect-resource-cleanup
(cherry picked from commit ee8abb2f70)
Signed-off-by: Ewen Cheslack-Postava <me@ewencp.org>
The leader must explicitly check if requested leader epoch is undefined, and return undefined offset so that the follower can fall back to truncating to high watermark. Otherwise, if the leader also is not tracking leader epochs, it may return its LEO, which will the follower to truncate to the incorrect offset.
We should remove the map entry from mbeans if it becomes
empty during metric removal.
Reviewers: Manikumar Reddy <manikumar.reddy@gmail.com>, Satish Duggana <satish.duggana@gmail.com>, Ismael Juma <ismael@juma.me.uk>
The tests were excluded from the backport to the 0.11.0
branch due to conflicts.
Debian installer packages are no longer available for Java 7.
Also upgrade AMI to latest ubuntu/trusty 14 amd64 as the older
one is no longer available.
Note that this only changes the JDK used to build and run
the system tests. We still have Jenkins jobs that compile
and run the JUnit tests with Java 7 so that we don't use
features that are only available in newer Java versions.
Changed WorkerSinkTaskContext to only resume the consumer topic partitions when the connector/task is not in the paused state.
The context tracks the set of topic partitions that are explicitly paused/resumed by the connector, and when the WorkerSinkTask resumes the tasks it currently resumes all topic partitions *except* those that are still explicitly paused in the context. Therefore, the change above should result in the desired behavior.
Several debug statements were added to record when the context is called by the connector.
This can be backported to older releases, since this bug goes back to 0.10 or 0.9.
Author: Randall Hauch <rhauch@gmail.com>
Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>
Closes#4716 from rhauch/kafka-6661
(cherry picked from commit e7ef719a5b)
Signed-off-by: Ewen Cheslack-Postava <me@ewencp.org>
This will allow us to trace leaked instances back to the job,
so that we can figure out what happened and fix the leak.
Reviewers: Ismael Juma <ismael@juma.me.uk>
This is a safe guard against users that do not properly close() the reporter, which causes
the process to hang even if its main() method returns. This was the case with Kafka Streams
apps in some cases in Kafka < 1.1.0 (KAFKA-6383). Without this fix, this behavior can make
it difficult to use this class in some types of system tests.
Author: Ewen Cheslack-Postava <me@ewencp.org>
Reviewers: Guozhang Wang <wangguoz@gmail.com>, Ismael Juma <ismael@juma.me.uk>
Closes#4629 from ewencp/http-metrics-dont-block-shutdown
(cherry picked from commit 73fb1bffe1)
Signed-off-by: Ewen Cheslack-Postava <me@ewencp.org>
ensure that brokers are registered at ZK before start() returns
Author: Matthias J. Sax <matthias@confluent.io>
Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>, Damian Guy <damian@confluent.io>, Guozhang Wang <guozhang@confluent.io>
Currently port forwarding is setup for HttpMetricsCollector when the Service's start_node method is called, but not canceled during stop. This hasn't presented a problem so far because we don't have tests that use this *and* restart the service. However, if a test/service does that, it will throw an exception since the port is already bound.
This just does the cleanup when stopping so a subsequent attempt to start again will succeed.
https://jenkins.confluent.io/job/system-test-kafka-branch-builder/1320 is a test run for a Test that uses ProducerPerformanceService, which in turn uses HttpMetricsCollector to validate the change.
Author: Ewen Cheslack-Postava <me@ewencp.org>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Apurva Mehta <apurva@confluent.io>
Closes#4604 from ewencp/cleanup-reverse-port-forward
(cherry picked from commit da379c95e4)
Signed-off-by: Ewen Cheslack-Postava <me@ewencp.org>
The log cleaner should not naively remove the partition from in progress map without checking the partition state. This may cause the other thread calling `LogCleanerManager.abortAndPauseCleaning()` to hang indefinitely.
Changed call to use the overload of ConnectSchema.validateValue() method with the field name passed in. Ensure that field in put call is not null.
Reviewers: Randall Hauch <rhauch@gmail.com>, Jason Gustafson <jason@confluent.io>
If an exception is encountered while sending data to a client connection, that connection is disconnected. If there are staged receives for that connection, they are tracked to process those records. However, if the exception was encountered during processing a `RequestChannel.Request`, the `KafkaChannel` for that connection is muted and won't be processed.
Disable processing of outstanding staged receives if a send fails. This stops the leak of the memory for pending requests and the file descriptor of the TCP socket.
Test that a channel is closed when an exception is raised while writing to a socket that has been closed by the client. Since sending a response requires acks != 0, allow specifying the required acks for test requests in SocketServerTest.scala.
Author: Graham Campbell <graham.campbell@salesforce.com>
Reviewers: Jason Gustafson <jason@confluent.io>, Rajini Sivaram <rajinisivaram@googlemail.com>, Ismael Juma <ismael@juma.me.uk>, Ted Yu <yuzhihong@gmail.com>
Robert Yokota
Randall Hauch
Matthias J. Sax
Bill Bejeck
Ismael Juma
John Roesler
Rajini Sivaram
Jason Gustafson
Matthias J. Sax
Anna Povzner
Gunnar Morling
Guozhang Wang
Damian Guy
Jagadesh Adireddi
Bill Bejeck
Gitomain
tedyu
Jarek Rudzinski
Max Zheng
Ewen Cheslack-Postava
Jiangjie (Becket) Qin
Yaswanth Kumar
Jeremy Custenborder
parafiend