Various converters (AvroConverter and JsonConverter) produce a
SchemaAndValue consisting of a logical schema type and a java.util.Date.
This is a fix for SchemaProjector to properly handle the Date.
Author: Robert Yokota <rayokota@gmail.com>
Reviewers: Konstantine Karantasis <konstantine@confluent.io>, Ewen Cheslack-Postava <ewen@confluent.io>
Closes#5736 from rayokota/KAFKA-7476
(cherry picked from commit 3edd8e7333)
Signed-off-by: Ewen Cheslack-Postava <me@ewencp.org>
https://issues.apache.org/jira/browse/KAFKA-7058
* Summary of testing strategy: Added new unit test
Author: Gunnar Morling <gunnar.morling@googlemail.com>
Reviewers: Randall Hauch <rhauch@gmail.com>, Ewen Cheslack-Postava <ewen@confluent.io>
Closes#5225 from gunnarmorling/KAFKA-7058
(cherry picked from commit be846d833c)
Signed-off-by: Ewen Cheslack-Postava <me@ewencp.org>
Changed WorkerSinkTaskContext to only resume the consumer topic partitions when the connector/task is not in the paused state.
The context tracks the set of topic partitions that are explicitly paused/resumed by the connector, and when the WorkerSinkTask resumes the tasks it currently resumes all topic partitions *except* those that are still explicitly paused in the context. Therefore, the change above should result in the desired behavior.
Several debug statements were added to record when the context is called by the connector.
This can be backported to older releases, since this bug goes back to 0.10 or 0.9.
Author: Randall Hauch <rhauch@gmail.com>
Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>
Closes#4716 from rhauch/kafka-6661
(cherry picked from commit e7ef719a5b)
Signed-off-by: Ewen Cheslack-Postava <me@ewencp.org>
Otherwise in this test the sink task goes through the pause/resume cycle with 0 assigned partitions, since the default metadata refresh interval is quite long
Author: Shikhar Bhushan <shikhar@confluent.io>
Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>
Closes#2313 from shikhar/kafka-4575
And improve readability by adding proper punctuations.
Author: Vahid Hashemian <vahidhashemian@us.ibm.com>
Reviewers: Jason Gustafson <jason@confluent.io>
Closes#2002 from vahidhashemian/doc/fix_typos
(cherry picked from commit e972d2afd3)
Signed-off-by: Jason Gustafson <jason@confluent.io>
Cleaner to just check once for optional & default value from the `convertToConnect()` function.
It also helps address an issue with conversions for logical type schemas that have default values and null as the included value. That test case is _probably_ not an issue in practice, since when using the `JsonConverter` to serialize a missing field with a default value, it will serialize the default value for the field. But in the face of JSON data streaming in from a topic being [generous on input, strict on output](http://tedwise.com/2009/05/27/generous-on-input-strict-on-output) seems best.
Author: Shikhar Bhushan <shikhar@confluent.io>
Reviewers: Randall Hauch <rhauch@gmail.com>, Jason Gustafson <jason@confluent.io>
Closes#1872 from shikhar/kafka-4183
The `JsonConverter` class has `LogicalTypeConverter` implementations for Date, Time, Timestamp, and Decimal, but these implementations fail when the input literal value (deserialized from the message) is null.
Test cases were added to check for these cases, and these failed before the `LogicalTypeConverter` implementations were fixed to consider whether the schema has a default value or is optional, similarly to how the `JsonToConnectTypeConverter` implementations do this. Once the fixes were made, the new tests pass.
Author: Randall Hauch <rhauch@gmail.com>
Reviewers: Shikhar Bhushan <shikhar@confluent.io>, Jason Gustafson <jason@confluent.io>
Closes#1867 from rhauch/kafka-4183
Invoke the statusListener.onFailure() callback on start failures so that the statusBackingStore is updated. This involved a fix to the putSafe() functionality which prevented any update that was not preceded by a (non-safe) put() from completing, so here when a connector or task is transitioning directly to FAILED.
Worker start methods can still throw if the same connector name or task ID is already registered with the worker, as this condition should not happen.
Author: Shikhar Bhushan <shikhar@confluent.io>
Reviewers: Jason Gustafson <jason@confluent.io>, Ewen Cheslack-Postava <ewen@confluent.io>
Closes#1778 from shikhar/distherder-stayup-take4
Author: Jason Gustafson <jason@confluent.io>
Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>, Ismael Juma <ismael@juma.me.uk>, Guozhang Wang <wangguoz@gmail.com>
Closes#1627 from hachikuji/KAFKA-3888
Author: Ewen Cheslack-Postava <me@ewencp.org>
Reviewers: Jason Gustafson, Gwen Shapira
Closes#1727 from ewencp/kafka-3847-per-task-producers and squashes the following commits:
7d39724 [Ewen Cheslack-Postava] Add timeout for closing producers.
98ec7f6 [Ewen Cheslack-Postava] KAFKA-3847: Use a separate producer per source task
ewencp I went down the list of connect configs and it looks like only the internal converter configs are mismarked. It looks like the `cluster` config that is present in the current docs is already gone. The only other values I can see arguing to change importance on are the ssl configs (marked high) but they are consistent with the producer/consumer config docs so that's at least consistent. Everything else marked high looks either mandatory or requires consideration in a production deployment to me.
Author: Dustin Cote <dustin@confluent.io>
Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>
Closes#1653 from cotedm/KAFKA-2932
Fix the test by using a more liberal timeout and forcing more frequent SinkTask.put() calls. Also add some logging to aid future debugging.
Author: Ewen Cheslack-Postava <me@ewencp.org>
Reviewers: Jason Gustafson <jason@confluent.io>, Ismael Juma <ismael@juma.me.uk>
Closes#1663 from ewencp/kafka-3935-fix-restart-system-test
Was just reading kafka source code, my favourite Friday afternoon activity, when I found these small grammatical errors in some `DataException` messages.
Could someone please review? ewencp dguy
Author: Laurier Mantel <laurier.mantel@shopify.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Ewen Cheslack-Postava <ewen@confluent.io>
Closes#1551 from LaurierMantel/maps-typos
And not the containing struct's default value.
The contribution is my original work and that I license the work to the project under the project's open source license.
ewencp
Author: Rollulus <roelboel@xs4all.nl>
Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>
Closes#1528 from rollulus/kafka-3864
ExecutorService needs to be shutdown on close, lest a zombie thread
prevent clean shutdown.
ewencp
Author: Peter Davis <peter.davis@expeditors.com>
Reviewers: Liquan Pei <liquanpei@gmail.com>, Ewen Cheslack-Postava <ewen@confluent.io>
Closes#1383 from davispw/KAFKA-3710
Author: Christian Posta <christian.posta@gmail.com>
Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>
Closes#1401 from christian-posta/ceposta-connect-class-cast-error
Author: Jason Gustafson <jason@confluent.io>
Reviewers: Grant Henke <granthenke@gmail.com>, Ewen Cheslack-Postava <ewen@confluent.io>
Closes#1322 from hachikuji/KAFKA-3659
hachikuji ewencp Can you take a look when you have time?
Author: Liquan Pei <liquanpei@gmail.com>
Reviewers: Jason Gustafson <jason@confluent.io>, Ewen Cheslack-Postava <ewen@confluent.io>
Closes#1248 from Ishiihara/kafka-3459
Robert Yokota
Gunnar Morling
Randall Hauch
Arjun Satish
Hooman Broujerdi
Shikhar Bhushan
Jason Gustafson
Vahid Hashemian
David Chen
Konstantine Karantasis
Ewen Cheslack-Postava
Dustin Cote
Liquan Pei
Laurier Mantel
Rollulus
Sasaki Toru
leisore
Peter Davis
Christian Posta
shikhar
dan norwood