KAFKA-4345; Run decktape test for each pull request

As of now the ducktape tests that we have for kafka are not run for pull request. We can run these test using travis-ci. Here is a sample run:
https://travis-ci.org/raghavgautam/kafka/builds/170574293

Author: Raghav Kumar Gautam <raghav@apache.org>

Reviewers: Sriharsha Chintalapani <harsha@hortonworks.com>

Closes #2064 from raghavgautam/trunk

.travis.yml

@@ -0,0 +1,48 @@
+#  Licensed under the Apache License, Version 2.0 (the "License");
+#  you may not use this file except in compliance with the License.
+#  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+
+sudo: required
+dist: trusty
+language: java
+
+# TODO enable failing splits after they have been stablized
+env:
+  - TC_PATHS="tests/kafkatest/tests/client1"
+  - TC_PATHS="tests/kafkatest/tests/client2"
+# - TC_PATHS="tests/kafkatest/tests/connect tests/kafkatest/tests/streams tests/kafkatest/tests/tools"
+# - TC_PATHS="tests/kafkatest/tests/mirror_maker"
+# - TC_PATHS="tests/kafkatest/tests/replication"
+# - TC_PATHS="tests/kafkatest/tests/upgrade"
+  - TC_PATHS="tests/kafkatest/tests/security1"
+# - TC_PATHS="tests/kafkatest/tests/security2"
+# - TC_PATHS="tests/kafkatest/tests/core1"
+  - TC_PATHS="tests/kafkatest/tests/core2"
+
+jdk:
+  - oraclejdk8
+
+before_install:
+
+script:
+  - ./gradlew releaseTarGz && /bin/bash ./tests/travis/run_tests.sh
+
+services:
+  - docker
+
+before_cache:
+  - rm -f  $HOME/.gradle/caches/modules-2/modules-2.lock
+  - rm -fr $HOME/.gradle/caches/*/plugin-resolution/
+cache:
+  directories:
+    - "$HOME/.m2/repository"
+    - "$HOME/.gradle/caches/"
+    - "$HOME/.gradle/wrapper/"

gradle/wrapper/gradle-wrapper.properties

@@ -0,0 +1,6 @@
+#Fri Oct 07 16:09:33 PDT 2016
+distributionBase=GRADLE_USER_HOME
+distributionPath=wrapper/dists
+zipStoreBase=GRADLE_USER_HOME
+zipStorePath=wrapper/dists
+distributionUrl=https\://services.gradle.org/distributions/gradle-3.0-bin.zip

tests/README.md

@@ -6,6 +6,43 @@ This directory contains Kafka system integration and performance tests.
 (ducktape is a distributed testing framework which provides test runner,
 result reporter and utilities to pull up and tear down services.)
 
+Running tests using docker
+--------------------------
+Docker is used for running kafka system tests on travis-ci. And exactly same setup can be run for development purposes.
+
+* Run all tests
+```
+bash tests/travis/run_tests.sh
+```
+* Run all tests with debug on (warning will produce log of logs)
+```
+_DUCKTAPE_OPTIONS="--debug" bash tests/travis/run_tests.sh | tee debug_logs.txt
+```
+* Run a subset of tests
+```
+TC_PATHS="tests/kafkatest/tests/streams tests/kafkatest/tests/tools" bash tests/travis/run_tests.sh
+```
+
+Examining CI run
+----------------
+* Set BUILD_ID is travis ci's build id. E.g. build id is 169519874 for the following build
+```
+https://travis-ci.org/raghavgautam/kafka/builds/169519874
+```
+
+* Getting number of tests that were actually run
+```
+for id in $(curl -sSL https://api.travis-ci.org/builds/$BUILD_ID | jq '.matrix|map(.id)|.[]'); do curl -sSL "https://api.travis-ci.org/jobs/$id/log.txt?deansi=true" ; done | egrep 'SerialTestRunner.*setting up' | wc
+```
+* Getting number of tests that passed
+```
+for id in $(curl -sSL https://api.travis-ci.org/builds/$BUILD_ID | jq '.matrix|map(.id)|.[]'); do curl -sSL "https://api.travis-ci.org/jobs/$id/log.txt?deansi=true" ; done | egrep 'SerialTestRunner.*PASS' | wc
+```
+* Getting all the logs produced from a run
+```
+for id in $(curl -sSL https://api.travis-ci.org/builds/169519874 | jq '.matrix|map(.id)|.[]'); do curl -sSL "https://api.travis-ci.org/jobs/$id/log.txt?deansi=true" ; done
+```
+
 Local Quickstart
 ----------------
 This quickstart will help you run the Kafka system tests on your local machine. Note this requires bringing up a cluster of virtual machines on your local computer, which is memory intensive; it currently requires around 10G RAM.

tests/cluster_file.json

@@ -0,0 +1,97 @@
+{
+  "_comment": [
+    "Licensed to the Apache Software Foundation (ASF) under one or more",
+    "contributor license agreements.  See the NOTICE file distributed with",
+    "this work for additional information regarding copyright ownership.",
+    "The ASF licenses this file to You under the Apache License, Version 2.0",
+    "(the \"License\"); you may not use this file except in compliance with",
+    "the License.  You may obtain a copy of the License at",
+    "",
+    "http://www.apache.org/licenses/LICENSE-2.0",
+    "",
+    "Unless required by applicable law or agreed to in writing, software",
+    "distributed under the License is distributed on an \"AS IS\" BASIS,",
+    "WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.",
+    "See the License for the specific language governing permissions and",
+    "limitations under the License."
+  ],
+  "nodes": [
+    {
+      "hostname": "knode02.knw",
+      "user": "root",
+      "ssh_args": "",
+      "ssh_hostname": "",
+      "externally_routable_ip": "knode02.knw"
+    },
+    {
+      "hostname": "knode03.knw",
+      "user": "root",
+      "ssh_args": "",
+      "ssh_hostname": "",
+      "externally_routable_ip": "knode03.knw"
+    },
+    {
+      "hostname": "knode04.knw",
+      "user": "root",
+      "ssh_args": "",
+      "ssh_hostname": "",
+      "externally_routable_ip": "knode04.knw"
+    },
+    {
+      "hostname": "knode05.knw",
+      "user": "root",
+      "ssh_args": "",
+      "ssh_hostname": "",
+      "externally_routable_ip": "knode05.knw"
+    },
+    {
+      "hostname": "knode06.knw",
+      "user": "root",
+      "ssh_args": "",
+      "ssh_hostname": "",
+      "externally_routable_ip": "knode06.knw"
+    },
+    {
+      "hostname": "knode07.knw",
+      "user": "root",
+      "ssh_args": "",
+      "ssh_hostname": "",
+      "externally_routable_ip": "knode07.knw"
+    },
+    {
+      "hostname": "knode08.knw",
+      "user": "root",
+      "ssh_args": "",
+      "ssh_hostname": "",
+      "externally_routable_ip": "knode08.knw"
+    },
+    {
+      "hostname": "knode09.knw",
+      "user": "root",
+      "ssh_args": "",
+      "ssh_hostname": "",
+      "externally_routable_ip": "knode09.knw"
+    },
+    {
+      "hostname": "knode10.knw",
+      "user": "root",
+      "ssh_args": "",
+      "ssh_hostname": "",
+      "externally_routable_ip": "knode10.knw"
+    },
+    {
+      "hostname": "knode11.knw",
+      "user": "root",
+      "ssh_args": "",
+      "ssh_hostname": "",
+      "externally_routable_ip": "knode11.knw"
+    },
+    {
+      "hostname": "knode12.knw",
+      "user": "root",
+      "ssh_args": "",
+      "ssh_hostname": "",
+      "externally_routable_ip": "knode12.knw"
+    }
+  ]
+}

tests/kafkatest/directory_layout/kafka_path.py

@@ -44,11 +44,11 @@ TOOLS_DEPENDANT_TEST_LIBS_JAR_NAME = "tools-dependant-libs"
 
 JARS = {
     "trunk": {
-        CORE_JAR_NAME: "core/build/*/*.jar",
+        CORE_JAR_NAME: "libs/*.jar",
-        CORE_LIBS_JAR_NAME: "core/build/libs/*.jar",
+        CORE_LIBS_JAR_NAME: "libs/*.jar",
-        CORE_DEPENDANT_TEST_LIBS_JAR_NAME: "core/build/dependant-testlibs/*.jar",
+        CORE_DEPENDANT_TEST_LIBS_JAR_NAME: "libs/*.jar",
-        TOOLS_JAR_NAME: "tools/build/libs/kafka-tools*.jar",
+        TOOLS_JAR_NAME: "libs/*.jar",
-        TOOLS_DEPENDANT_TEST_LIBS_JAR_NAME: "tools/build/dependant-libs*/*.jar"
+        TOOLS_DEPENDANT_TEST_LIBS_JAR_NAME: "libs/*.jar"
     }
 }
 

tests/kafkatest/services/mirror_maker.py

@@ -180,13 +180,13 @@ class MirrorMaker(KafkaPathResolverMixin, Service):
         cmd = self.start_cmd(node)
         self.logger.debug("Mirror maker command: %s", cmd)
         node.account.ssh(cmd, allow_fail=False)
-        wait_until(lambda: self.alive(node), timeout_sec=10, backoff_sec=.5,
+        wait_until(lambda: self.alive(node), timeout_sec=30, backoff_sec=.5,
                    err_msg="Mirror maker took to long to start.")
         self.logger.debug("Mirror maker is alive")
 
     def stop_node(self, node, clean_shutdown=True):
         node.account.kill_process("java", allow_fail=True, clean_shutdown=clean_shutdown)
-        wait_until(lambda: not self.alive(node), timeout_sec=10, backoff_sec=.5,
+        wait_until(lambda: not self.alive(node), timeout_sec=30, backoff_sec=.5,
                    err_msg="Mirror maker took to long to stop.")
 
     def clean_node(self, node):

tests/kafkatest/tests/produce_consume_validate.py

@@ -12,6 +12,7 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
+import traceback
 
 from ducktape.tests.test import Test
 from ducktape.utils.util import wait_until
@@ -102,7 +103,7 @@ class ProduceConsumeValidateTest(Test):
         except BaseException as e:
             for s in self.test_context.services:
                 self.mark_for_collect(s)
-            raise
+            raise Exception(traceback.format_exc(e))
 
     @staticmethod
     def annotate_missing_msgs(missing, acked, consumed, msg):

tests/kafkatest/tests/upgrade/upgrade_test.py

@@ -111,7 +111,7 @@ class TestUpgrade(ProduceConsumeValidateTest):
 
         # TODO - reduce the timeout
         self.consumer = ConsoleConsumer(self.test_context, self.num_consumers, self.kafka,
-                                        self.topic, consumer_timeout_ms=30000, new_consumer=new_consumer,
+                                        self.topic, consumer_timeout_ms=200000, new_consumer=new_consumer,
                                         message_validator=is_int, version=KafkaVersion(from_kafka_version))
 
         self.run_produce_consume_validate(core_test_action=lambda: self.perform_upgrade(from_kafka_version,

tests/travis/Dockerfile

@@ -0,0 +1,38 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+FROM openjdk:8
+
+MAINTAINER Raghav Kumar Gautam
+# commands to update docker image:
+#   - docker build . -t raghavgautam/kfk-image
+#   - docker push raghavgautam/kfk-image
+RUN apt update
+RUN apt install -y unzip wget curl jq coreutils openssh-server net-tools vim openjdk-8-jdk python-pip
+RUN pip install ducktape
+
+VOLUME ["/kafka"]
+VOLUME ["/kfk_src"]
+
+ENV MIRROR="http://apache.cs.utah.edu/"
+RUN wget -q "${MIRROR}kafka/0.8.2.2/kafka_2.10-0.8.2.2.tgz" -O "/tmp/kafka_2.10-0.8.2.2.tgz" && tar xfz /tmp/kafka_2.10-0.8.2.2.tgz -C /opt && mv "/opt/kafka_2.10-0.8.2.2" "/opt/kafka-0.8.2.2"
+RUN wget -q "${MIRROR}kafka/0.9.0.1/kafka_2.10-0.9.0.1.tgz" -O "/tmp/kafka_2.10-0.9.0.1.tgz" && tar xfz /tmp/kafka_2.10-0.9.0.1.tgz -C /opt && mv "/opt/kafka_2.10-0.9.0.1" "/opt/kafka-0.9.0.1"
+RUN wget -q "${MIRROR}kafka/0.10.0.1/kafka_2.10-0.10.0.1.tgz" -O "/tmp/kafka_2.10-0.10.0.1.tgz" && tar xfz /tmp/kafka_2.10-0.10.0.1.tgz -C /opt && mv "/opt/kafka_2.10-0.10.0.1" "/opt/kafka-0.10.0.1"
+
+RUN rm /tmp/kafka_*.tgz
+ADD ssh /root/.ssh
+RUN chmod 600 /root/.ssh/id_rsa
+
+CMD service ssh start && tail -f /dev/null

tests/travis/run_tests.sh

@@ -0,0 +1,58 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# To run tests use a command like:
+#   TC_PATHS="tests/kafkatest/tests/streams tests/kafkatest/tests/tools" bash tests/travis/run_tests.sh
+set -x
+
+SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
+TESTS_DIR=`dirname ${SCRIPT_DIR}`
+KFK_SRC=`dirname ${TESTS_DIR}`
+
+
+cd ${SCRIPT_DIR}
+chmod 600 ssh/id_rsa
+
+docker network rm knw
+docker network create knw
+
+docker kill $(docker ps -f=network=knw -q)
+docker rm $(docker ps -a -f=network=knw -q)
+
+for i in $(seq -w 1 12); do
+  docker run -d -t --name knode${i} --network knw -v ${KFK_SRC}:/kfk_src raghavgautam/kfk-image
+done
+
+docker info
+docker ps
+docker network inspect knw
+
+for i in $(seq -w 1 12); do
+  echo knode${i}
+  docker exec knode${i} bash -c "(tar xfz /kfk_src/core/build/distributions/kafka_*SNAPSHOT.tgz -C /opt || echo missing kafka tgz did you build kafka tarball) && mv /opt/kafka*SNAPSHOT /opt/kafka-trunk && ls -l /opt"
+  docker exec knode01 bash -c "ssh knode$i hostname"
+done
+
+# hack to copy test dependencies
+# this is required for running MiniKDC
+(cd ${KFK_SRC} && ./gradlew copyDependantTestLibs)
+for i in $(seq -w 1 12); do
+  echo knode${i}
+  docker exec knode${i} bash -c "cp /kfk_src/core/build/dependant-testlibs/* /opt/kafka-trunk/libs/"
+  docker exec knode01 bash -c "ssh knode$i hostname"
+done
+
+docker exec knode01 bash -c "cd /kfk_src; ducktape ${_DUCKTAPE_OPTIONS} --cluster-file tests/cluster_file.json ${TC_PATHS:-tests/kafkatest/tests}"

tests/travis/ssh/authorized_keys

@@ -0,0 +1,15 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0qDT9kEPWc8JQ53b4KnT/ZJOLwb+3c//jpLW/2ofjDyIsPW4FohLpicfouch/zsRpN4G38lua+2BsGls9sMIZc6PXY2L+NIGCkqEMdCoU1Ym8SMtyJklfzp3m/0PeK9s2dLlR3PFRYvyFA4btQK5hkbYDNZPzf4airvzdRzLkrFf81+RemaMI2EtONwJRcbLViPaTXVKJdbFwJTJ1u7yu9wDYWHKBMA92mHTQeP6bhVYCqxJn3to/RfZYd+sHw6mfxVg5OrAlUOYpSV4pDNCAsIHdtZ56V8NQlJL6NJ2vzzSSYUwLMqe88fhrC8yYHoxC07QPy1EdkSTHdohAicyT root@knode01.knw

tests/travis/ssh/config

@@ -0,0 +1,21 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+Host *
+  ControlMaster auto
+  ControlPath ~/.ssh/master-%r@%h:%p
+  StrictHostKeyChecking no
+  ConnectTimeout=10
+  IdentityFile ~/.ssh/id_rsa

tests/travis/ssh/id_rsa

@@ -0,0 +1,27 @@
+-----BEGIN RSA PRIVATE KEY-----
+MIIEpQIBAAKCAQEAtKg0/ZBD1nPCUOd2+Cp0/2STi8G/t3P/46S1v9qH4w8iLD1u
+BaIS6YnH6LnIf87EaTeBt/JbmvtgbBpbPbDCGXOj12Ni/jSBgpKhDHQqFNWJvEjL
+ciZJX86d5v9D3ivbNnS5UdzxUWL8hQOG7UCuYZG2AzWT83+Goq783Ucy5KxX/Nfk
+XpmjCNhLTjcCUXGy1Yj2k11SiXWxcCUydbu8rvcA2FhygTAPdph00Hj+m4VWAqsS
+Z97aP0X2WHfrB8Opn8VYOTqwJVDmKUleKQzQgLCB3bWeelfDUJSS+jSdr880kmFM
+CzKnvPH4awvMmB6MQtO0D8tRHZEkx3aIQInMkwIDAQABAoIBAQCz6EMFNNLp0NP1
+X9yRXS6wW4e4CRWUazesiw3YZpcmnp6IchCMGZA99FEZyVILPW1J3tYWyotBdw7Z
++RFeCRXy5L+IMtiVkNJcpwss7M4ve0w0LkY0gj5V49xJ+3Gp4gDnZSxcguvrAem5
+yP5obR572fDpl0SknB4HCr6U2l+rauzrLyevy5eeDT/vmXbuM1cdHpNIXmmElz4L
+t31n+exQRn6tP1h516iXbcYbopxDgdv2qKGAqzWKE6TyWpzF5x7kjOEYt0bZ5QO3
+Lwh7AAqE/3mwxlYwng1L4WAT7RtcP19W+9JDIc7ENInMGxq6q46p1S3IPZsf1cj/
+aAJ9q3LBAoGBAOVJr0+WkR786n3BuswpGQWBgVxfai4y9Lf90vuGKawdQUzXv0/c
+EB/CFqP/dIsquukA8PfzjNMyTNmEHXi4Sf16H8Rg4EGhIYMEqIQojx1t/yLLm0aU
+YPEvW/02Umtlg3pJw9fQAAzFVqCasw2E2lUdAUkydGRwDUJZmv2/b3NzAoGBAMm0
+Jo7Et7ochH8Vku6uA+hG+RdwlKFm5JA7/Ci3DOdQ1zmJNrvBBFQLo7AjA4iSCoBd
+s9+y0nrSPcF4pM3l6ghLheaqbnIi2HqIMH9mjDbrOZiWvbnjvjpOketgNX8vV3Ye
+GUkSjoNcmvRmdsICmUjeML8bGOmq4zF9W/GIfTphAoGBAKGRo8R8f/SLGh3VtvCI
+gUY89NAHuEWnyIQii1qMNq8+yjYAzaHTm1UVqmiT6SbrzFvGOwcuCu0Dw91+2Fmp
+2xGPzfTOoxf8GCY/0ROXlQmS6jc1rEw24Hzz92ldrwRYuyYf9q4Ltw1IvXtcp5F+
+LW/OiYpv0E66Gs3HYI0wKbP7AoGBAJMZWeFW37LQJ2TTJAQDToAwemq4xPxsoJX7
+2SsMTFHKKBwi0JLe8jwk/OxwrJwF/bieHZcvv8ao2zbkuDQcz6/a/D074C5G8V9z
+QQM4k1td8vQwQw91Yv782/gvgvRNX1iaHNCowtxURgGlVEirQoTc3eoRZfrLkMM/
+7DTa2JEhAoGACEu3zHJ1sgyeOEgLArUJXlQM30A/ulMrnCd4MEyIE+ReyWAUevUQ
+0lYdVNva0/W4C5e2lUOJL41jjIPLqI7tcFR2PZE6n0xTTkxNH5W2u1WpFeKjx+O3
+czv7Bt6wYyLHIMy1JEqAQ7pw1mtJ5s76UDvXUhciF+DU2pWYc6APKR0=
+-----END RSA PRIVATE KEY-----

tests/travis/ssh/id_rsa.pub

@@ -0,0 +1 @@
+ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0qDT9kEPWc8JQ53b4KnT/ZJOLwb+3c//jpLW/2ofjDyIsPW4FohLpicfouch/zsRpN4G38lua+2BsGls9sMIZc6PXY2L+NIGCkqEMdCoU1Ym8SMtyJklfzp3m/0PeK9s2dLlR3PFRYvyFA4btQK5hkbYDNZPzf4airvzdRzLkrFf81+RemaMI2EtONwJRcbLViPaTXVKJdbFwJTJ1u7yu9wDYWHKBMA92mHTQeP6bhVYCqxJn3to/RfZYd+sHw6mfxVg5OrAlUOYpSV4pDNCAsIHdtZ56V8NQlJL6NJ2vzzSSYUwLMqe88fhrC8yYHoxC07QPy1EdkSTHdohAicyT root@knode01.knw