Revert "KAFKA-4345; Run decktape test for each pull request"

This reverts commit e035fc0395 for the
following reasons:

1. License files are missing causing local builds to fail during the
rat task (rat is not being run in Jenkins for some reason, filed
KAFKA-4459 for that)
2. It renames a number of system test files when there's a better
way to achieve the goal of running a subset of system tests to stay
under the Travis limit.
3. It adds the gradle wrapper binary even though this was removed
intentionally a while back.

A new PR will be submitted for KAFKA-4345 without the undesired
changes.

Author: Ismael Juma <ismael@juma.me.uk>

Reviewers: Jason Gustafson <jason@confluent.io>, Ewen Cheslack-Postava <ewen@confluent.io>

Closes #2187 from ijuma/kafka-4345-revert
This commit is contained in:
Ismael Juma 2016-11-29 09:11:21 -08:00 committed by Ewen Cheslack-Postava
parent 3e3b7a010b
commit a5d28149fb
40 changed files with 9 additions and 358 deletions

View File

@ -1,48 +0,0 @@
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
sudo: required
dist: trusty
language: java
# TODO enable failing splits after they have been stablized
env:
- TC_PATHS="tests/kafkatest/tests/client1"
- TC_PATHS="tests/kafkatest/tests/client2"
# - TC_PATHS="tests/kafkatest/tests/connect tests/kafkatest/tests/streams tests/kafkatest/tests/tools"
# - TC_PATHS="tests/kafkatest/tests/mirror_maker"
# - TC_PATHS="tests/kafkatest/tests/replication"
# - TC_PATHS="tests/kafkatest/tests/upgrade"
- TC_PATHS="tests/kafkatest/tests/security1"
# - TC_PATHS="tests/kafkatest/tests/security2"
# - TC_PATHS="tests/kafkatest/tests/core1"
- TC_PATHS="tests/kafkatest/tests/core2"
jdk:
- oraclejdk8
before_install:
script:
- ./gradlew releaseTarGz && /bin/bash ./tests/travis/run_tests.sh
services:
- docker
before_cache:
- rm -f $HOME/.gradle/caches/modules-2/modules-2.lock
- rm -fr $HOME/.gradle/caches/*/plugin-resolution/
cache:
directories:
- "$HOME/.m2/repository"
- "$HOME/.gradle/caches/"
- "$HOME/.gradle/wrapper/"

Binary file not shown.

View File

@ -1,6 +0,0 @@
#Fri Oct 07 16:09:33 PDT 2016
distributionBase=GRADLE_USER_HOME
distributionPath=wrapper/dists
zipStoreBase=GRADLE_USER_HOME
zipStorePath=wrapper/dists
distributionUrl=https\://services.gradle.org/distributions/gradle-3.0-bin.zip

View File

@ -6,43 +6,6 @@ This directory contains Kafka system integration and performance tests.
(ducktape is a distributed testing framework which provides test runner, (ducktape is a distributed testing framework which provides test runner,
result reporter and utilities to pull up and tear down services.) result reporter and utilities to pull up and tear down services.)
Running tests using docker
--------------------------
Docker is used for running kafka system tests on travis-ci. And exactly same setup can be run for development purposes.
* Run all tests
```
bash tests/travis/run_tests.sh
```
* Run all tests with debug on (warning will produce log of logs)
```
_DUCKTAPE_OPTIONS="--debug" bash tests/travis/run_tests.sh | tee debug_logs.txt
```
* Run a subset of tests
```
TC_PATHS="tests/kafkatest/tests/streams tests/kafkatest/tests/tools" bash tests/travis/run_tests.sh
```
Examining CI run
----------------
* Set BUILD_ID is travis ci's build id. E.g. build id is 169519874 for the following build
```
https://travis-ci.org/raghavgautam/kafka/builds/169519874
```
* Getting number of tests that were actually run
```
for id in $(curl -sSL https://api.travis-ci.org/builds/$BUILD_ID | jq '.matrix|map(.id)|.[]'); do curl -sSL "https://api.travis-ci.org/jobs/$id/log.txt?deansi=true" ; done | egrep 'SerialTestRunner.*setting up' | wc
```
* Getting number of tests that passed
```
for id in $(curl -sSL https://api.travis-ci.org/builds/$BUILD_ID | jq '.matrix|map(.id)|.[]'); do curl -sSL "https://api.travis-ci.org/jobs/$id/log.txt?deansi=true" ; done | egrep 'SerialTestRunner.*PASS' | wc
```
* Getting all the logs produced from a run
```
for id in $(curl -sSL https://api.travis-ci.org/builds/169519874 | jq '.matrix|map(.id)|.[]'); do curl -sSL "https://api.travis-ci.org/jobs/$id/log.txt?deansi=true" ; done
```
Local Quickstart Local Quickstart
---------------- ----------------
This quickstart will help you run the Kafka system tests on your local machine. Note this requires bringing up a cluster of virtual machines on your local computer, which is memory intensive; it currently requires around 10G RAM. This quickstart will help you run the Kafka system tests on your local machine. Note this requires bringing up a cluster of virtual machines on your local computer, which is memory intensive; it currently requires around 10G RAM.

View File

@ -1,97 +0,0 @@
{
"_comment": [
"Licensed to the Apache Software Foundation (ASF) under one or more",
"contributor license agreements. See the NOTICE file distributed with",
"this work for additional information regarding copyright ownership.",
"The ASF licenses this file to You under the Apache License, Version 2.0",
"(the \"License\"); you may not use this file except in compliance with",
"the License. You may obtain a copy of the License at",
"",
"http://www.apache.org/licenses/LICENSE-2.0",
"",
"Unless required by applicable law or agreed to in writing, software",
"distributed under the License is distributed on an \"AS IS\" BASIS,",
"WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.",
"See the License for the specific language governing permissions and",
"limitations under the License."
],
"nodes": [
{
"hostname": "knode02.knw",
"user": "root",
"ssh_args": "",
"ssh_hostname": "",
"externally_routable_ip": "knode02.knw"
},
{
"hostname": "knode03.knw",
"user": "root",
"ssh_args": "",
"ssh_hostname": "",
"externally_routable_ip": "knode03.knw"
},
{
"hostname": "knode04.knw",
"user": "root",
"ssh_args": "",
"ssh_hostname": "",
"externally_routable_ip": "knode04.knw"
},
{
"hostname": "knode05.knw",
"user": "root",
"ssh_args": "",
"ssh_hostname": "",
"externally_routable_ip": "knode05.knw"
},
{
"hostname": "knode06.knw",
"user": "root",
"ssh_args": "",
"ssh_hostname": "",
"externally_routable_ip": "knode06.knw"
},
{
"hostname": "knode07.knw",
"user": "root",
"ssh_args": "",
"ssh_hostname": "",
"externally_routable_ip": "knode07.knw"
},
{
"hostname": "knode08.knw",
"user": "root",
"ssh_args": "",
"ssh_hostname": "",
"externally_routable_ip": "knode08.knw"
},
{
"hostname": "knode09.knw",
"user": "root",
"ssh_args": "",
"ssh_hostname": "",
"externally_routable_ip": "knode09.knw"
},
{
"hostname": "knode10.knw",
"user": "root",
"ssh_args": "",
"ssh_hostname": "",
"externally_routable_ip": "knode10.knw"
},
{
"hostname": "knode11.knw",
"user": "root",
"ssh_args": "",
"ssh_hostname": "",
"externally_routable_ip": "knode11.knw"
},
{
"hostname": "knode12.knw",
"user": "root",
"ssh_args": "",
"ssh_hostname": "",
"externally_routable_ip": "knode12.knw"
}
]
}

View File

@ -44,11 +44,11 @@ TOOLS_DEPENDANT_TEST_LIBS_JAR_NAME = "tools-dependant-libs"
JARS = { JARS = {
"trunk": { "trunk": {
CORE_JAR_NAME: "libs/*.jar", CORE_JAR_NAME: "core/build/*/*.jar",
CORE_LIBS_JAR_NAME: "libs/*.jar", CORE_LIBS_JAR_NAME: "core/build/libs/*.jar",
CORE_DEPENDANT_TEST_LIBS_JAR_NAME: "libs/*.jar", CORE_DEPENDANT_TEST_LIBS_JAR_NAME: "core/build/dependant-testlibs/*.jar",
TOOLS_JAR_NAME: "libs/*.jar", TOOLS_JAR_NAME: "tools/build/libs/kafka-tools*.jar",
TOOLS_DEPENDANT_TEST_LIBS_JAR_NAME: "libs/*.jar" TOOLS_DEPENDANT_TEST_LIBS_JAR_NAME: "tools/build/dependant-libs*/*.jar"
} }
} }

View File

@ -180,13 +180,13 @@ class MirrorMaker(KafkaPathResolverMixin, Service):
cmd = self.start_cmd(node) cmd = self.start_cmd(node)
self.logger.debug("Mirror maker command: %s", cmd) self.logger.debug("Mirror maker command: %s", cmd)
node.account.ssh(cmd, allow_fail=False) node.account.ssh(cmd, allow_fail=False)
wait_until(lambda: self.alive(node), timeout_sec=30, backoff_sec=.5, wait_until(lambda: self.alive(node), timeout_sec=10, backoff_sec=.5,
err_msg="Mirror maker took to long to start.") err_msg="Mirror maker took to long to start.")
self.logger.debug("Mirror maker is alive") self.logger.debug("Mirror maker is alive")
def stop_node(self, node, clean_shutdown=True): def stop_node(self, node, clean_shutdown=True):
node.account.kill_process("java", allow_fail=True, clean_shutdown=clean_shutdown) node.account.kill_process("java", allow_fail=True, clean_shutdown=clean_shutdown)
wait_until(lambda: not self.alive(node), timeout_sec=30, backoff_sec=.5, wait_until(lambda: not self.alive(node), timeout_sec=10, backoff_sec=.5,
err_msg="Mirror maker took to long to stop.") err_msg="Mirror maker took to long to stop.")
def clean_node(self, node): def clean_node(self, node):

View File

@ -111,7 +111,7 @@ class TestUpgrade(ProduceConsumeValidateTest):
# TODO - reduce the timeout # TODO - reduce the timeout
self.consumer = ConsoleConsumer(self.test_context, self.num_consumers, self.kafka, self.consumer = ConsoleConsumer(self.test_context, self.num_consumers, self.kafka,
self.topic, consumer_timeout_ms=200000, new_consumer=new_consumer, self.topic, consumer_timeout_ms=30000, new_consumer=new_consumer,
message_validator=is_int, version=KafkaVersion(from_kafka_version)) message_validator=is_int, version=KafkaVersion(from_kafka_version))
self.run_produce_consume_validate(core_test_action=lambda: self.perform_upgrade(from_kafka_version, self.run_produce_consume_validate(core_test_action=lambda: self.perform_upgrade(from_kafka_version,

View File

@ -12,7 +12,6 @@
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and # See the License for the specific language governing permissions and
# limitations under the License. # limitations under the License.
import traceback
from ducktape.tests.test import Test from ducktape.tests.test import Test
from ducktape.utils.util import wait_until from ducktape.utils.util import wait_until
@ -103,7 +102,7 @@ class ProduceConsumeValidateTest(Test):
except BaseException as e: except BaseException as e:
for s in self.test_context.services: for s in self.test_context.services:
self.mark_for_collect(s) self.mark_for_collect(s)
raise Exception(traceback.format_exc(e)) raise
@staticmethod @staticmethod
def annotate_missing_msgs(missing, acked, consumed, msg): def annotate_missing_msgs(missing, acked, consumed, msg):

View File

@ -1,38 +0,0 @@
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
FROM openjdk:8
MAINTAINER Raghav Kumar Gautam
# commands to update docker image:
# - docker build . -t raghavgautam/kfk-image
# - docker push raghavgautam/kfk-image
RUN apt update
RUN apt install -y unzip wget curl jq coreutils openssh-server net-tools vim openjdk-8-jdk python-pip
RUN pip install ducktape
VOLUME ["/kafka"]
VOLUME ["/kfk_src"]
ENV MIRROR="http://apache.cs.utah.edu/"
RUN wget -q "${MIRROR}kafka/0.8.2.2/kafka_2.10-0.8.2.2.tgz" -O "/tmp/kafka_2.10-0.8.2.2.tgz" && tar xfz /tmp/kafka_2.10-0.8.2.2.tgz -C /opt && mv "/opt/kafka_2.10-0.8.2.2" "/opt/kafka-0.8.2.2"
RUN wget -q "${MIRROR}kafka/0.9.0.1/kafka_2.10-0.9.0.1.tgz" -O "/tmp/kafka_2.10-0.9.0.1.tgz" && tar xfz /tmp/kafka_2.10-0.9.0.1.tgz -C /opt && mv "/opt/kafka_2.10-0.9.0.1" "/opt/kafka-0.9.0.1"
RUN wget -q "${MIRROR}kafka/0.10.0.1/kafka_2.10-0.10.0.1.tgz" -O "/tmp/kafka_2.10-0.10.0.1.tgz" && tar xfz /tmp/kafka_2.10-0.10.0.1.tgz -C /opt && mv "/opt/kafka_2.10-0.10.0.1" "/opt/kafka-0.10.0.1"
RUN rm /tmp/kafka_*.tgz
ADD ssh /root/.ssh
RUN chmod 600 /root/.ssh/id_rsa
CMD service ssh start && tail -f /dev/null

View File

@ -1,58 +0,0 @@
#!/usr/bin/env bash
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# To run tests use a command like:
# TC_PATHS="tests/kafkatest/tests/streams tests/kafkatest/tests/tools" bash tests/travis/run_tests.sh
set -x
SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
TESTS_DIR=`dirname ${SCRIPT_DIR}`
KFK_SRC=`dirname ${TESTS_DIR}`
cd ${SCRIPT_DIR}
chmod 600 ssh/id_rsa
docker network rm knw
docker network create knw
docker kill $(docker ps -f=network=knw -q)
docker rm $(docker ps -a -f=network=knw -q)
for i in $(seq -w 1 12); do
docker run -d -t --name knode${i} --network knw -v ${KFK_SRC}:/kfk_src raghavgautam/kfk-image
done
docker info
docker ps
docker network inspect knw
for i in $(seq -w 1 12); do
echo knode${i}
docker exec knode${i} bash -c "(tar xfz /kfk_src/core/build/distributions/kafka_*SNAPSHOT.tgz -C /opt || echo missing kafka tgz did you build kafka tarball) && mv /opt/kafka*SNAPSHOT /opt/kafka-trunk && ls -l /opt"
docker exec knode01 bash -c "ssh knode$i hostname"
done
# hack to copy test dependencies
# this is required for running MiniKDC
(cd ${KFK_SRC} && ./gradlew copyDependantTestLibs)
for i in $(seq -w 1 12); do
echo knode${i}
docker exec knode${i} bash -c "cp /kfk_src/core/build/dependant-testlibs/* /opt/kafka-trunk/libs/"
docker exec knode01 bash -c "ssh knode$i hostname"
done
docker exec knode01 bash -c "cd /kfk_src; ducktape ${_DUCKTAPE_OPTIONS} --cluster-file tests/cluster_file.json ${TC_PATHS:-tests/kafkatest/tests}"

View File

@ -1,15 +0,0 @@
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0qDT9kEPWc8JQ53b4KnT/ZJOLwb+3c//jpLW/2ofjDyIsPW4FohLpicfouch/zsRpN4G38lua+2BsGls9sMIZc6PXY2L+NIGCkqEMdCoU1Ym8SMtyJklfzp3m/0PeK9s2dLlR3PFRYvyFA4btQK5hkbYDNZPzf4airvzdRzLkrFf81+RemaMI2EtONwJRcbLViPaTXVKJdbFwJTJ1u7yu9wDYWHKBMA92mHTQeP6bhVYCqxJn3to/RfZYd+sHw6mfxVg5OrAlUOYpSV4pDNCAsIHdtZ56V8NQlJL6NJ2vzzSSYUwLMqe88fhrC8yYHoxC07QPy1EdkSTHdohAicyT root@knode01.knw

View File

@ -1,21 +0,0 @@
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
Host *
ControlMaster auto
ControlPath ~/.ssh/master-%r@%h:%p
StrictHostKeyChecking no
ConnectTimeout=10
IdentityFile ~/.ssh/id_rsa

View File

@ -1,27 +0,0 @@
-----BEGIN RSA PRIVATE KEY-----
MIIEpQIBAAKCAQEAtKg0/ZBD1nPCUOd2+Cp0/2STi8G/t3P/46S1v9qH4w8iLD1u
BaIS6YnH6LnIf87EaTeBt/JbmvtgbBpbPbDCGXOj12Ni/jSBgpKhDHQqFNWJvEjL
ciZJX86d5v9D3ivbNnS5UdzxUWL8hQOG7UCuYZG2AzWT83+Goq783Ucy5KxX/Nfk
XpmjCNhLTjcCUXGy1Yj2k11SiXWxcCUydbu8rvcA2FhygTAPdph00Hj+m4VWAqsS
Z97aP0X2WHfrB8Opn8VYOTqwJVDmKUleKQzQgLCB3bWeelfDUJSS+jSdr880kmFM
CzKnvPH4awvMmB6MQtO0D8tRHZEkx3aIQInMkwIDAQABAoIBAQCz6EMFNNLp0NP1
X9yRXS6wW4e4CRWUazesiw3YZpcmnp6IchCMGZA99FEZyVILPW1J3tYWyotBdw7Z
+RFeCRXy5L+IMtiVkNJcpwss7M4ve0w0LkY0gj5V49xJ+3Gp4gDnZSxcguvrAem5
yP5obR572fDpl0SknB4HCr6U2l+rauzrLyevy5eeDT/vmXbuM1cdHpNIXmmElz4L
t31n+exQRn6tP1h516iXbcYbopxDgdv2qKGAqzWKE6TyWpzF5x7kjOEYt0bZ5QO3
Lwh7AAqE/3mwxlYwng1L4WAT7RtcP19W+9JDIc7ENInMGxq6q46p1S3IPZsf1cj/
aAJ9q3LBAoGBAOVJr0+WkR786n3BuswpGQWBgVxfai4y9Lf90vuGKawdQUzXv0/c
EB/CFqP/dIsquukA8PfzjNMyTNmEHXi4Sf16H8Rg4EGhIYMEqIQojx1t/yLLm0aU
YPEvW/02Umtlg3pJw9fQAAzFVqCasw2E2lUdAUkydGRwDUJZmv2/b3NzAoGBAMm0
Jo7Et7ochH8Vku6uA+hG+RdwlKFm5JA7/Ci3DOdQ1zmJNrvBBFQLo7AjA4iSCoBd
s9+y0nrSPcF4pM3l6ghLheaqbnIi2HqIMH9mjDbrOZiWvbnjvjpOketgNX8vV3Ye
GUkSjoNcmvRmdsICmUjeML8bGOmq4zF9W/GIfTphAoGBAKGRo8R8f/SLGh3VtvCI
gUY89NAHuEWnyIQii1qMNq8+yjYAzaHTm1UVqmiT6SbrzFvGOwcuCu0Dw91+2Fmp
2xGPzfTOoxf8GCY/0ROXlQmS6jc1rEw24Hzz92ldrwRYuyYf9q4Ltw1IvXtcp5F+
LW/OiYpv0E66Gs3HYI0wKbP7AoGBAJMZWeFW37LQJ2TTJAQDToAwemq4xPxsoJX7
2SsMTFHKKBwi0JLe8jwk/OxwrJwF/bieHZcvv8ao2zbkuDQcz6/a/D074C5G8V9z
QQM4k1td8vQwQw91Yv782/gvgvRNX1iaHNCowtxURgGlVEirQoTc3eoRZfrLkMM/
7DTa2JEhAoGACEu3zHJ1sgyeOEgLArUJXlQM30A/ulMrnCd4MEyIE+ReyWAUevUQ
0lYdVNva0/W4C5e2lUOJL41jjIPLqI7tcFR2PZE6n0xTTkxNH5W2u1WpFeKjx+O3
czv7Bt6wYyLHIMy1JEqAQ7pw1mtJ5s76UDvXUhciF+DU2pWYc6APKR0=
-----END RSA PRIVATE KEY-----

View File

@ -1 +0,0 @@
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0qDT9kEPWc8JQ53b4KnT/ZJOLwb+3c//jpLW/2ofjDyIsPW4FohLpicfouch/zsRpN4G38lua+2BsGls9sMIZc6PXY2L+NIGCkqEMdCoU1Ym8SMtyJklfzp3m/0PeK9s2dLlR3PFRYvyFA4btQK5hkbYDNZPzf4airvzdRzLkrFf81+RemaMI2EtONwJRcbLViPaTXVKJdbFwJTJ1u7yu9wDYWHKBMA92mHTQeP6bhVYCqxJn3to/RfZYd+sHw6mfxVg5OrAlUOYpSV4pDNCAsIHdtZ56V8NQlJL6NJ2vzzSSYUwLMqe88fhrC8yYHoxC07QPy1EdkSTHdohAicyT root@knode01.knw