diff --git a/test-common/test-common-internal-api/src/main/java/org/apache/kafka/common/test/api/README.md b/test-common/test-common-internal-api/src/main/java/org/apache/kafka/common/test/api/README.md index 10a7b6dcf9a..3c4efda30cc 100644 --- a/test-common/test-common-internal-api/src/main/java/org/apache/kafka/common/test/api/README.md +++ b/test-common/test-common-internal-api/src/main/java/org/apache/kafka/common/test/api/README.md @@ -3,46 +3,80 @@ cluster configurations. # Annotations -A new `@ClusterTest` annotation is introduced which allows for a test to declaratively configure an underlying Kafka cluster. +Three annotations are provided for defining a template of a Kafka cluster. -```scala -@ClusterTest -def testSomething(): Unit = { ... } -``` +* `@ClusterTest`: declarative style cluster definition +* `@ClusterTests`: wrapper around multiple `@ClusterTest`-s +* `@ClusterTemplate`: points to a function for imperative cluster definition -This annotation has fields for a set of cluster types and number of brokers, as well as commonly parameterized configurations. -Arbitrary server properties can also be provided in the annotation: +Another helper annotation `@ClusterTestDefaults` allows overriding the defaults for +all `@ClusterTest` in a single test class. + +# Usage + +The simplest usage is `@ClusterTest` by itself which will use some reasonable defaults. ```java -@ClusterTest( - types = {Type.KRAFT}, - brokerSecurityProtocol = SecurityProtocol.PLAINTEXT, - properties = { - @ClusterProperty(key = "inter.broker.protocol.version", value = "2.7-IV2"), - @ClusterProperty(key = "socket.send.buffer.bytes", value = "10240"), -}) -void testSomething() { ... } +public class SampleTest { + @ClusterTest + void testSomething() { ... } +} ``` -Multiple `@ClusterTest` annotations can be given to generate more than one test invocation for the annotated method. +The defaults can be modified by setting specific paramters on the annotation. -```scala -@ClusterTests(Array( - new ClusterTest(brokerSecurityProtocol = SecurityProtocol.PLAINTEXT), - new ClusterTest(brokerSecurityProtocol = SecurityProtocol.SASL_PLAINTEXT) -)) -def testSomething(): Unit = { ... } +```java +public class SampleTest { + @ClusterTest(brokers = 3, metadataVersion = MetadataVersion.IBP_4_0_IV3) + void testSomething() { ... } +} ``` -A class-level `@ClusterTestDefaults` annotation is added to provide default values for `@ClusterTest` defined within -the class. The intention here is to reduce repetitive annotation declarations and also make changing defaults easier -for a class with many test cases. +It is also possible to modify the defaults for a whole class using `@ClusterTestDefaults`. + +```java +@ClusterTestDefaults(brokers = 3, metadataVersion = MetadataVersion.IBP_4_0_IV3) +public class SampleTest { + @ClusterTest + void testSomething() { ... } +} +``` + +To set some specific config, an array of `@ClusterProperty` annotations can be +given. + +```java +public class SampleTest { + @ClusterTest( + types = {Type.KRAFT}, + brokerSecurityProtocol = SecurityProtocol.PLAINTEXT, + properties = { + @ClusterProperty(key = "inter.broker.protocol.version", value = "2.7-IV2"), + @ClusterProperty(key = "socket.send.buffer.bytes", value = "10240"), + }) + void testSomething() { ... } +} +``` + +Using the `@ClusterTests` annotation, multiple declarative cluster templates can +be given. + +```java +public class SampleTest { + @ClusterTests({ + @ClusterTest(brokerSecurityProtocol = SecurityProtocol.PLAINTEXT), + @ClusterTest(brokerSecurityProtocol = SecurityProtocol.SASL_PLAINTEXT) + }) + void testSomething() { ... } +} +``` # Dynamic Configuration -In order to allow for more flexible cluster configuration, a `@ClusterTemplate` annotation is also introduced. This -annotation takes a single string value which references a static method on the test class. This method is used to -produce any number of test configurations using a fluent builder style API. +In order to allow for more flexible cluster configuration, a `@ClusterTemplate` +annotation is also introduced. This annotation takes a single string value which +references a static method on the test class. This method is used to produce any +number of test configurations using a fluent builder style API. ```java import java.util.Arrays; @@ -69,57 +103,90 @@ static List generateConfigs() { } ``` -This "escape hatch" from the simple declarative style configuration makes it easy to dynamically configure clusters. +This alternate configuration style makes it easy to create any number of complex +configurations. Each returned ClusterConfig by a template method will result in +an additional variation of the run. # JUnit Extension -One thing to note is that our "test*" methods are no longer _tests_, but rather they are test templates. We have added -a JUnit extension called `ClusterTestExtensions` which knows how to process these annotations in order to generate test -invocations. Test classes that wish to make use of these annotations need to explicitly register this extension: +The core logic of our test framework lies in `ClusterTestExtensions` which is a +JUnit extension. It is automatically registered using SPI and will look for test +methods that include one of the three annotations mentioned above. -```scala -import org.apache.kafka.common.test.junit.ClusterTestExtensions - -@ExtendWith(value = Array(classOf[ClusterTestExtensions])) -class ApiVersionsRequestTest { - ... -} -``` +This way of dynamically generating tests uses the JUnit concept of test templates. # JUnit Lifecycle -The lifecycle of a test class that is extended with `ClusterTestExtensions` follows: +JUnit discovers test template methods that are annotated with `@ClusterTest`, +`@ClusterTests`, or `@ClusterTemplate`. These annotations are processed and some +number of test invocations are created. -* JUnit discovers test template methods that are annotated with `@ClusterTest`, `@ClusterTests`, or `@ClusterTemplate` -* `ClusterTestExtensions` is called for each of these template methods in order to generate some number of test invocations +For each generated test invocation we have the following lifecycle: -For each generated invocation: * Static `@BeforeAll` methods are called * Test class is instantiated +* Kafka Cluster is started (if autoStart=true) * Non-static `@BeforeEach` methods are called -* Kafka Cluster is started * Test method is invoked * Kafka Cluster is stopped * Non-static `@AfterEach` methods are called * Static `@AfterAll` methods are called -`@BeforeEach` methods give an opportunity to setup additional test dependencies before the cluster is started. +`@BeforeEach` methods give an opportunity to set up additional test dependencies +after the cluster has started but before the test method is run. # Dependency Injection -The class is introduced to provide context to the underlying cluster and to provide reusable functionality that was -previously garnered from the test hierarchy. +A ClusterInstance object can be injected into the test method or the test class constructor. +This object is a shim to the underlying test framework and provides access to things like +SocketServers and has convenience factory methods for getting a client. -* ClusterInstance: a shim to the underlying class that actually runs the cluster, provides access to things like SocketServers +The class is introduced to provide context to the underlying cluster and to provide reusable +functionality that was previously garnered from the test hierarchy. -In order to inject the object, simply add it as a parameter to your test class, `@BeforeEach` method, or test method. +Common usage is to inject this class into a test method -| Injection | Class | BeforeEach | Test | Notes -| --- | --- | --- | --- | --- | -| ClusterInstance | yes* | no | yes | Injectable at class level for convenience, can only be accessed inside test | +```java +class SampleTest { + @ClusterTest + public void testOne(ClusterInstance cluster) { + this.cluster.admin().createTopics(...); + // Test code + } +} +``` + +For cases where there is common setup code that involves the cluster (such as +creating topics), it is possible to access the ClusterInstance from a `@BeforeEach` +method. This requires injecting the object in the constructor. For example, + +```java +class SampleTest { + private final ClusterInstance cluster; + + SampleTest(ClusterInstance cluster) { + this.cluster = cluster; + } + + @BeforeEach + public void setup() { + // Common setup code with started ClusterInstance + this.cluster.admin().createTopics(...); + } + + @ClusterTest + public void testOne() { + // Test code + } +} +``` + +It is okay to inject the ClusterInstance in both ways. The same object will be +provided in either case. # Gotchas +* Cluster tests are not compatible with other test templates like `@ParameterizedTest` * Test methods annotated with JUnit's `@Test` will still be run, but no cluster will be started and no dependency injection will happen. This is generally not what you want. * Even though ClusterConfig is accessible, it is immutable inside the test method. diff --git a/test-common/test-common-runtime/src/main/java/org/apache/kafka/common/test/ClusterInstance.java b/test-common/test-common-runtime/src/main/java/org/apache/kafka/common/test/ClusterInstance.java index 26b4e1e471a..2bcfc879f0e 100644 --- a/test-common/test-common-runtime/src/main/java/org/apache/kafka/common/test/ClusterInstance.java +++ b/test-common/test-common-runtime/src/main/java/org/apache/kafka/common/test/ClusterInstance.java @@ -263,8 +263,12 @@ public interface ClusterInstance { void start(); + boolean started(); + void stop(); + boolean stopped(); + void shutdownBroker(int brokerId); void startBroker(int brokerId); diff --git a/test-common/test-common-runtime/src/main/java/org/apache/kafka/common/test/junit/RaftClusterInvocationContext.java b/test-common/test-common-runtime/src/main/java/org/apache/kafka/common/test/junit/RaftClusterInvocationContext.java index 76bb24a32b4..956013e5d13 100644 --- a/test-common/test-common-runtime/src/main/java/org/apache/kafka/common/test/junit/RaftClusterInvocationContext.java +++ b/test-common/test-common-runtime/src/main/java/org/apache/kafka/common/test/junit/RaftClusterInvocationContext.java @@ -38,7 +38,7 @@ import org.apache.kafka.server.common.MetadataVersion; import org.apache.kafka.server.fault.FaultHandlerException; import org.junit.jupiter.api.extension.AfterTestExecutionCallback; -import org.junit.jupiter.api.extension.BeforeTestExecutionCallback; +import org.junit.jupiter.api.extension.BeforeEachCallback; import org.junit.jupiter.api.extension.Extension; import org.junit.jupiter.api.extension.TestTemplateInvocationContext; @@ -90,7 +90,7 @@ public class RaftClusterInvocationContext implements TestTemplateInvocationConte public List getAdditionalExtensions() { RaftClusterInstance clusterInstance = new RaftClusterInstance(clusterConfig, isCombined); return Arrays.asList( - (BeforeTestExecutionCallback) context -> { + (BeforeEachCallback) context -> { clusterInstance.format(); if (clusterConfig.isAutoStart()) { clusterInstance.start(); @@ -189,6 +189,11 @@ public class RaftClusterInvocationContext implements TestTemplateInvocationConte } } + @Override + public boolean started() { + return started.get(); + } + @Override public void stop() { if (stopped.compareAndSet(false, true)) { @@ -196,6 +201,11 @@ public class RaftClusterInvocationContext implements TestTemplateInvocationConte } } + @Override + public boolean stopped() { + return stopped.get(); + } + @Override public void shutdownBroker(int brokerId) { findBrokerOrThrow(brokerId).shutdown(); diff --git a/test-common/test-common-runtime/src/test/java/org/apache/kafka/common/test/junit/ClusterTestBeforeEachTest.java b/test-common/test-common-runtime/src/test/java/org/apache/kafka/common/test/junit/ClusterTestBeforeEachTest.java new file mode 100644 index 00000000000..32fadf3cc65 --- /dev/null +++ b/test-common/test-common-runtime/src/test/java/org/apache/kafka/common/test/junit/ClusterTestBeforeEachTest.java @@ -0,0 +1,59 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.kafka.common.test.junit; + +import org.apache.kafka.common.test.ClusterInstance; +import org.apache.kafka.common.test.api.AutoStart; +import org.apache.kafka.common.test.api.ClusterTest; + +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.extension.ExtendWith; + +import static org.junit.jupiter.api.Assertions.assertDoesNotThrow; +import static org.junit.jupiter.api.Assertions.assertNotNull; +import static org.junit.jupiter.api.Assertions.assertTrue; + +@ExtendWith(ClusterTestExtensions.class) +public class ClusterTestBeforeEachTest { + private final ClusterInstance clusterInstance; + + ClusterTestBeforeEachTest(ClusterInstance clusterInstance) { // Constructor injections + this.clusterInstance = clusterInstance; + } + + @BeforeEach + void before() { + assertNotNull(clusterInstance); + if (!clusterInstance.started()) { + clusterInstance.start(); + } + assertDoesNotThrow(clusterInstance::waitForReadyBrokers); + } + + @ClusterTest + public void testDefault() { + assertTrue(true); + assertNotNull(clusterInstance); + } + + @ClusterTest(autoStart = AutoStart.NO) + public void testNoAutoStart() { + assertTrue(true); + assertNotNull(clusterInstance); + } +}