Misc. minor house-keeping fixes: add reasonable GC settings, fix up README, fix up example configs, fix the logging for tools, use a log directory for logs instead of the root directory.

This commit is contained in:
Jay Kreps 2013-07-11 08:06:43 -07:00
parent f3f8fa5c8c
commit 14af713252
21 changed files with 110 additions and 109 deletions

View File

@ -1,59 +1,52 @@
# Kafka is a distributed publish/subscribe messaging system #
# Apache Kafka #
It is designed to support the following
See our [web site](http://kafka.apache.org) details on the project.
* Persistent messaging with O(1) disk structures that provide constant time performance even with many TB of stored messages.
* High-throughput: even with very modest hardware Kafka can support hundreds of thousands of messages per second.
* Explicit support for partitioning messages over Kafka servers and distributing consumption over a cluster of consumer machines while maintaining per-partition ordering semantics.
* Support for parallel data load into Hadoop.
Kafka is aimed at providing a publish-subscribe solution that can handle all activity stream data and processing on a consumer-scale web site. This kind of activity (page views, searches, and other user actions) are a key ingredient in many of the social feature on the modern web. This data is typically handled by "logging" and ad hoc log aggregation solutions due to the throughput requirements. This kind of ad hoc solution is a viable solution to providing logging data to an offline analysis system like Hadoop, but is very limiting for building real-time processing. Kafka aims to unify offline and online processing by providing a mechanism for parallel load into Hadoop as well as the ability to partition real-time consumption over a cluster of machines.
See our [web site](http://kafka.apache.org/) for more details on the project.
## Contribution ##
Kafka is a new project, and we are interested in building the community; we would welcome any thoughts or [patches](https://issues.apache.org/jira/browse/KAFKA). You can reach us [on the Apache mailing lists](http://kafka.apache.org/contact.html).
The Kafka code is available from:
* git clone http://git-wip-us.apache.org/repos/asf/kafka.git kafka
To contribute you can follow:
* https://cwiki.apache.org/confluence/display/KAFKA/Git+Workflow
## Building It ##
To build for all supported versions of Scala:
1. ./sbt +package
To build for a particular version of Scala (either 2.8.0, 2.8.2, 2.9.1 or 2.9.2):
1. ./sbt "++2.8.0 package" *or* ./sbt "++2.8.2 package" *or* ./sbt "++2.9.1 package" *or* ./sbt "++2.9.2 package"
## Running It ##
To run follow the instructions here:
1. http://kafka.apache.org/08/quickstart.html
## Other Build Tips ##
Here are some useful sbt commands, to be executed at the sbt command prompt (./sbt). Prefixing with "++<version> " runs the
command for a specific Scala version, prefixing with "+" will perform the action for all versions of Scala, and no prefix
runs the command for the default (2.8.0) version of Scala. -
tasks : Lists all the sbt commands and their descriptions
clean : Deletes all generated files (the target directory).
compile : Compile all the sub projects, but not create the jars
test : Run all unit tests in all sub projects
release-zip : Create all the jars, run unit tests and create a deployable release zip
package: Creates jars for src, test, docs etc
projects : List all the sub projects
project sub_project_name : Switch to a particular sub-project. For example, to switch to the core kafka code, use "project core-kafka"
Following commands can be run only on a particular sub project -
The following commands can be run only on a particular sub project -
test-only package.test.TestName : Runs only the specified test in the current sub project
run : Provides options to run any of the classes that have a main method. For example, you can switch to project java-examples, and run the examples there by executing "project java-examples" followed by "run"
For more details please see the [SBT documentation](https://github.com/harrah/xsbt/wiki)
## Contribution ##
Kafka is a new project, and we are interested in building the community; we would welcome any thoughts or [patches](https://issues.apache.org/jira/browse/KAFKA). You can reach us [on the Apache mailing lists](http://kafka.apache.org/contact.html).
To contribute follow the instructions here:
* http://kafka.apache.org/contributing.html
We also welcome patches for the website and documentation which can be found here:
* https://svn.apache.org/repos/asf/kafka/site

View File

@ -15,5 +15,4 @@
# limitations under the License.
base_dir=$(dirname $0)
export KAFKA_OPTS="-Xmx512M -server -Dcom.sun.management.jmxremote -Dlog4j.configuration=file:$base_dir/kafka-console-consumer-log4j.properties"
$base_dir/kafka-run-class.sh kafka.admin.AddPartitionsCommand $@

View File

@ -14,5 +14,4 @@
# See the License for the specific language governing permissions and
# limitations under the License.
base_dir=$(dirname $0)
$base_dir/kafka-run-class.sh kafka.admin.CheckReassignmentStatus $@
$(dirname $0)/kafka-run-class.sh kafka.admin.CheckReassignmentStatus $@

View File

@ -1,21 +0,0 @@
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
log4j.rootLogger=INFO, stderr
log4j.appender.stderr=org.apache.log4j.ConsoleAppender
log4j.appender.stderr.target=System.err
log4j.appender.stderr.layout=org.apache.log4j.PatternLayout
log4j.appender.stderr.layout.ConversionPattern=[%d] %p %m (%c)%n

View File

@ -14,6 +14,5 @@
# See the License for the specific language governing permissions and
# limitations under the License.
base_dir=$(dirname $0)
export KAFKA_OPTS="-Xmx512M -server -Dcom.sun.management.jmxremote -Dlog4j.configuration=file:$base_dir/kafka-console-consumer-log4j.properties"
$base_dir/kafka-run-class.sh kafka.consumer.ConsoleConsumer $@
export KAFKA_HEAP_OPTS="-Xmx512M"
$(dirname $0)/kafka-run-class.sh kafka.consumer.ConsoleConsumer $@

View File

@ -14,5 +14,5 @@
# See the License for the specific language governing permissions and
# limitations under the License.
base_dir=$(dirname $0)
$base_dir/kafka-run-class.sh kafka.producer.ConsoleProducer $@
export KAFKA_HEAP_OPTS="-Xmx512M"
$(dirname $0)/kafka-run-class.sh kafka.producer.ConsoleProducer $@

View File

@ -14,4 +14,5 @@
# See the License for the specific language governing permissions and
# limitations under the License.
export KAFKA_HEAP_OPTS="-Xmx512M"
$(dirname $0)/kafka-run-class.sh kafka.perf.ConsumerPerformance $@

View File

@ -14,6 +14,4 @@
# See the License for the specific language governing permissions and
# limitations under the License.
base_dir=$(dirname $0)
export KAFKA_OPTS="-Xmx512M -server -Dcom.sun.management.jmxremote -Dlog4j.configuration=file:$base_dir/kafka-console-consumer-log4j.properties"
$base_dir/kafka-run-class.sh kafka.admin.CreateTopicCommand $@
$(dirname $0)/kafka-run-class.sh kafka.admin.CreateTopicCommand $@

View File

@ -14,6 +14,4 @@
# See the License for the specific language governing permissions and
# limitations under the License.
base_dir=$(dirname $0)
export KAFKA_OPTS="-Xmx512M -server -Dcom.sun.management.jmxremote -Dlog4j.configuration=file:$base_dir/kafka-console-consumer-log4j.properties"
$base_dir/kafka-run-class.sh kafka.admin.ListTopicCommand $@
$(dirname $0)/kafka-run-class.sh kafka.admin.ListTopicCommand $@

View File

@ -14,5 +14,4 @@
# See the License for the specific language governing permissions and
# limitations under the License.
base_dir=$(dirname $0)
$base_dir/kafka-run-class.sh kafka.admin.PreferredReplicaLeaderElectionCommand $@
$(dirname $0)/kafka-run-class.sh kafka.admin.PreferredReplicaLeaderElectionCommand $@

View File

@ -14,4 +14,5 @@
# See the License for the specific language governing permissions and
# limitations under the License.
export KAFKA_HEAP_OPTS="-Xmx512M"
$(dirname $0)/kafka-run-class.sh kafka.perf.ProducerPerformance $@

View File

@ -14,5 +14,4 @@
# See the License for the specific language governing permissions and
# limitations under the License.
base_dir=$(dirname $0)
$base_dir/kafka-run-class.sh kafka.admin.ReassignPartitionsCommand $@
$(dirname $0)/kafka-run-class.sh kafka.admin.ReassignPartitionsCommand $@

View File

@ -14,6 +14,4 @@
# See the License for the specific language governing permissions and
# limitations under the License.
base_dir=$(dirname $0)
export KAFKA_OPTS="-Xmx512M -server -Dcom.sun.management.jmxremote -Dlog4j.configuration=file:$base_dir/../config/log4j.properties"
$base_dir/kafka-run-class.sh kafka.tools.ReplayLogProducer $@
$(dirname $0)/kafka-run-class.sh kafka.tools.ReplayLogProducer $@

View File

@ -22,10 +22,18 @@ fi
base_dir=$(dirname $0)/..
SCALA_VERSION=2.8.0
# create logs directory
LOG_DIR=$base_dir/logs
if [ ! -d $LOG_DIR ]; then
mkdir $LOG_DIR
fi
if [ -z "$SCALA_VERSION" ]; then
SCALA_VERSION=2.8.0
fi
# assume all dependencies have been packaged into one jar with sbt-assembly's task "assembly-package-dependency"
for file in $base_dir/core/target/scala-2.8.0/*.jar;
for file in $base_dir/core/target/scala-${SCALA_VERSION}/*.jar;
do
CLASSPATH=$CLASSPATH:$file
done
@ -46,30 +54,59 @@ do
CLASSPATH=$CLASSPATH:$file
done
# JMX settings
if [ -z "$KAFKA_JMX_OPTS" ]; then
KAFKA_JMX_OPTS="-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false "
fi
if [ -z "$KAFKA_OPTS" ]; then
KAFKA_OPTS="-Xmx512M -server -Dlog4j.configuration=file:$base_dir/config/log4j.properties"
fi
# JMX port to use
if [ $JMX_PORT ]; then
KAFKA_JMX_OPTS="$KAFKA_JMX_OPTS -Dcom.sun.management.jmxremote.port=$JMX_PORT "
fi
# Log4j settings
if [ -z "$KAFKA_LOG4J_OPTS" ]; then
KAFKA_LOG4J_OPTS="-Dlog4j.configuration=file:$base_dir/config/tools-log4j.properties"
fi
# Generic jvm settings you want to add
if [ -z "$KAFKA_OPTS" ]; then
KAFKA_OPTS=""
fi
# Which java to use
if [ -z "$JAVA_HOME" ]; then
JAVA="java"
else
JAVA="$JAVA_HOME/bin/java"
fi
$JAVA $KAFKA_OPTS $KAFKA_JMX_OPTS -cp $CLASSPATH "$@"
# Memory options
if [ -z "$KAFKA_HEAP_OPTS" ]; then
KAFKA_HEAP_OPTS="-Xmx256M"
fi
# JVM performance options
if [ -z "$KAFKA_JVM_PERFORMANCE_OPTS" ]; then
KAFKA_JVM_PERFORMANCE_OPTS="-server -XX:+UseCompressedOops -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -XX:+CMSScavengeBeforeRemark -XX:+DisableExplicitGC"
fi
# GC options
GC_FILE_SUFFIX='-gc.log'
GC_LOG_FILE_NAME=''
if [ "$1" = "daemon" ] && [ -z "$KAFKA_GC_LOG_OPTS"] ; then
shift
GC_LOG_FILE_NAME=$1$GC_FILE_SUFFIX
shift
KAFKA_GC_LOG_OPTS="-Xloggc:$LOG_DIR/$GC_LOG_FILE_NAME -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps "
fi
$JAVA $KAFKA_HEAP_OPTS $KAFKA_JVM_PERFORMANCE_OPTS $KAFKA_GC_LOG_OPTS $KAFKA_JMX_OPTS $KAFKA_LOG4J_OPTS -cp $CLASSPATH $KAFKA_OPTS "$@"
exitval=$?
if [ $exitval -eq "1" ] ; then
$JAVA $KAFKA_OPTS $KAFKA_JMX_OPTS -cp $CLASSPATH "$@" >& exception.txt
$JAVA $KAFKA_HEAP_OPTS $KAFKA_JVM_PERFORMANCE_OPTS $KAFKA_GC_LOG_OPTS $KAFKA_JMX_OPTS $KAFKA_LOG4J_OPTS -cp $CLASSPATH $KAFKA_OPTS "$@" >& exception.txt
exception=`cat exception.txt`
noBuildMessage='Please build the project using sbt. Documentation is available at http://kafka.apache.org/'
pattern="(Could not find or load main class)|(java\.lang\.NoClassDefFoundError)"

View File

@ -19,7 +19,7 @@ then
echo "USAGE: $0 server.properties"
exit 1
fi
export JMX_PORT=${JMX_PORT:-9999}
$(dirname $0)/kafka-run-class.sh kafka.Kafka $@
base_dir=$(dirname $0)
export KAFKA_LOG4J_OPTS="-Dlog4j.configuration=file:$base_dir/../config/log4j.properties"
export KAFKA_HEAP_OPTS="-Xmx1G -Xms1G"
$base_dir/kafka-run-class.sh daemon kafkaServer kafka.Kafka $@

View File

@ -14,4 +14,5 @@
# See the License for the specific language governing permissions and
# limitations under the License.
export KAFKA_HEAP_OPTS="-Xmx512M"
$(dirname $0)/kafka-run-class.sh kafka.perf.SimpleConsumerPerformance $@

View File

@ -19,5 +19,8 @@ then
echo "USAGE: $0 zookeeper.properties"
exit 1
fi
base_dir=$(dirname $0)
export KAFKA_LOG4J_OPTS="-Dlog4j.configuration=file:$base_dir/../config/log4j.properties"
export KAFKA_HEAP_OPTS="-Xmx512M -Xms512M"
$base_dir/kafka-run-class.sh daemon zookeeper org.apache.zookeeper.server.quorum.QuorumPeerMain $@
$(dirname $0)/kafka-run-class.sh org.apache.zookeeper.server.quorum.QuorumPeerMain $@

View File

@ -20,25 +20,25 @@ log4j.appender.stdout.layout.ConversionPattern=[%d] %p %m (%c)%n
log4j.appender.kafkaAppender=org.apache.log4j.DailyRollingFileAppender
log4j.appender.kafkaAppender.DatePattern='.'yyyy-MM-dd-HH
log4j.appender.kafkaAppender.File=server.log
log4j.appender.kafkaAppender.File=logs/server.log
log4j.appender.kafkaAppender.layout=org.apache.log4j.PatternLayout
log4j.appender.kafkaAppender.layout.ConversionPattern=[%d] %p %m (%c)%n
log4j.appender.stateChangeAppender=org.apache.log4j.DailyRollingFileAppender
log4j.appender.stateChangeAppender.DatePattern='.'yyyy-MM-dd-HH
log4j.appender.stateChangeAppender.File=state-change.log
log4j.appender.stateChangeAppender.File=logs/state-change.log
log4j.appender.stateChangeAppender.layout=org.apache.log4j.PatternLayout
log4j.appender.stateChangeAppender.layout.ConversionPattern=[%d] %p %m (%c)%n
log4j.appender.requestAppender=org.apache.log4j.DailyRollingFileAppender
log4j.appender.requestAppender.DatePattern='.'yyyy-MM-dd-HH
log4j.appender.requestAppender.File=kafka-request.log
log4j.appender.requestAppender.File=logs/kafka-request.log
log4j.appender.requestAppender.layout=org.apache.log4j.PatternLayout
log4j.appender.requestAppender.layout.ConversionPattern=[%d] %p %m (%c)%n
log4j.appender.controllerAppender=org.apache.log4j.DailyRollingFileAppender
log4j.appender.controllerAppender.DatePattern='.'yyyy-MM-dd-HH
log4j.appender.controllerAppender.File=controller.log
log4j.appender.controllerAppender.File=logs/controller.log
log4j.appender.controllerAppender.layout=org.apache.log4j.PatternLayout
log4j.appender.controllerAppender.layout.ConversionPattern=[%d] %p %m (%c)%n

View File

@ -16,7 +16,7 @@
############################# Producer Basics #############################
# list of brokers used for bootstrapping
# list of brokers used for bootstrapping knowledge about the rest of the cluster
# format: host1:port1,host2:port2 ...
metadata.broker.list=localhost:9092

View File

@ -47,8 +47,8 @@ socket.request.max.bytes=104857600
############################# Log Basics #############################
# The directory under which to store log files
log.dir=/tmp/kafka-logs
# A comma seperated list of directories under which to store log files
log.dirs=/tmp/kafka-logs
# The number of logical partitions per topic per server. More partitions allow greater parallelism
# for consumption, but also mean more files.
@ -56,12 +56,12 @@ num.partitions=2
############################# Log Flush Policy #############################
# The following configurations control the flush of data to disk. This is the most
# The following configurations control the flush of data to disk. This is among the most
# important performance knob in kafka.
# There are a few important trade-offs here:
# 1. Durability: Unflushed data is at greater risk of loss in the event of a crash.
# 2. Latency: Data is not made available to consumers until it is flushed (which adds latency).
# 3. Throughput: The flush is generally the most expensive operation.
# 1. Durability: Unflushed data may be lost if you are not using replication.
# 2. Latency: Very large flush intervals may lead to latency spikes when the flush does occur as there will be a lot of data to flush.
# 3. Throughput: The flush is generally the most expensive operation, and a small flush interval may lead to exceessive seeks.
# The settings below allow one to configure the flush policy to flush data after a period of time or
# every N messages (or both). This can be done globally and overridden on a per-topic basis.
@ -107,10 +107,4 @@ zookeeper.connect=localhost:2181
# Timeout in ms for connecting to zookeeper
zookeeper.connection.timeout.ms=1000000
# metrics reporter properties
kafka.metrics.polling.interval.secs=5
kafka.metrics.reporters=kafka.metrics.KafkaCSVMetricsReporter
kafka.csv.metrics.dir=/tmp/kafka_metrics
# Disable csv reporting by default.
kafka.csv.metrics.reporter.enabled=false

View File

@ -1,19 +1,22 @@
#!/bin/bash
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
#
# http://www.apache.org/licenses/LICENSE-2.0
#
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
base_dir=$(dirname $0)
export KAFKA_OPTS="-Xmx512M -server -Dcom.sun.management.jmxremote -Dlog4j.configuration=file:$base_dir/kafka-console-consumer-log4j.properties"
$base_dir/kafka-run-class.sh kafka.admin.DeleteTopicCommand $@
log4j.rootLogger=WARN, stdout
log4j.appender.stdout=org.apache.log4j.ConsoleAppender
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern=[%d] %p %m (%c)%n