Doco updates based on recent mailing list discussions/feedback
This commit is contained in:
		
							parent
							
								
									5cd3bd2b97
								
							
						
					
					
						commit
						7b788fcb89
					
				| 
						 | 
				
			
			@ -20,34 +20,35 @@ of a publisher and a consumer:
 | 
			
		|||
 | 
			
		||||

 | 
			
		||||
 | 
			
		||||
As you can see in the graphic, the producers publishes a series of
 | 
			
		||||
On the picture above the producers publishes a series of
 | 
			
		||||
messages, those messages get partitioned to different queues, and then
 | 
			
		||||
our consumer get messages from one of those queues. Therefore if you
 | 
			
		||||
have a partition with 3 queues, then you will need to have at least 3
 | 
			
		||||
our consumer get messages from one of those queues. Therefore if there is
 | 
			
		||||
a partition with 3 queues, it is assumed that there are at least 3
 | 
			
		||||
consumers to get all the messages from those queues.
 | 
			
		||||
 | 
			
		||||
## Auto-scaling ##
 | 
			
		||||
Queues in RabbitMQ are [units of concurrency](http://www.rabbitmq.com/queues.html#runtime-characteristics)
 | 
			
		||||
(and, if there are enough cores available, parallelism). This plugin makes
 | 
			
		||||
it possible to have a single logical queue that is partitioned into
 | 
			
		||||
multiple regular queues ("shards"). This trades off total ordering
 | 
			
		||||
on the logical queue for gains in parallelism.
 | 
			
		||||
 | 
			
		||||
One interesting property of this plugin, is that if you add more nodes
 | 
			
		||||
to your RabbitMQ cluster, then the plugin will automatically create
 | 
			
		||||
more shards in the new node. Say you had a shard with 4 queues in
 | 
			
		||||
`node a` and `node b` just joined the cluster. The plugin will
 | 
			
		||||
automatically create 4 queues in `node b` and join them to the shard
 | 
			
		||||
partition. Already delivered messages _will not_ be rebalanced, but
 | 
			
		||||
newly arriving messages will be partitioned to the new queues.
 | 
			
		||||
Message distribution between shards (partitioning) is achieved
 | 
			
		||||
with a custom exchange type that distributes messages by applying
 | 
			
		||||
a hashing function to the routing key.
 | 
			
		||||
 | 
			
		||||
## Partitioning Messages ##
 | 
			
		||||
 | 
			
		||||
The exchanges that ship by default with RabbitMQ work in a "all or
 | 
			
		||||
## Messages Distribution Between Shards (Partitioning)
 | 
			
		||||
 | 
			
		||||
The exchanges that ship by default with RabbitMQ work in an "all or
 | 
			
		||||
nothing" fashion, i.e: if a routing key matches a set of queues bound
 | 
			
		||||
to the exchange, then RabbitMQ will route the message to all the
 | 
			
		||||
queues in that set. Therefore for this plugin to work, we need to
 | 
			
		||||
queues in that set. For this plugin to work it is necessary to
 | 
			
		||||
route messages to an exchange that would partition messages, so they
 | 
			
		||||
are routed to _at most_ one queue.
 | 
			
		||||
are routed to _at most_ one queue (a subset).
 | 
			
		||||
 | 
			
		||||
The plugin provides a new exchange type `"x-modulus-hash"` that will use
 | 
			
		||||
the traditional hashing technique applying to partition messages
 | 
			
		||||
across queues.
 | 
			
		||||
The plugin provides a new exchange type, `"x-modulus-hash"`, that will use
 | 
			
		||||
a hashing function to partition messages routed to a logical queue
 | 
			
		||||
across a number of regular queues (shards).
 | 
			
		||||
 | 
			
		||||
The `"x-modulus-hash"` exchange will hash the routing key used to
 | 
			
		||||
publish the message and then it will apply a `Hash mod N` to pick the
 | 
			
		||||
| 
						 | 
				
			
			@ -55,13 +56,27 @@ queue where to route the message, where N is the number of queues
 | 
			
		|||
bound to the exchange. **This exchange will completely ignore the
 | 
			
		||||
binding key used to bind the queue to the exchange**.
 | 
			
		||||
 | 
			
		||||
You could also use other exchanges that have similar behaviour like
 | 
			
		||||
the _Consistent Hash Exchange_ or the _Random Exchange_.  The first
 | 
			
		||||
one has the advantage of shipping directly with RabbitMQ.
 | 
			
		||||
There are other exchanges with similar behaviour:
 | 
			
		||||
the _Consistent Hash Exchange_ or the _Random Exchange_.
 | 
			
		||||
Those were designed with regular queues in mind, not this plugin, so `"x-modulus-hash"`
 | 
			
		||||
is highly recommended.
 | 
			
		||||
 | 
			
		||||
If message partitioning is the only feature necessary and the automatic scaling
 | 
			
		||||
of the number of shards (covered below) is not needed or desired, consider using
 | 
			
		||||
[Consistent Hash Exchange](https://github.com/rabbitmq/rabbitmq-consistent-hash-exchange)
 | 
			
		||||
instead of this plugin.
 | 
			
		||||
 | 
			
		||||
 | 
			
		||||
## Auto-scaling
 | 
			
		||||
 | 
			
		||||
One of the main properties of this plugin is that when a new node
 | 
			
		||||
is added to the RabbitMQ cluster, then the plugin will automatically create
 | 
			
		||||
more shards on the new node. Say there is a shard with 4 queues on
 | 
			
		||||
`node a` and `node b` just joined the cluster. The plugin will
 | 
			
		||||
automatically create 4 queues on `node b` and "join" them to the shard
 | 
			
		||||
partition. Already delivered messages _will not_ be rebalanced but
 | 
			
		||||
newly arriving messages will be partitioned to the new queues.
 | 
			
		||||
 | 
			
		||||
If _just need message partitioning_ but not the automatic queue
 | 
			
		||||
creation provided by this plugin, then you can just use the
 | 
			
		||||
[Consistent Hash Exchange](https://github.com/rabbitmq/rabbitmq-consistent-hash-exchange).
 | 
			
		||||
 | 
			
		||||
## Consuming From a Sharded [Pseudo-]Queue ##
 | 
			
		||||
 | 
			
		||||
| 
						 | 
				
			
			@ -109,6 +124,15 @@ this plugin.
 | 
			
		|||
For load balancers, the "least connections" strategy is more likely to produce an even distribution compared
 | 
			
		||||
to round robin and other strategies.
 | 
			
		||||
 | 
			
		||||
### How Evenly Will Messages Be Distributed?
 | 
			
		||||
 | 
			
		||||
As with many data distribution approaches based on a hashing function,
 | 
			
		||||
even distribution between shards depends on the distribution (variability) of inputs,
 | 
			
		||||
that is, routing keys. In other words the larger the set of routing keys is,
 | 
			
		||||
the more even will message distribution between shareds be. If all messages had
 | 
			
		||||
the same routing key, they would all end up on the same shard.
 | 
			
		||||
 | 
			
		||||
 | 
			
		||||
 | 
			
		||||
## Installing ##
 | 
			
		||||
 | 
			
		||||
| 
						 | 
				
			
			@ -116,7 +140,7 @@ to round robin and other strategies.
 | 
			
		|||
 | 
			
		||||
As of RabbitMQ `3.6.0` this plugin is included into the RabbitMQ distribution.
 | 
			
		||||
 | 
			
		||||
Enable it with the following command:
 | 
			
		||||
Like any other [RabbitMQ plugin](http://www.rabbitmq.com/plugins.html) it has to be enabled before it can be used:
 | 
			
		||||
 | 
			
		||||
```bash
 | 
			
		||||
rabbitmq-plugins enable rabbitmq_sharding
 | 
			
		||||
| 
						 | 
				
			
			
 | 
			
		|||
		Loading…
	
		Reference in New Issue