0d84dc1e48
Merge pull request #3098 from rabbitmq/unexpected-socket-input
...
Close stream socket if client doesn't follow authentication protocol
2021-06-11 03:25:45 +03:00
1cca0f1e4c
Categorize connection log messages
2021-06-10 19:50:48 +02:00
fcc8dbeab6
Close client connections that don't follow authentication protocol
...
Before this commit sending garbarge data to the server stream port
caused the RabbitMQ node to eat more and more memory.
In this commit, we fix it by expecting the client to go through the
proper authentication sequence. Otherwise, the server closes the socket.
Co-authored-by: Michal Kuratczyk <mkuratczyk@pivotal.io>
2021-06-10 15:44:51 +02:00
dcd65572a0
Remove correlation ID from commit_offset
...
In stream protocol. commit_offset is asynchronous and does
not expect a response, so the correlation ID is not required.
2021-06-10 15:21:38 +02:00
b77fb27af3
Merge pull request #3083 from rabbitmq/raw-reader
...
Raw reader option
2021-06-08 17:48:20 +02:00
935f57b608
Chunk selector option in offset reader
2021-06-08 15:17:22 +02:00
7407a5a100
Apply policy in rabbit_queue_type
2021-06-07 12:30:33 +02:00
7d5c8f402a
Merge pull request #3071 from rabbitmq/subscribe-props
...
Fix properties binary construction
2021-06-03 18:09:52 +02:00
761af0a7a0
Extract publishing IDs from batch publishing
...
In stream plugin, to e.g. send publish errors in case the stream
does not exist. Batches were not taken into account.
2021-05-31 15:35:10 +02:00
65c9dae53f
Fix properties binary construction
2021-05-28 15:45:19 +02:00
f36751aa6d
make rabbit_stream_SUITE more reliable
...
By having rabbit_stream_core cache it's incoming command internally.
2021-05-27 13:15:55 +01:00
2ab5cb22ca
Expose TLS info for stream connections (CLI, REST API)
2021-05-27 10:43:33 +02:00
5a6dbef372
Return TLS port in stream connection properties
2021-05-26 12:30:54 +02:00
69ad6969e6
Add stream.advertised_tls_port setting
2021-05-26 11:08:43 +02:00
35ef1e5ade
Merge pull request #3038 from rabbitmq/stream-tls
...
TLS support for streams
2021-05-25 15:39:40 +02:00
a6f70b8dda
Add xref for remaining tier-1 plugins
2021-05-25 11:39:03 +02:00
b7a2e9a792
Fix comment
2021-05-25 09:53:04 +02:00
4a9d8115f8
rebase fixes
...
post rebase test fixes
Make socket initialisation more lenient
correct return types
fix
remove commented code
2021-05-24 15:53:10 +01:00
3fefa8e8d4
Use ssl option when initialising data reader
2021-05-21 17:13:15 +01:00
b2a7884a45
TLS support for streams
2021-05-21 16:40:57 +01:00
03063f2eed
Merge pull request #3043 from rabbitmq/streams-consumer-lag-metrics
...
Add consumer offset_lag to rabbitmq-stream CLI command & Management
2021-05-21 16:38:50 +01:00
f8b4e1e298
Add consumer offset_lag to rabbitmq-stream CLI command & Management
...
This is an important metric to keep track of and be aware (maybe even
alert on) when consumers fall behind consuming stream messages. While
they should be able to catch up, if they fall behind too much and the
stream gets truncated, they may miss on messages.
This is something that we want to expose via Prometheus metrics as well,
but we've started closer to the core, CLI & Management.
This should be merged as soon as it passes CI, we shouldn't wait on the
Prometheus changes - they can come later.
Pair: @kjnilsson
Signed-off-by: Gerhard Lazu <gerhard@lazu.co.uk>
2021-05-21 13:02:48 +01:00
8f8e08a9a9
Send advertised host and port in open
...
More secure than in peer properties frame, which is just
at the beginning of the connection opening.
2021-05-21 13:03:38 +02:00
6c38b42ad6
Log heartbets at debug; remove newlines
...
A log line every minute about a successful heartbeat pollutes the logs.
So do empty log lines.
2021-05-20 18:46:15 +02:00
c30e013d7a
Rename max-segment-size to stream-max-segment-size-bytes
2021-05-20 10:16:19 +02:00
0b73c9337c
Fix some logging statements in stream plugin
...
Missing arguments in the format.
2021-05-19 11:22:15 +02:00
d9b7523987
Handle connection closing when dispatching stream messages
2021-05-19 10:05:37 +02:00
7145a1a2ad
Trigger event on stream consumer cancellation
...
To make metrics are cleaned up.
2021-05-18 17:42:33 +02:00
194198a450
Add stream consumer properties to list command
2021-05-18 17:16:06 +02:00
77b99a2efd
Add stream subscription properties to metrics
2021-05-18 16:54:41 +02:00
7a32a4483f
Use ~0p to print map on one line
2021-05-18 14:52:12 +02:00
90a8c11450
Stream: tweak type and tidy up
2021-05-18 10:02:38 +01:00
28f3de7f2a
Support properties in stream subscription
...
To add metadata. Can be used to map a subscription to
the commit offset reference.
2021-05-18 10:15:26 +02:00
312690315d
rabbit_stream_core: remove duplicate clause
2021-05-17 10:39:55 +01:00
b21966cf86
stream plugin: use core module in test suite
...
this implements a basic "client" and tests the core module on both server and client.
2021-05-17 10:39:55 +01:00
a1ce34c1b8
Extract stream protocol framing and binary generation
...
into a separate module.
2021-05-17 10:39:52 +01:00
4f3b068886
Fix for a compiler warning in Erlang 24
2021-05-17 11:09:19 +02:00
787b46e3a1
Propagate advertised host and port to client
2021-05-03 10:32:33 +02:00
e3c4c9a471
Use stream leader to query offset
2021-04-27 15:38:21 +02:00
e4307b6180
Adapt stream code to osiris tracking types introduction
2021-04-27 09:44:59 +02:00
39f8be929d
Block stream publishing on disk alarm
2021-04-20 16:49:01 +02:00
9bd02b3510
Make streams compatible with osiris API changes
...
In osiris:init_reader
2021-04-13 15:02:32 +01:00
c70de74c5e
Handle "undefined" PID when writing osiris tracking
2021-04-09 15:45:28 +02:00
88154a72d0
Handle closed osiris reader when dispatching chunk
2021-04-09 15:26:06 +02:00
23aee072eb
Handle creation of stream with different parameters
2021-04-08 15:50:51 +02:00
e46216b5a8
Check PID on leader lookup in stream plugin
...
To make sure the PID is alive, as the mnesia record can stale after a
failure.
Make also the local PID lookup in the stream coordinator do a consistent
query over the cluster if the PID is not alive.
Co-authored-by: Karl Nilsson <kjnilsson@users.noreply.github.com>
2021-03-12 15:04:40 +00:00
e80db9d46a
Adapt stream plugin after coordinator refactoring
...
Do not use mnesia record for topology info, use coordinator instead.
2021-03-12 15:04:35 +00:00
97ff62d3b2
Drop trailing newlines from logged messages where possible
...
Lager strips trailing newline characters but OTP logger with the default
formatter adds a newline at the end. To avoid unintentional multi-line log
messages we have to revisit most messages logged.
Some log entries are intentionally multiline, others
are printed to stdout directly: newlines are required there
for sensible formatting.
2021-03-11 15:17:37 +01:00
570eb509c2
Naming
2021-03-10 07:45:49 +03:00
e0d567828f
Update ranch supervisor structure on list/0 command
2021-03-09 23:17:03 +01:00
61f7b2a723
Update to ranch 2.0
2021-03-08 23:11:05 +01:00
26566c10e2
Add comment
2021-02-26 12:14:21 +01:00
9f0bec8def
Start constants at 1 for stream plugin
2021-02-26 11:48:45 +01:00
768a61792e
Use MSB in keys to distinguish requests/responses
...
In stream protocol.
2021-02-26 11:03:36 +01:00
d70b55aa0c
Add route and partitions command to stream plugin
...
For publishing to super streams.
2021-02-25 17:22:42 +01:00
8868ad97b3
Add logging to stream subscription
2021-02-11 16:16:26 +01:00
d63756ed67
Set correct offset metrics on stream consumer creation
2021-02-09 11:36:20 +01:00
78b723d7c3
Report stream consumers as regular consumers in management
...
This helps to see that a queue is consumed from.
WIP
2021-01-20 15:36:16 +01:00
2f903843e0
Add list_stream_publishers CLI command
2021-01-19 14:49:30 +01:00
1e6244990c
Add list_stream_consumers CLI command
2021-01-19 11:31:39 +01:00
0d73b58be0
Re-order stream command constants
...
The order is more usage-oriented.
2021-01-18 11:25:21 +01:00
5f0df3bdb0
Format stream and stream management plugins code
2021-01-06 15:47:27 +01:00
fc88f3ba21
Check that the queue exists before calling declare
...
Rabbit channels are responsible of this check before calling declare,
skipping it on the manager meant that the queue was partly redeclared
and a new data directory created. The old one was still on disk with
a different timestamp, but from the user point of view the queue data
has been erased.
2020-12-23 12:18:33 +01:00
c0bb7ec4a7
Add offset & consumed metrics to stream consumer
2020-12-15 15:37:26 +01:00
c4d07467da
Merge branch 'master' into rabbitmq-stream-management
2020-12-09 12:00:56 +01:00
9a590647b3
Add chunk header parameter to send_file callback
2020-12-09 11:33:28 +01:00
2b45d8635b
Add publisher reference to stream endpoint
2020-12-07 10:17:01 +01:00
1faeaad499
Add publisher REST endpoint to stream consumer management
2020-12-04 17:22:14 +01:00
88b4ddb6c2
Track publishers in stream plugin
2020-12-03 17:57:48 +01:00
c14c9bb6da
Fix some dialyzer warnings in stream plugin
2020-12-03 10:09:29 +01:00
66867a4062
Add GC to stream consumer monitoring table
2020-12-02 18:12:45 +01:00
41160210ab
Start stream consumer REST endpoint
2020-12-01 18:23:22 +01:00
08891a734e
Merge branch 'master' into rabbitmq-stream-management
2020-11-30 09:42:54 +01:00
67308be13c
Remove stream field from publish command
...
Only the publisher ID is necessary now the publisher declaration is
mandatory before publishing.
2020-11-27 11:07:10 +01:00
bfa541a095
Include publisher state management
2020-11-26 18:18:43 +01:00
8cf5fb0775
Enforce publisher ID uniqueness
2020-11-26 15:19:02 +01:00
59bc60fe36
Add query publisher sequence
2020-11-26 10:02:13 +01:00
8f97ea400a
Start adding publishing dedup support for streams
2020-11-24 17:48:41 +01:00
580beb21bf
Include publish dedup in stream protocol
2020-11-20 11:43:04 +01:00
c8249a304f
Filter stream connections where metrics are disabled
...
This implied defining the protocol field in tracked connection to be
able to filter out non-stream connections.
2020-11-20 09:29:55 +01:00
e516aab737
Display only stream connections in plugin
2020-11-19 17:02:35 +01:00
1ed8f76eed
Check if member processes are alive in metadata command
...
In case the Mnesia record is stale.
2020-11-17 10:15:09 +01:00
29e74cbfca
Move some code to utils module
2020-11-05 15:33:13 +01:00
fa586140d3
Call rabbit_log:warning/2
2020-11-03 18:04:22 +01:00
0f569d705a
Return Erlang node long name in metadata
...
Fixes #2
2020-11-03 11:45:22 +01:00
2813356682
Use non_amqp_connection name for force_refresh_event
2020-10-23 15:37:07 +02:00
a2f05f5b75
Support shutdown call from management plugin
2020-10-23 11:14:04 +02:00
bd191cf40d
Register connections and handle force_event_refresh
...
For existing connections that must emit stats when the management plugin
is enabled.
References rabbitmq/rabbitmq-server#2481
2020-10-23 10:17:39 +02:00
bb510b0e14
Merge branch 'master' into management-integration
...
Conflicts:
src/rabbit_stream_manager.erl
2020-10-22 15:41:42 +02:00
dd8b3ea008
Check stats timer in listen loop
2020-10-22 14:27:06 +02:00
43898e59b1
Support max-age in stream creation
2020-10-22 13:42:33 +02:00
a1f6006356
Support leader locator strategies
...
References rabbitmq/rabbitmq-server#2471
2020-10-21 15:25:31 +02:00
20681aca63
Add event notification during authentication
2020-10-20 18:20:03 +02:00
184f177d78
Emit stats for management
...
WIP
2020-10-20 15:09:34 +02:00
cc030ac195
Support initial-cluster-size argument on creation
...
See rabbitmq/rabbitmq-server#2467
2020-10-16 14:07:19 +02:00
b704e2f8ef
Check stream name before creation
...
Should not start with "amq.", be empty. Strips also newline
and carriage return characters.
2020-10-16 11:22:16 +02:00
27d06e8021
Add CLI command to list stream connections
2020-10-16 09:18:36 +02:00
5e80aa9f39
Add correlation ID field in commit offset frame
...
Not used yet, but can avoid a command version change if the command
should have confirm response.
2020-10-02 14:16:22 +02:00
03a11e0771
Adapt failure tests
...
They are a bit more defensive. The subscription is also now more
reliable by returning a stream-not-available code if necessary.
Using also Aten poll interval to 1 second (bumped to 5 seconds in master
now).
2020-09-29 11:29:56 +02:00
04e981e9e6
Use rabbit_durable_queue to check if stream exists
...
This is done after checking rabbit_queue and if it returns that the
queue does not exist. The coordinator may be recovering the queue, so
thanks to this double check we know the queue exists but is not
available, instead of thinking it does not exist at all.
2020-09-17 09:30:20 +02:00
cf5e99cd29
Add offset tracking commands
2020-09-15 17:52:05 +02:00
21d9aea96c
Add publisher ID
2020-09-02 15:13:37 +02:00
b2f30a55ba
Use byte for sub ID in credit notification
2020-09-01 18:14:07 +02:00
db93e14932
Use byte for subscription ID
...
Instead of integer. This is to be consistent with the introduction of a
similar ID for publishers (which will be a byte as well).
2020-09-01 18:07:27 +02:00
ea2636cbb3
Add debug log during deletion
2020-08-07 16:43:34 +02:00
40f2db4291
Add function to kill connection based on name
...
For test purposes.
2020-08-05 14:43:07 +02:00
a886cd8872
Send heartbeat from the reader process
...
If the heartbeat frame is sent from a dedicated process, it interleaves
between 2 socket calls from the reader process. Frames are typically
sent in one call, so this is fine, but a chunk is delivered with 2
calls, one for the frame header and one send_file for the chunk. So the
heartbeat frame can sneak in between these 2 calls, which makes clients
fail to parse frames.
2020-07-10 09:42:00 +02:00
92ebb825cb
Handle error case when creating a stream
2020-07-10 09:24:24 +02:00
7740fee13e
Pass in monitor reference, not stream name
2020-07-09 11:33:44 +02:00
af2dcf7f57
Handle publishing sub-entries (batch)
2020-06-30 17:29:28 +02:00
7270125c0b
Fix closing sequence
...
When server sends close, it would not listen on the socket for the
client confirmation.
2020-06-26 16:33:39 +02:00
4a05b6b62a
Ignore non-responding brokers in metadata command
2020-06-25 15:32:33 +02:00
1d3978ae40
Change response code label
...
From "stream deleted" to "stream not available". It covers now the
deletion of a stream and the unavailibility due to a failure. This is up
to the client to find out what to do (typically send a metadata request
about the stream and see if it's still there).
2020-06-23 10:51:34 +02:00
7b3d8c4ad0
Use Erlang monitor to detect stream deletion/failure
2020-06-22 18:33:54 +02:00
5bd352737c
Pass in node information to Java test suite
2020-06-22 16:35:07 +02:00
f8074a9f93
Notify client when it credits an unknown subscription
2020-06-19 15:19:39 +02:00
27c4e1e707
Add authorisation on publishing
2020-06-18 17:00:02 +02:00
5f69ef7cf7
Add authorisation for subscription
2020-06-18 16:13:14 +02:00
397c5ca078
Add authorisation for creation/deletion
2020-06-18 15:25:50 +02:00
3c18498734
Clean manager listener code
...
Now the manager uses the coordinator to know about deleted streams.
2020-06-18 10:57:01 +02:00
a1314a13c9
Use coordinator to get notified on stream deletion
2020-06-18 08:58:59 +02:00
625eaec004
Use rabbit_log
2020-06-15 11:17:27 +02:00
6a07d385ae
Add specs t stream manager
2020-06-11 11:05:02 +02:00
bb5efad9ef
Fix dialyzer errors
2020-06-10 18:13:55 +02:00
76162b18a2
Add type specifications
2020-06-10 16:24:18 +02:00
19a1eb4013
Remove some unnecessary comments
2020-06-10 15:06:51 +02:00
16288a2596
Separate changing state from (almost) immutable state
2020-06-10 15:04:31 +02:00
c39852fa72
Introduce PeerProperties command
...
To exchange client/server properties at the beginning of the connection.
2020-06-10 10:42:29 +02:00
d2a29c5aef
Support client properties
2020-06-09 17:48:23 +02:00
6ef10c8573
Use MPL 2.0
2020-06-08 09:54:33 +02:00
a56cf4013c
Support first/last/next/offset/timestamp offset spec
2020-06-02 16:35:00 +02:00
ecf47a8a42
Fix typo
2020-06-02 10:47:46 +02:00
2ab7e864dc
Consume from a local member
2020-05-28 11:59:13 +02:00
efc6fbc089
Add advertised host/port settings
...
This allows to communicate whatever it's appropriate to clients when
they issue the metadata command.
2020-05-28 10:47:30 +02:00
b54026aa32
Add arguments for stream creation
...
To handle retention settings.
2020-05-19 18:18:30 +02:00
55adca1b44
Make sure queue is stream queue
2020-05-07 11:00:45 +02:00
c2e35eb55f
Use stream coordinator to manage streams
2020-05-06 17:35:41 +02:00
2d19e85925
Use stream instead of target
2020-05-06 09:15:16 +02:00
b61a79b9ff
Credit value is signed
2020-04-03 11:00:35 +02:00
204f61399b
Handle client heartbeat
2020-04-01 10:08:32 +02:00
8c97d442ce
Add heartbeat
2020-03-31 14:43:06 +02:00
ece2a4770f
Refactoring before introducing heartbeat
...
Introduce a level of supervisors to have a parent PID for the heartbeat
process.
2020-03-31 10:12:35 +02:00
b693f71c5d
Add todo task
2020-03-30 17:46:27 +02:00
47d46d145b
Send close when client sends frame too large
2020-03-30 15:18:57 +02:00
e899bdfb3e
Refactor tune
...
Use int32 for both max frame size and bump max frame size to 1 MB.
2020-03-27 15:55:57 +01:00
9667525cd7
Update test for opening sequence
2020-03-26 17:52:54 +01:00
4e035dfbdb
Add support to send close
2020-03-26 16:56:53 +01:00
a26962cfcb
Handle close command from client
2020-03-26 14:52:19 +01:00
7c33112f25
Check virtual host access in open
2020-03-26 09:53:28 +01:00
728a017f9e
Add skeleton for tune and open
2020-03-25 17:06:04 +01:00
b84de96d0b
Add test for authentication
2020-03-25 10:08:10 +01:00
55a836b488
Refactor to pre/post authentication frame handling
2020-03-24 17:42:10 +01:00
6ae44768f3
Handle null binary in SASL authenticate
2020-03-24 16:40:05 +01:00
22dc72c9d9
Handle SASL authenticate
2020-03-24 11:47:50 +01:00
e2f11fdafc
Handle SASL handshake
2020-03-23 14:54:59 +01:00
1f42faeab8
Refactor after some Osiris changes
2020-03-23 10:48:46 +01:00
4e8a801e8d
Add license information
2020-03-17 10:59:37 +01:00
ab98aecd17
Initial import
2020-03-16 16:40:02 +01:00
Michael Klishin
David Ansari
Arnaud Cogoluègnes
dcorbacho
Michal Kuratczyk
Karl Nilsson
Philip Kuryloski
Gerhard Lazu
Michal Kuratczyk
kjnilsson
Michael Klishin