Commit Graph

365 Commits

Author SHA1 Message Date
Philip Kuryloski a22234f6eb Updates for rules_erlang 2.5.0
rabbitmq_cli uses some private rules_erlang apis that have changed in
the upcoming release

Additionally:
- Avoid including both standard and test versions of amqp_client in
integration test suites
- Eliminate most of the compilation order hints (explicit first_srcs)
in the bazel build
- Fix an include statement - in bazel, an app is not available to
itself as a library at compilation time
2022-04-07 14:54:37 +02:00
Arnaud Cogoluègnes a9cc401e3c
Bump Java test suites dependencies for stream plugins 2022-04-01 16:42:32 +02:00
Arnaud Cogoluègnes 90d110eb20
Return consumer in send_chunks
Not a tuple. The passed-in consumer is updated with the returned
tuple, so we're better off returning the updated consumer record.

References #4368
2022-03-29 16:49:24 +02:00
Arnaud Cogoluègnes aa88fd5b29
Make Osiris listener registration idempotent
Several listeners for the same offset can be registered,
which causes problems for low rate, where consumers hit
the end of the stream often. Then Osiris messages accumulate
in the process mailbox, which slows it down considerably after
some time.

This commit makes the listener registration idempotent for a given
offset.
2022-03-29 11:52:42 +02:00
Michael Klishin c38a3d697d
Bump (c) year 2022-03-21 01:21:56 +04:00
Philip Kuryloski dabf053cf8 Additional dialyzer warning fixes
Currently loading of the rabbitmq_cli defined behaviors compiled with
Elixir does not work, so we ignore the callback definitions contained therein
2022-02-25 18:14:35 +01:00
Philip Kuryloski 226e00fcd2 Tighten up dialyzer usage
now that rules_erlang no longer cascades up dialyzer warnings from deps
2022-02-24 11:18:41 +01:00
Philip Kuryloski d8201726ae Ignore dialyzer warnings for most apps 2022-02-21 09:19:56 +01:00
Arnaud Cogoluègnes 1c57c69df2
Mention strings are UTF8-encoded in stream protocol 2022-02-17 16:13:40 +01:00
Arnaud Cogoluègnes 043989f4fb
Handle stream coordinator error response for topology request
A stream may not be available or in an inconsistent state and the
stream coordinator reports this with an error that the stream
manager handles like an appropriate response. The stream protocol
adapter then tries to extract the topology from the error.

This commit makes the stream manager handles the error correctly
so that the stream protocol adapter reports the unavailability
of the stream to the client.
2022-02-07 10:28:20 +01:00
Arnaud Cogoluègnes 4dc04926ff
Tolerate unknown fields in stream CLI command requests
This is useful in case new fields are needed in further
versions. A new version node can ask for new fields to an
old version node in a mixed-version cluster.
If the old version node returns a known
value for unknown fields instead of failing, the new
node can set up appropriate default value for these
fields in the result of the CLI commands.
2022-02-02 17:10:56 +01:00
Philip Kuryloski efcd881658 Use rules_erlang v2
bazel-erlang has been renamed rules_erlang. v2 is a substantial
refactor that brings Windows support. While this alone isn't enough to
run all rabbitmq-server suites on windows, one can at least now start
the broker (bazel run broker) and run the tests that do not start a
background broker process
2022-01-18 13:43:46 +01:00
Arnaud Cogoluègnes 96c9270fb3
Bump dependencies in streams Java tests 2022-01-17 16:13:29 +01:00
Arnaud Cogoluègnes 1bcca0745e
Increase timeout in stream test
Some CI environment may need a bit more time.
2021-12-10 12:08:02 +01:00
Arnaud Cogoluègnes 4c397c54fa
Use AssertJ conditions in Java stream test suite 2021-12-10 11:52:54 +01:00
Arnaud Cogoluègnes bd4771addb
Return appropriate response when stream leader not available
Fixes #3874
2021-12-10 10:58:48 +01:00
Arnaud Cogoluègnes 8f40e534ed
Bump Java test dependencies in stream test suites 2021-12-10 09:18:22 +01:00
Arnaud Cogoluègnes b6f5b1c21a
Merge pull request #3842 from rabbitmq/rabbitmq-server-3841-stream-consumer-nested-record
Create configuration nested record in stream consumer record
2021-12-03 11:11:40 +01:00
Arnaud Cogoluègnes 854efb3e85
Declare variable for better formatting
References #3841
2021-12-03 10:59:45 +01:00
Arnaud Cogoluègnes 81b9632535
Create configuration nested record in stream consumer record
Only a couple of fields of the stream consumer record change
very frequently (credits and Osiris log reference), so this commit
introduces a nested record in the main consumer record that
contains the immutable fields. This potentially avoids producing
a lot of garbage, especially when the consumer state contains
several properties (consumer name, or single active consumer information
in the future).

Fixes #3841
2021-12-03 10:34:51 +01:00
Arnaud Cogoluègnes 17d0ba9317
Return "no offset" (19) code when stored offset is undefined
Fixes #3783
2021-12-01 17:59:14 +01:00
Michael Klishin ce497a5b52
Merge pull request #3798 from wrobell/rabbitmq-streams-doc-protocol-timestamp
Link to most recent description of Osiris chunk format
2021-11-24 21:48:07 +03:00
wrobell 07dc62af7c Link to most recent description of Osiris chunk format
The most recent description of Osiris chunk format does not reference
the timestamp field to be "posix-ish" anymore. This was bit misleading
as it is Erlang's system time.

Add link to Erlang system time documentation to the subscription command
description to avoid confusion about the timestamp field.
2021-11-24 18:34:55 +00:00
Michael Klishin 01f68b80c9
Merge pull request #3723 from wrobell/rabbitmq-streams-doc-protocol-deliver-fix
Fix description of "deliver" command in RabbitMQ Streams protocol documentation
2021-11-14 21:59:12 +03:00
wrobell b4f39031f6 Fix description of "deliver" command in RabbitMQ Streams protocol documentation
Add missing fields - chunk type, timestamp, trailer length and reserved.
The fields are added after looking at

    3fd348d225/src/main/java/com/rabbitmq/stream/impl/ServerFrameHandler.java (L267)
    https://github.com/qweeze/rstream/blob/master/rstream/schema.py#L376

Change type of "version" field - it shall be uint16 like in other
commands.
2021-11-14 14:51:35 +00:00
Philip Kuryloski e4505bea3d Use unique group names in rabbit_stream_SUITE
Technically duplicate names is supported by common test, but we have
seen it contribute to flakiness in our suite in practice

(cherry picked from commit 513446b6d1)
2021-11-09 13:52:41 +01:00
Arnaud Cogoluègnes ba5cfb1bae
Use version range for stream client Maven dependency
In test suite. Note a snapshot for 1.0-SNAPSHOT has been
pushed by mistake a while ago, so this one must be
excluded. It's unlikely it will be erased, as the snapshots
for the first stable version should be 1.0.0-SNAPSHOT.
2021-11-03 14:05:23 +01:00
Michael Klishin 40db671e0b
Merge pull request #3633 from wrobell/rabbitmq-streams-doc-protocol-hex
Use hex values in RabbitMQ Streams protocol description
2021-11-02 21:58:42 +03:00
wrobell 08c417fdb7 Use hex values in RabbitMQ Streams protocol description
The protocol documentation uses decimal values for error and request key
codes.

Let's use hex values instead. This helps when looking at a request and
its response - 0x0006 and 0x8006 vs. 6 and 32774.

Also, when looking at output of protocol analysis tools like Wireshark,
a hexadecimal value will be printed, for example:

    "Nov  1, 2021 23:05:19.395825508 GMT","60216,5552","00000009000600010000000701"
    "Nov  1, 2021 23:05:19.396069528 GMT","5552,60216","0000000a80060001000000070001"

Above, we can visually identify delete publisher request and response
(0x0006 and 0x8006) and easily match them in the documentation of the
protocol.

Finally, above argument applies to logging as it is common to log
hex values, not decimal.
2021-11-02 18:31:30 +00:00
Philip Kuryloski 76287fb30c format with bazel files with buildifier 2021-11-02 18:24:26 +01:00
wrobell e74823d68c Fix use of "RabbitMQ Streams" in protocol documentation
The following RabbitMQ page uses "RabbitMQ Streams" or "streams",
therefore use it in the protocol documentation as well

    https://blog.rabbitmq.com/posts/2021/07/rabbitmq-streams-overview
2021-10-29 19:06:06 +01:00
wrobell 57ff5dbf4a Fix RabbitMQ Streams protocol documentation
1. Response for publisher declaration request does not contain
   publisher id.

2. Add mechanism entry to the details of SASL handshake request.

3. SASL handshake response contains list of mechanisms, not just single
   mechanism.
2021-10-28 17:53:22 +01:00
Arnaud Cogoluègnes c0f1e23b17
Disable stream test in mixed versions mode
Not worth testing in this mode, as the stream plugin
does not start without the stream_queue feature flag.
2021-10-20 15:16:45 +02:00
Arnaud Cogoluègnes 030a08fa54
Enable stream_queue flag with env variable in test suite
For mixed version mode, where feature flags are
not explictely enabled.
2021-10-20 15:08:05 +02:00
Arnaud Cogoluègnes d87fa91a52
Enable stream_queue flag in test suite
For mixed version mode, where feature flags are
not explictely enabled.
2021-10-20 14:55:54 +02:00
Arnaud Cogoluègnes 6aabfdbbb4
Format stream plugin 2021-10-20 14:53:08 +02:00
Michael Klishin 2140562209
Merge pull request #3562 from rabbitmq/disable_stream_plugin_3557
Disable the stream_plugin if stream feature flag is not enabled
2021-10-14 18:45:26 +03:00
Arnaud Cogoluègnes 6d0dc3f6b4
Attach add/delete super stream command to streams scope only
Not to ctl.

References #3503
2021-10-14 17:09:18 +02:00
Michael Klishin 2fbc5fab25
Formatting, wording 2021-10-14 17:20:59 +03:00
Gabriele Santomaggio f2df98aef9 change message 2021-10-13 12:01:21 +02:00
Gabriele Santomaggio 2eb625531d Disable the stream_plugin and stream management if the feature flag
is not enabled.
fixes https://github.com/rabbitmq/rabbitmq-server/issues/3557
2021-10-13 11:59:11 +02:00
Gabriele Santomaggio c270ebf037 Disable the stream_plugin if the feature flag
is not enabled.
fixes https://github.com/rabbitmq/rabbitmq-server/issues/3557
2021-10-12 19:09:34 +02:00
Arnaud Cogoluègnes 6b9589bae4
Handle stream arguments in add_super_stream command
max-age, leader-locator, etc.
2021-10-11 16:50:03 +02:00
Arnaud Cogoluègnes ecbd969834
Embed RPC call function in test 2021-10-11 16:50:02 +02:00
Arnaud Cogoluègnes 8ae60bef73
Fix group spec in test 2021-10-11 16:50:02 +02:00
Arnaud Cogoluègnes b0bd5f8a00
Add delete_super_stream CLI command 2021-10-11 16:50:02 +02:00
Arnaud Cogoluègnes a73b1a3d0d
Add add_super_stream CLI command 2021-10-11 16:50:01 +02:00
Arnaud Cogoluègnes 147659093f
Add functions to create/delete super stream in manager 2021-10-11 16:50:01 +02:00
Michael Klishin 72c2e152ef
Merge pull request #3499 from rabbitmq/cla-url
Update CLA URL
2021-10-01 00:16:03 +03:00
Arnaud Cogoluègnes e8d0d380a2
Bump dependencies in stream Java tests 2021-09-28 14:02:10 +02:00